User avatar
Gr33nDrag0n
Lisker
Posts: 123
Joined: Sat Mar 26, 2016 8:22 pm
Location: Quebec, Canada

HowTo: Install a Public Auto-Snapshot Server (Updated 2016-11-30)

Sat Oct 08, 2016 7:33 pm

# Warning(s)

  • Lisk v0.4.0+ is mandatory. Latest lisk.sh and lisk_snapshot.sh included in this version are required.
  • This guide is assuming that you already are able to build a basic functional Lisk server.
  • The snapshot server need a lisk client instance running and in sync. on the server.
  • The snapshots are made from a second instance running in parallel.
  • Both node instances will share the postgres service.
  • Using the Lisk client as a forging server for delegate on same server not recommended.
  • Do NOT use the Lisk client './public/' directory to host your backups.
  • HTTP (port 80) redirection to HTTPS (port 443) isn't needed. I'll give the recipe with (optional) tag in the guide.

# Prepare Firewall

Make sure port 80 is open and without redirection in your firewall.

(Optional) Make sure port 443 is open and without redirection in your firewall.


# Users

I'm using the following users, adjust accordingly to your setup.

The "lisk" user is the user running lisk client.
It will need write permission in the target directory where the snapshots will be save.

The "nginx" user is the user that will run nginx.
It will need read permission in the target directory where the snapshots will be save.


# Directories

I'm using the following directories, adjust accordingly to your setup.
Do NOT use a sub-directory of Lisk client for 'shared backup directory'.
This will allow to keep it untouched and keep service online when updating Lisk client.

Lisk wallet directory ==> /opt/lisk/client/
Shared backup directory ==> /opt/lisk/backup/
(Optional) SSL directory ==> /etc/letsencrypt/live/snapshot.lisknode.io/

If you need help to generate a free SSL certificate, refer to this guide. (You can skip port redirection parts and lisk configuration parts.) ==> HowTo: Free SSL Certificate


(Optional) Write a Tag file in the shared directory

Code: Select all

touch '/opt/lisk/backup/### Vote Gr33nDrag0n ###'



Nginx Installation and Configuration

Install nginx

Code: Select all

# Ubuntu
apt-get install nginx
# CentOS
yum install nginx


Backup original nginx configuration file.

Code: Select all

cd /etc/nginx
cp nginx.conf nginx.conf.original


Open the configuration file.

Code: Select all

vi nginx.conf


Remove all server configuration block.

Select one of the 2 following examples.

HTTP Only

Code: Select all

server {
    listen 80;
    server_name snapshot.lisknode.io;
    root /opt/lisk/backup;

    location / {
        autoindex on;
    }
}

(Optional) HTTP to HTTPS redirection

Code: Select all

server {
    listen 80;
    server_name snapshot.lisknode.io;
    return 301 https://snapshot.lisknode.io$request_uri;
}

server {
    listen 443;
    server_name snapshot.lisknode.io;
    root /opt/lisk/backup;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/snapshot.lisknode.io/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/snapshot.lisknode.io/privkey.pem;
    ssl_protocols TLSv1.1 TLSv1.2;
    ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 10m;
   
    location / {
        autoindex on;
    }
}

Replace 'snapshot.lisknode.io' by your domain.

Replace '/opt/lisk/backup' by your backup directory path.

Save the modified configuration.

Test Configuration

Code: Select all

nginx -t

Start nginx server

Code: Select all

systemctl start nginx

Test nginx web server is up and running. Using your browser, go to your website URL.

Enable nginx server at system’s booting time.

Code: Select all

systemctl enable nginx

Stop nginx server (for reference only)

Code: Select all

systemctl stop nginx



Lisk Client - Create snapshot.json

Install jq using 'apt-get install jq' or 'yum install jq'

Adjust the path in 'mainconfig' & 'snapshotconfig' and run:

Code: Select all

mainconfig='/opt/lisk/client/config.json'
snapshotconfig='/opt/lisk/client/snapshot.json'

cp $mainconfig $snapshotconfig
jq '.port=9000' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.version="9.9.9"' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.minVersion="9.9.9"' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.logFileName="logs/snapshot.log"' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.loading.loadPerIteration=101' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.db.database="lisk_snapshot"' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.peers.list=[]' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig
jq '.ssl.enabled=false' $snapshotconfig > tmp.$$.json && mv -f tmp.$$.json $snapshotconfig


Test manual snapshot

I recommend running 1st execution manually to confirm everything work as expected.

For reference, this is lisk_snapshot.sh usage manual.

Code: Select all

Usage: lisk_snapshot.sh [-t <snapshot.json>] [-s <config.json>] [-b <backup directory>] [-d <days to keep>] [-r <round>] [-g]
 -t <snapshot.json>        -- config.json to use for validation
 -s <config.json>          -- config.json to create target database
 -b <backup directory>     -- Backup direcory
 -d <days to keep>         -- Days to keep backups
 -r <round>                -- Round height to snapshot at
 -g                        -- Make a copy of backup file named blockchain.db.gz


For 1st manual execution, I like to work with two different SSH connections.

# Tab 1 - Adjust the paths and run:

Code: Select all

bash /opt/lisk/client/lisk_snapshot.sh -t /opt/lisk/client/snapshot.json -s /opt/lisk/client/config.json -b /opt/lisk/backup -d 3 -r highest -g


# Tab 2 - Adjust the path and run:

Code: Select all

tail -f /opt/lisk/client/logs/snapshot.log



Test Rebuild for manual snapshot

Using your browser, go to your website URL.
Confirm that you see and are able to download your manual snapshot.
Use the following HowTo with your server URL to test a rebuild from your server.
HowTo: Coldstart or Rebuild from an Auto-Snapshot Server


Cronjob creation (Enabling Auto-Snapshot)

Edit the crontab. The -u is parameter is the user that is running the Lisk client.

Code: Select all

sudo crontab -u lisk -e

Edit and Add the following crontab entry.

*/6 represent 00:00, 06:00, 12:00, 18:00.
I recommend using a number that can divide 24 equally. I.e. 2, 3, 4, 6, 8, 12.

The timing must respect the time it took for your manual snapshot to complete.
I recommend a good buffer because blockchain will always increase in size.

For example, I personally use on my server */6 on TestNet and */8 on MainNet.
It take about 4 hours for TestNet and 5.5 hours in MainNet.

Code: Select all

0 */6 * * * /usr/bin/bash /opt/lisk/client/lisk_snapshot.sh -t /opt/lisk/client/snapshot.json -s /opt/lisk/client/config.json -b /opt/lisk/backup -d 3 -r highest -g



Lisk client update/upgrade notes

Each Lisk client update, some special steps will be needed for the auto-snapshots server.

  • The nginx shared directory can stay online for the whole process.
  • Before update, the cronjob must be disabled.
  • Before update, all instances of Lisk "node" process(es) must be stopped or killed.
  • After update, the snapshot.json file must be re-created.
  • After update, test manual snapshot and take note of the time it take.
  • After update, the cronjob must be re-enabled and timing adjusted if needed based on manual test.

Conclusion

If you got a question, comment don't be shy to reply to this post.

If you implemented a auto-snapshot server. Include your URL with a MainNet or TestNet tag.
Last edited by Gr33nDrag0n on Fri Dec 16, 2016 1:15 am, edited 11 times in total.
Gr33nDrag0n | 194109334904015388L | Delegate | lisknode.io

User avatar
Gr33nDrag0n
Lisker
Posts: 123
Joined: Sat Mar 26, 2016 8:22 pm
Location: Quebec, Canada

List of existing public Auto-Snapshot servers (Updated 2016-10-08)

Sat Oct 08, 2016 7:33 pm

Gr33nDrag0n | 194109334904015388L | Delegate | lisknode.io

User avatar
oliver
CTO
Posts: 23
Joined: Sat Jan 30, 2016 9:23 am

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-08)

Sat Oct 08, 2016 7:37 pm

Excellent job Gr33nDrag0n! A very worthwhile contribution. Many thanks :)

Phoenix1969
Lisker
Posts: 99
Joined: Sat Apr 02, 2016 6:21 pm
Location: Hawaii
Contact: Website

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-08)

Sat Oct 08, 2016 8:07 pm

Sweet Drag0n! Damn!
Vote for Phoenix1969

User avatar
Gr33nDrag0n
Lisker
Posts: 123
Joined: Sat Mar 26, 2016 8:22 pm
Location: Quebec, Canada

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-08)

Sat Oct 08, 2016 8:29 pm

Phoenix1969 wrote:Sweet Drag0n! Damn!


After Drag0nSnaps TM will we see PhoenixSnaps TM too ?
Gr33nDrag0n | 194109334904015388L | Delegate | lisknode.io

User avatar
punkrock
Lisker
Posts: 96
Joined: Thu Feb 25, 2016 6:41 pm
Location: Germany
Contact: Website Twitter

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-08)

Sun Oct 09, 2016 1:41 am

This is awesome! As soon as I can, I will check use your guide!

Thank you very much for the work!
Delegate name: punkrock
Address: 6853061742992593192L


Daniel B. - Lisk Ambassador for Germany
Website | My delegate proposal | Twitter | Bitcointalk

User avatar
redsn0w
Global Moderator
Posts: 201
Joined: Sat Apr 02, 2016 8:41 am
Contact: Website Twitter

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-08)

Sun Oct 09, 2016 3:02 pm

Well done Gr33nDrag0n, like always... thanks for your contribution !

User avatar
Poly#Crypto
Lisker
Posts: 430
Joined: Thu May 05, 2016 2:19 pm
Contact: Twitter

Re: HowTo: Install an Auto-Snapshot Server (Updated 2016-10-10)

Tue Oct 11, 2016 11:37 am

As always a great HowTo by Gr33nDrag0n, thanks very much.

Image
polycrypto ❖ Lisker since day one ❖ 7 secure nodes - Mainnet/Testnet

My Delegate Proposal | My Transparency Reports | My Twitter | Bitcointalk Profil

Diana Smith
Posts: 3
Joined: Tue Sep 06, 2016 3:50 am
Location: Augusta GA
Contact: Website

Re: HowTo: Install a Public Auto-Snapshot Server (Updated 2016-10-10)

Mon Oct 17, 2016 10:50 am

Thanks Drag0n given very informative answer....its really great info.
Diana Smith is a professional web content writer in Sagenext Infotech LLC. I have extensive experience in writing about Sage in the cloud and Sage cloud hosting

User avatar
Gr33nDrag0n
Lisker
Posts: 123
Joined: Sat Mar 26, 2016 8:22 pm
Location: Quebec, Canada

Re: HowTo: Install a Public Auto-Snapshot Server (Updated 2016-10-10)

Mon Oct 17, 2016 10:48 pm

Your welcome. (punkrock, redsn0w, polycrypto and diana) Happy you find it well made and usefull.
Gr33nDrag0n | 194109334904015388L | Delegate | lisknode.io

Return to “Guides”

Who is online

Users browsing this forum: No registered users and 1 guest