markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

[Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:17 am

This guide aims to deliver a stable and decent setup. We only tune what makes sense from a SysOp point of view and not some fancy stuff. This is serious business, so please hold this into mind.

Hello Liskers,

Most people want to run a Lisk Node as a VPS in the cloud.
This has many advantages, especially when it comes to costs (ROI), though there are some draw backs we have to deal with. I am not only talking about IO and RAM, but also about limited CPU, Cycles, Beancounters, etc.

Standard Linux Kernels are not tuned for special purposes, but running in a more generic scheme that fits all purposes. Even the server kernels run in a "one fits all" mode - which we can tackle.
So, as you can imagine, there is plenty of room for optimizatons.

I want to share some Systems-Administrator tricks with you, that help you tuning the Linux Kernel and network stack to make your Lisk Node running as a forging power horse under limited conditions!

We need to tune RAM, Swapping, Firewall and last but not least, some kernel parameters that control the network stack directly:
There are a couple of sysctl and iptables tweaks, that ubuntu 14 LTS has not enabled by default! :!:

I will make it quick'n easy for beginners and point you to some tweaks that help to handle fast round robin forging und higher loads.
Believe me, this will make a HUGE IMPACT if you are on a very small VPS running with default settings.

########

Let's assume

1. you are running on Ubuntu 14.04
2. you installed Lisk node
maybe you followed excellent "Secure basic setup of a delegate server"
3. just started with bash lisk start, gave it some time until the blockchain synced
and now you see everything is up and running well and things are good to go.

What to do now?
In the next posts we will look into this topics and tweak with

* Network tunings
* RAM tunings
* Firewall settings
* Lisk Node / Database settings

Please share with us your tweaks and help making this more valuable for the community.
You can reach me in the https://lisk.chat or drop me a PM to contribute to this topic.

########

HOW YOU CAN HELP
Some network providers have special VPS that are already tuned, so lucky you! But most aren't.

If you are on a such machine, it would be really cool if you share you're settings with us.
I will collect them in the thread and on another central place. Thanks!

########

QUICK NOTE ABOUT ME AND MY RUN FOR DELEGATE:
I am a radical common sense extremist and libertarist, studying how to build a free society with free markets - and as a true dissident I am opposing central powers in general.
That's why I am setting up some lisk nodes to decentralize powers and build a new economy, together with you, folks.

If you like to join in, if you like this thread and find it valuable for the commuity, I kindly ask you to vote for me as delegate.
Personally I will vote back all the guys, that are contributing to this thread with valuable information and helping me to make this guide more complete.

My data:
Delegate ID: markus
Lisk Address: 17634557053657750312L

Markus for Delegate [SysOp & Rebel]

Now, please go on an read the next posts and hopefully you can find something usable that makes LISK network stronger and more attractive.
Last edited by markus on Sat May 14, 2016 2:32 am, edited 2 times in total.
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:17 am

- Reserved for later -
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:18 am

Network Tunings


# SYN TUNING

Let's get right into TCP Stack SYN tuning.

Before we start tweaking around with network stack settings, you should make a quick test, wether SYN tuning is something for you, or if you are fine and good to go to the next section.

1.
First, let's see the open sockets in your VPS while lisk is up and running. Check if this applies to your VPS:

Code: Select all

netstat -tlna

Is this a huge list? do you see many TIME_WAIT entries?

Make a test and count the open sockets that are open in your VPS

Code: Select all

netstat -tlna | wc -l
698


wait 10 seconds and do it again:

Code: Select all

netstat -tlna | wc -l
757


If this is a number above, let's say 100 and going up and down, but mostly just rising, this is definetly too high for just forging and we have room to optimize.

TIME_WAIT tcp syn requests could overflow the buffer on many VPS after some days or even hours in use , which depends on RAM and other other (mostly from hoster controlled) ressources.

That means the server is crammed full and not able to establish new connections (open new sockets) you can't help forging when it is your turn in the round robin.

I assume you do all steps as root. If unsure, do now:

Code: Select all

sudo su


Now go ahead.

2.
Load necessary kernel modules

Code: Select all

modprobe -a nf_conntrack nf_conntrack_netlink nf_conntrack_ipv4


No news are good news.

3.
Use sysctl to set network stack parameters.

Therefore you create a new config in sysctl that will load at system start:

Code: Select all

nano -w /etc/sysctl.d/99-markus-lisk-tuning.conf


The next is not often taught. I want to share this with Lisk node users, please use them wisely and respectful!

My recommended values for a common VPS are:

Code: Select all

# markus experten.team tuning
# get rid of netstat TIME_WAIT faster
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1

# Change to '1' to enable TCP/IP SYN cookies. This disables TCP Window Scaling
# (http://lkml.org/lkml/2008/2/5/167)
net/ipv4/tcp_syncookies=1
net/ipv4/tcp_fin_timeout=30
net/ipv4/tcp_keepalive_intvl=180
net.core.somaxconn = 2048
net.ipv4.tcp_syn_retries = 1
net.ipv4.tcp_synack_retries = 1
net.ipv4.tcp_keepalive_intvl = 5
net.ipv4.tcp_keepalive_probes = 1
net.ipv4.tcp_keepalive_time = 30

# Disable ICMP redirects. ICMP redirects are rarely used but can be used in
# MITM (man-in-the-middle) attacks. Disabling ICMP may disrupt legitimate
# traffic to those sites.
net/ipv4/conf/default/accept_redirects=0
net/ipv4/conf/all/accept_redirects=0
net/ipv6/conf/default/accept_redirects=0
net/ipv6/conf/all/accept_redirects=0

# Ignore bogus ICMP errors
net/ipv4/icmp_echo_ignore_broadcasts=1
net/ipv4/icmp_ignore_bogus_error_responses=1
net/ipv4/icmp_echo_ignore_all=0

# Don't log Martian Packets (impossible packets)
net/ipv4/conf/default/log_martians=0
net/ipv4/conf/all/log_martians=0

net.ipv4.ip_local_port_range = 18000 65535
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 30
net.netfilter.nf_conntrack_tcp_timeout_established = 600

#if you use persistent http sonnections
net.ipv4.tcp_slow_start_after_idle = 0


just copy and paste them into the editor, then save the file (strg+o,strg+x).

To test the tweaks straight away, just run:

Code: Select all

sysctl -p


No errors? Perfect!

Now double check, if the new settings have an effect:

Code: Select all

netstat -tlna | wc -l
68


YEAH! Seems to make an instant impact! That's it for now!

Head to the next topics if applicable.

###########################

My personal opinion about SYN:
Syncookies are a nuisance and should be switched off. Period.

Since my tuning, the buffer of the server no longer overflows that hinders taking a handshake.

There are likely some newcomers here that have a problem, to efficiently serve the Round Robin, if they do not run a kernel tuning.
Firewall Tips are also no. Also no resources to other system except scrap Ubuntu.

Nobody needs TCP Window Scaling anymore, but ubuntu still has them enabled.

After all tuning, I have just some TIME_WAIT Slots at once, all connections are established and immediately closed after use.

This network stack is a fast and the VPS a responsive killer machine now, even with slowbuntu on it 8-) :twisted:

Let me know what you think and tell us your tunings!
Last edited by markus on Sat May 14, 2016 9:06 pm, edited 7 times in total.
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:19 am

## Basic RAM, Swap, Cache, Filesystem Tuning

On a VPS the ressources are limited. Even on advertized monster VPS with decent network connections (Bandwidth is not that important for running lisk), there are some bottlenecks we could try to address:

1. FS - Filesystem Performance

make sure that all FS mounts are mounted with the "relatime" option:

Code: Select all

cat /proc/mounts | grep relatime


If it lists your /home and / or root fs then you are good. go one step further.

Some VPS Providers forget this easy tuning, force enable this as root:

Code: Select all

nano -w /etc/fstab

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/vda1 during installation
UUID=cb6e57f9-4353-4f9f-86f9-e6a41sdvs48e /               ext4    errors=remount-ro,realtime 0       1
# swap was on /dev/vda5 during installation
UUID=999acc6d-e97f-4693-9cc1-45bhfd91b5bf none            swap    sw              0       0


More FS tunings are possible, esp. depending on your filesystem (ext3, ext4, xfs, etc), maybe we integrate them if you have suggestions.

################

2. Swap tuning

You cannot tune Swap itself, despite using special caching filesystems, that are not available in VPS. But we should ALWAYS check RAM size, free memory and if there is enough Swap.

Please double check that lisk is up and running at least 1 hour!
Then find out, how much free memory your server has:

Code: Select all

# Calculate how much memory and swap is FREE AVAILABLE
free_data="$(free)"
mem_data="$(echo "$free_data" | grep 'Mem:')"
free_mem="$(echo "$mem_data" | awk '{print $4}')"
buffers="$(echo "$mem_data" | awk '{print $6}')"
cache="$(echo "$mem_data" | awk '{print $7}')"
total_free=$((free_mem + buffers + cache))
used_swap="$(echo "$free_data" | grep 'Swap:' | awk '{print $3}')"

echo -e "Free memory:\t$total_free kB ($((total_free / 1024)) MB)\nUsed swap:\t$used_swap kB ($((used_swap / 1024)) MB)"


It says something like:

Code: Select all

Free memory:    1378460 kB (1346 MB)
Used swap:      208 kB (0 MB)


The free memory parameter is important to calculate the next tunings!

A small excerpt about Swap:
How can swap speed up my system? Doesn't swapping slow things down?

The act of transferring data from RAM to swap is a slow operation, but it's only taken when the kernel is pretty sure the overall benefit will outweigh this. For example, if your application memory has risen to the point that you have almost no cache left and your I/O is very inefficient because of this, you can actually get a lot more speed out of your system by freeing up some memory, even after the initial expense of swapping data in order to free it up.
It's also a last resort should your applications actually request more memory than you actually have. In this case, swapping is necessary to prevent an out-of-memory situation which will often result in an application crashing or having to be forcibly killed or making your VPS unresponsive at all.
Swapping is only associated with times where your system is performing poorly because it happens at times when you are running out of usable RAM, which would slow your system down (or make it unstable) even if you didn't have swap. So to simplify things, swapping happens because your system is becoming bogged down, rather than the other way around.


So swap could save your ass in many situations, where memory is low, and even if not, you still should have it, believe me.

a) Does your VPS have an SSD with high IOPS? But not much swap? or the swap is on a slow HD? and the FREE memory is below 864MB?

Then this is something for you!
As we are on a SSD, Swap performance should be decent enough to use it a bit more than as last ressort.

Rules of thumb (in conjunction with next option):
If you are running below 512MB free RAM you must setup swap, I suggest 1.5GB.
If you are running below 864MB free RAM you should setup swap, I suggest 1GB.
If you are above free 1.5GB RAM and have no swap, please add 1GB for safety.

So let's create the swapfile:

Code: Select all

fallocate -l 1G /swapfile-1

no news are good news...

Code: Select all

ls -alht /
total 1.1G
drwxr-xr-x  23 root root 4.0K May 23 06:18 .
drwxr-xr-x  23 root root 4.0K May 23 06:18 ..
-rw-r--r--   1 root root 1.0G May 23 06:18 swapfile-1


A world-readable swap file is a huge local vulnerability, so let's secure the swapfile!

Code: Select all

chown root:root /swapfile
chmod 0600 /swapfile
ls -lh /swapfile


Turn on the swap file

First, use the mkswap command as follows to enable the swap space on Ubuntu:

Code: Select all

mkswap /swapfile-1


Sample outputs:

Code: Select all

Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=38cf6c53-af61-492a-be74-92963961be9d


Finally, activate the swap file, enter:

Code: Select all

swapon /swapfile-1


Verify new swap file and settings on Ubuntu and type the following command

Code: Select all

swapon -s


Sample outputs:

Code: Select all

Filename            Type      Size   Used   Priority
/dev/sda5                               partition   3998716   704   -1
/swapfile-1                               file      2097148   0   -2


Now add the new Swapfile to the fstab to make it reboot persistent.

Code: Select all

echo '/swapfile-1     none    swap    sw      0       0' >> /etc/fstab

If you are unsure, take a look into the file:

Code: Select all

nano -w /etc/fstab

Hint: you can disable the old swap, if this is on a HDD and use the new on SSD, which is faster.

That's it, double check if the system is using more Swap now buy using swapon -s or the top program.

# Swap tuning
Check the swappiness value

Code: Select all

cat /proc/sys/vm/swappiness

If this is ubuntu style 60 or more, then you are too high, I recommend anything below 50 for SSD Swaps for nodes.

So make the VPS use this Swap more wisely

Code: Select all

echo '
## SWAP TUNING
vm.swappiness = 30
vm.vfs_cache_pressure = 60
' >> /etc/sysctl.d/99-markus-lisk-tuning.conf


you can apply this option immediately without reboot with this command:

Code: Select all

sysctl -pw /etc/sysctl.d/99-markus-lisk-tuning.conf


b) Does your VPS have no SSD, no SAS or SSD with low IOPS, or just normal HD and less than 2GB RAM?
Then look that you have at least 1GB Swap and set swappiness to 20.

If you have more than 3GB RAM only for lisk and nothing else to compute, and no SSDs, no SAS or SSD with low IOPS, or just normal HD, then set swappiness to 5. I mean it, set it to 1-10. This will incredibly improve server speed.

## Proof of work

If you are unsure or sceptical, or you just want to find better values do following:

Code: Select all

cd;
sync
echo 3 > /proc/sys/vm/drop_caches

dd if=/dev/zero of=/tmp/testfile count=1 bs=900M

sysctl -w vm.vfs_cache_pressure=100
find / > /dev/null
cp /tmp/testfile /tmp/testfile2
time find / > /dev/null


sysctl -w vm.vfs_cache_pressure=50
find /  > /dev/null
cp /tmp/testfile2 /tmp/testfile3
time find / > /dev/null

rm -f /tmp/testfile /tmp/testfile2 /tmp/testfile3


And? did it make it faster? Let me know how it goes.

Cheers,
Markus
Last edited by markus on Mon May 23, 2016 4:52 am, edited 5 times in total.
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:52 am

Firewall Settings

Firewall is a special topic.

As a long exeperienced Administrator I can reveal to you:
If you dont have services running (exposed to the world), you don't need a firewall. Period.

Since modern firewalls can do more than just blocking packets, they can become quiet handy, though, especially with Packet Filtering - which we will look into now.

Take away:
1. Packetfilters can help regulating the network packets which can reduce load and prevent abuse.
2. Having no FW is better than a bad configured one.

Will say: Don't tinker around if you dont understand anything. I mean it. Better leave the system as it is, that wont have any impact on your performance!

### Harden the server

1. switch off all unneccessary services
2. and shut down the server ports despite needed ones
3. regulate the traffic flow / speed


- Reserved for later -
Last edited by markus on Sat May 14, 2016 1:08 am, edited 1 time in total.
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sat May 14, 2016 12:52 am

Lisk Node.js and Database Tunings


- Reserved for later -
This guide is work in progress!

If you like, you can contribute your tweaks. just contact me!
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

User avatar
crilleaz
Lisker
Posts: 42
Joined: Thu Apr 21, 2016 10:23 am
Location: The Netherlands
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Sun May 15, 2016 11:33 pm

Hi Markus,

Thanks for providing us with these tweaks! I would love to see more.
liskfaucets. Please read my announcement: https://forum.lisk.io/viewtopic.php?f=6&t=415

markus
Lisker
Posts: 12
Joined: Fri May 13, 2016 8:37 am
Location: Earth / Internet
Contact: Website

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Wed May 18, 2016 10:51 pm

Thanks!

More are coming soon!
Help me protecting the network against hackers and governments!
Markus Delegate Proposal
Member of the LFFF (Foundation) to protect Lisk in the future!

---> freiheitswelle | 10931950175235745529L

User avatar
cc001
Lisker
Posts: 103
Joined: Sat Mar 12, 2016 12:48 pm

Re: [Tuning LISK Node] Markus' Ultimate Tweaking Guide {ubuntu 14.04}

Thu May 19, 2016 9:35 am

Very nice network tuning! Thanks!
I had up to 1700 connections, most of them waiting. Now it's down to below 100, only established ones
My Delegate Node
My Websites:
My Scripts:
My Tutorials:
  • Secure basic setup of a delegate server
  • how to set up 'lisk-rake' (outdated)

Return to “Guides”

Who is online

Users browsing this forum: No registered users and 1 guest