User avatar
punkrock
Lisker
Posts: 105
Joined: Thu Feb 25, 2016 6:41 pm
Location: Germany
Contact: Website Twitter

HowTo: Easy monitoring and notification system for your node (MONIT + Gmail + Missed Notifications Reminder)

Mon Apr 25, 2016 11:37 pm

notification system.png
notification system.png (74.25 KiB) Viewed 942 times


This guide shows you how to monitor the logfile of your Lisk node on forks. If your node forks, you'll get an email to your Android smartphone immediately and a permanent alarm sound + vibration appears every 15 seconds unless you open your Gmail app. With this setup, you'll never miss any downtime of your node. Monitoring the logsfile on forks is just an example. MONIT is much more powerful and can monitor nearly everything related to your node. Check this website for further infos: https://mmonit.com/monit/documentation/monit.html

Requirements
  • A Lisk node running on Ubuntu (tested only on Ubuntu!)
  • MONIT for Ubuntu (~$ sudo apt-get install monit)
  • An Android smartphone that can run the Gmail app
  • Gmail for Android
  • Missed Notifications Reminder for Android

Installation of MONIT and creation of the first config-files
Install MONIT and configure its conf-file.
~$ sudo apt-get install monit
~$ sudo nano /etc/monit/monitrc

Now clear out everything holding CTRL + K and enter following stuff and change the commented stuff as needed.

Code: Select all

####################### B E G I N   O F   C O N F - F I L E #######################

set daemon 120                                 # check services at 2-minute intervals
set logfile /var/log/monit.log
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set mailserver SMTP.MAILHOSTER.COM port 587    # primary mailserver
      username "LoginUsername" password "LoginPassword"
      using tlsv1                              # you may have to use ssl or tlsv12 here
      with timeout 30 seconds
set eventqueue
      basedir /var/lib/monit/events            # set the base directory where events will be stored
      slots 100                                # optionally limit the queue size
set mail-format {
      from: SEND@MAILHOSTER.COM
        subject: ALARM on Test-Server -- $EVENT $SERVICE
        message: $EVENT Service $SERVICE
        Date:        $DATE
        Action:      $ACTION
        Host:        $HOST
        Description: $DESCRIPTION

        Bye,
        Monit
}
set alert RECEIVE@MAILHOSTER.COM                # this mail will receive all alerts from MONIT
include /etc/monit/conf.d/*

######################### E N D   O F   C O N F - F I L E #########################


Check status of MONIT and restart it to reload the config
~$ sudo /etc/init.d/monit status
~$ sudo /etc/init.d/monit restart
Important! You should receive a mail now. if not, mail transport is not working. check /var/log/syslog for more infos.


Go into conf.d-folder...
~$ cd /etc/monit/conf.d/

...and create some conf-files for different monitoring stuff.
~$ sudo nano /etc/monit/conf.d/lisk-monitoring-forks-in-logfile

Attention! Correct the path to make shure, that it's linked to your log file!

Code: Select all

####################### B E G I N   O F   C O N F - F I L E #######################

### MAKE SHURE THAT THE PATH IS CORRECT! ###

check file messages with path /home/YOURUSERNAME/lisk-main/logs/lisk.log
ignore match 'cause":5'
if match "Fork" then alert

######################### E N D   O F   C O N F - F I L E #########################


Restart MONIT to reload the config
~$ sudo /etc/init.d/monit restart


Download an alarm sound for your smartphone
Here is a list of alarm sounds to download on your smartphone. I am using this one.


Gmail for Android (Settings)
  • Start the Gmail app and add your the mail address where you will receive the alarm mails from MONIT
  • Go into the Gmail app settings, click on your mail address and do these settings:
    Inbox type (Default)
    Notifications (✓)

    Navigate to Inbox sound & vibrate and make this settings:
    - Label notifications (✓)
    - Sound (choose your favorite alarm sound)
    - Vibrate (✓)
    - Notify for every message (✓)

Android Settings
  • Open the Android system settings of your smartphone and navigate to:
    Android Settings -> Sound & Notification
    - Make sure that your ring tone is loud as the "Missed Notifications Reminder" is attached to the ring tone volume
  • Navigate one step forward to:
    Android Settings -> Sound & Notification -> App notifications
  • Search for the Gmail app, tap on it and make this settings:
    - Treat as priority (✓)
    - Allow peeking (✓)

Missed Notifications Reminder for Android (Settings)
notificator.png
notificator.png (122.61 KiB) Viewed 942 times


Last check
  • Now navigate into Android system settings -> Sound & Notification -> (Advanced) Notification access
    - Make sure that "Missed Notifications Reminder" has access by turning it on

That's it!
Now lock your smartphone and send yourself a mail and wait. The alarm should ring now every 15 seconds and your smartphone should vibrate as long as you don't access the Gmail application.
Last edited by punkrock on Thu Nov 24, 2016 6:07 pm, edited 57 times in total.
Delegate name: punkrock
Address: 6853061742992593192L


Daniel B. - Former Lisk ambassador for Germany
Website | My delegate proposal | Twitter | Bitcointalk

User avatar
ViperTKD
Lisker
Posts: 66
Joined: Mon Apr 04, 2016 11:06 pm
Location: Quebec, Canada
Contact: Facebook Twitter

Re: Monitor your node with MONIT

Tue Apr 26, 2016 3:13 am

Nice guide punkrock! I will definitely try MONIT!
Check my Delegate Candidacy : https://forum.lisk.io/viewtopic.php?f=6&t=246

User avatar
MrV
Lisker
Posts: 60
Joined: Fri Apr 08, 2016 10:35 pm

Re: Monitor your node with MONIT

Mon May 02, 2016 1:55 pm

Thanks! Have it running on lisktools.io now to make sure that node stays accurate
Haven't had to test it yet though :)
Please vote for me: MrV for delegate
Status of my delegate: Delegate Status

User avatar
punkrock
Lisker
Posts: 105
Joined: Thu Feb 25, 2016 6:41 pm
Location: Germany
Contact: Website Twitter

Re: Monitor your node with MONIT

Tue May 03, 2016 4:47 pm

If you are using MONIT, pelase post the scripts you are using here and I will add 'em to the OP. Thank you very much!
Delegate name: punkrock
Address: 6853061742992593192L


Daniel B. - Former Lisk ambassador for Germany
Website | My delegate proposal | Twitter | Bitcointalk

User avatar
cc001
Lisker
Posts: 103
Joined: Sat Mar 12, 2016 12:48 pm

Re: Monitor your node with MONIT

Tue May 03, 2016 9:38 pm

hey punkrock

I use MONIT as well, it is really great, thanks for the tip! I have following comments and additions:

Your "check node process" config would restart lisk as root (because monit is running as root), which is not what you want. I told monit to restart lisk as my regular user with the following config (see as uid xxx and gid yyy):

Code: Select all

  check process node
    matching "node"
    start program = "/opt/lisk/lisk/lisk.sh restart"
      as uid cc001 and gid cc001
    stop program = "/opt/lisk/lisk/lisk.sh stop"
      as uid cc001 and gid cc001


Additionally, I'm also ignoring fork log messages with cause 1. Maybe we should recheck this log messages with the new version of Lisk.

Next point: As I see it, app.log is much bigger than logs.log, maybe we should check both for their size.

I have an additional configuration. I use a self-written python script, that monitors all my nodes and if something is wrong with my master node, it switches automatically to my backup node and vice versa. It looks like this:

Code: Select all

  check program monitor_delegate with path /usr/local/bin/monitor_delegate_nodes.py
    if status != 0 then alert
    if status = 1 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch master backup"
      as uid cc001 and gid cc001
    if status = 4 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch master backup"
      as uid cc001 and gid cc001
    if status = 2 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch backup master"
      as uid cc001 and gid cc001
    if status = 5 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch backup master"
      as uid cc001 and gid cc001


TLDR: MONIT is great :D
My Delegate Node
My Websites:
My Scripts:
My Tutorials:
  • Secure basic setup of a delegate server
  • how to set up 'lisk-rake' (outdated)

User avatar
Grumlin
Lisker
Posts: 122
Joined: Sat Mar 26, 2016 7:15 pm

Re: Monitor your node with MONIT

Thu May 05, 2016 6:38 pm

cc001 wrote:hey punkrock

I use MONIT as well, it is really great, thanks for the tip! I have following comments and additions:

Your "check node process" config would restart lisk as root (because monit is running as root), which is not what you want. I told monit to restart lisk as my regular user with the following config (see as uid xxx and gid yyy):

Code: Select all

  check process node
    matching "node"
    start program = "/opt/lisk/lisk/lisk.sh restart"
      as uid cc001 and gid cc001
    stop program = "/opt/lisk/lisk/lisk.sh stop"
      as uid cc001 and gid cc001


Additionally, I'm also ignoring fork log messages with cause 1. Maybe we should recheck this log messages with the new version of Lisk.

Next point: As I see it, app.log is much bigger than logs.log, maybe we should check both for their size.

I have an additional configuration. I use a self-written python script, that monitors all my nodes and if something is wrong with my master node, it switches automatically to my backup node and vice versa. It looks like this:

Code: Select all

  check program monitor_delegate with path /usr/local/bin/monitor_delegate_nodes.py
    if status != 0 then alert
    if status = 1 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch master backup"
      as uid cc001 and gid cc001
    if status = 4 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch master backup"
      as uid cc001 and gid cc001
    if status = 2 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch backup master"
      as uid cc001 and gid cc001
    if status = 5 for 3 cycles then exec "/usr/local/bin/manage_forging.py switch backup master"
      as uid cc001 and gid cc001


TLDR: MONIT is great :D


can we see monitor_delegate_nodes.py and manage_forging.py?

Rabid
Lisker
Posts: 53
Joined: Sun Apr 03, 2016 9:00 pm

Re: Monitor your node with MONIT

Fri May 06, 2016 3:57 am

cc001 wrote:hey punkrock

I use MONIT as well, it is really great, thanks for the tip! I have following comments and additions:

Your "check node process" config would restart lisk as root (because monit is running as root), which is not what you want. I told monit to restart lisk as my regular user with the following config (see as uid xxx and gid yyy):

Code: Select all

  check process node
    matching "node"
    start program = "/opt/lisk/lisk/lisk.sh restart"
      as uid cc001 and gid cc001
    stop program = "/opt/lisk/lisk/lisk.sh stop"
      as uid cc001 and gid cc001



I did some testing on this, if postgres is down on your node, it will fail to start lisk correctly. you have to use start instead to get postgres running. Theres no mechanism for only starting postgres with lisk.sh, and restart only stops and starts lisk process. However, even with start there can be issues... I asked for restart to have postgres stop and start added to its clause for the next release so hopefully that will be fixed.

Code: Select all

####################### B E G I N   O F   C O N F - F I L E #######################

check process node with pidfile lisk_home/app.pid
matching "node"
    start program = "lisk_home/lisk.sh start"  as uid "user" and gid "user"
    stop program = "lisk_home/lisk.sh stop" as uid "user" and gid "user"
   
######################### E N D   O F   C O N F - F I L E #########################
nickname: isabella
Address: 12668885769632475474L

User avatar
punkrock
Lisker
Posts: 105
Joined: Thu Feb 25, 2016 6:41 pm
Location: Germany
Contact: Website Twitter

Re: HowTo: Easy monitoring and notification system for your node (MONIT + Gmail + Missed Notifications Reminder)

Wed Nov 23, 2016 4:32 pm

Updated!

Added interval notification with "Missed Notifications Reminder".
Delegate name: punkrock
Address: 6853061742992593192L


Daniel B. - Former Lisk ambassador for Germany
Website | My delegate proposal | Twitter | Bitcointalk

Return to “Guides”

Who is online

Users browsing this forum: No registered users and 1 guest