Monitor with Monit

A few days ago I posted a message about creating a pid file for linbpq. So that it becomes possible to monitor the process with watchdog.

You can read that here.

Now I use Monit to display this graphically. Why, because it looks nice. Now you immediately have an overview of running processes and their status.

The configuration file that I use for this.

/etc/monit/monitrc

## Start Monit in the background (run as a daemon):
#
  set daemon 300            # check services at 5-minute intervals
  set logfile /var/log/monit.log
  set pidfile /var/run/monit.pid
  set idfile /var/lib/monit/id
  set statefile /var/lib/monit/state

  set httpd port 2812
        use address 192.168.1.168
        allow 192.168.1.0/255.255.255.0
        allow admin:pancake

  set mailserver localhost               # primary mailserver

  set eventqueue
      basedir /var/lib/monit/events # set the base directory where events will be stored
      slots 100                     # optionally limit the queue size
  set alert pd9q@packet-radio.net                      # receive all alerts

  check system $HOST
    if loadavg (1min) > 4 then alert
    if loadavg (5min) > 2 then alert
    if cpu usage > 95% for 10 cycles then alert
    if memory usage > 75% then alert
    if swap usage > 25% then alert

  check process Linbpq with pidfile /home/pd9q/linbpq/run/linbpq.pid
  if changed pid then alert
  check file bpq32.cfg with path /home/pd9q/linbpq/bpq32.cfg

  check process Direwolf with pidfile /home/pd9q/linbpq/run/direwolf.pid
  if changed pid then alert
  check file direwolf.conf with path /home/pd9q/linbpq/direwolf.conf
 check host Router with address 192.168.1.1
    if failed ping then alert

  check network Eth0 with interface eth0
    if failed link then alert
    if changed link then alert
    if saturation > 90% then alert
    if download > 10 MB/s then alert
    if total uploaded > 1 GB in last hour then alert

check host Packet-radio.net with address packet-radio.net
      if failed icmp type echo count 5 with timeout 15 seconds
         then alert
      if failed port 80 proto http then alert
      if failed port 443 type TCPSSL proto http then alert
      alert pd9q@packet-radio.net

   include /etc/monit/conf.d/*
   include /etc/monit/conf-enabled/*

 

Watchdog

I use watchdog to monitor my RPI. Do services go offline or the RPI is not responding, watchdog kicks in and restart the system. Now I also want watchdog to watch my Linbpq. This is possible with watchdog, watchdog looks at a PID file. As far as I know, no PID file is created when starting Linbpq.

Now there is a possibility that when starting Linbpq with systemd a command can be given so that a PID file is created.

This is my linbpq.service file from systemd

pd9q@packet:~ $ sudo cat /etc/systemd/system/linbpq.service
[Unit]
Description=Linbpq Daemon
After=network.target
After=direwolf.target
StartLimitInterval=0

[Service]
Type=simple
Restart=always
RestartSec=5
ExecStartPost=/bin/sh -c "echo $MAINPID > /home/pd9q/linbpq/run/linbpq.pid"
ExecStart=/bin/bash /home/pd9q/linbpq/linbpq.start
WorkingDirectory=/home/pd9q/linbpq

[Install]
WantedBy=multi-user.target
Alias=linbpq.service

This command creates a linbpq.pid file, now it is possible that watchdog monitors linbpq.

ExecStartPost=/bin/sh -c "echo $MAINPID > /home/pd9q/linbpq/run/linbpq.pid"
pd9q@packet:~ $ sudo ls -l /home/pd9q/linbpq/run
-rw-r--r-- 1 root root 4 Oct 14 18:36 linbpq.pid
pd9q@packet:~ $

At the end of my watchdog configuration file (/etc/watchdog.conf) I have added the following line.

# Check if Linbpq is running
pidfile         = /home/pd9q/linbpq/run/linbpq.pid

Take a look at what happens when you stop Linbpq.

sudo systemctl stop linbpq.service

Yes the system is ReBoOtInG (I hoop)