You are reading the documentation for Statusengine 3.x - Switch to Version 2.x for old stable

Worker

Statusengine Worker is a PHP application that will consume the events provided by the Statusengine Broker Module.
All status data are save in a database (CrateDB, MySQL or Redis)
In addition, the Worker is able to parse and process performance data to store them in a time series databases like Graphite.

If you work load grows, and you split up your monitoring configuration through multiple monitoring nodes, you shoud deploy a new instance of Statusengine Worker on every monitoring node in your cluster.
Make sure to set a unique node_name for each Statusengine Worker in the cluster!
The default value of node_name is set to Crowbar.

  • php >= 7.0
  • Redis Server
  • php redis extension
  • php gearman extension
  • php composer

Please select your operating system first. If your operating system is not in the list, pick the operating system version that matches your operating system best. You can still install Statusengine on your system, even if it is not in the list.

PHP Composer required

Check out the tutorial, of how to install php composer, if you don't have it installed to your system yet.

  1. Load EPEL Repository for PHP 7.2
    yum install epel-release
    yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
    yum check-update
    yum install yum-utils
    yum-config-manager --enable remi-php72
    
  2. Load EPEL Repository for PHP 7.3
    yum install epel-release
    yum install http://rpms.remirepo.net/enterprise/remi-release-8.rpm
    yum check-update
    
    dnf module reset php
    dnf module install php:remi-7.3
    
  3. Install dependencies
    {[{commands[selectedOs].dependencies}]}

    php-gearman

    Unfortunately the php-gearman extension is missing or broken in Ubuntu 20.04 / Ubuntu 16.04 Follow this guide to install it manually

  4. Start and enable Redis server
    systemctl start redis
    systemctl enable redis
  5. Download and Install
    mkdir -p /opt/statusengine
    cd /opt/statusengine
    git clone https://github.com/statusengine/worker.git worker
    cd /opt/statusengine/worker
    chmod +x bin/*
    composer install
    

At this point you need to decide, which database you prefer to use.

For large environments or high availability you should use CrateDB. For smaller systems or to keep things simple, you should go with MySQL.

CrateDB
Did not have a running CrateDB Cluster yet? Check the How To.
Import the file /opt/statusengine/worker/lib/crateDB.sql via the Crate Admin UI.
Notice: You need to run every sql query separately! Do NOT PASTE the whole file at once into the SQL Console
Unfortunately for now, CrateDB users have to manage the database schema manually due to crate/crate-dbal#92

With CrateDB, you don't have to create a separate database or user for Statusengine.

MySQL
Did not have a running MySQL server yet? Check the How To.
Create a MySQL database and user for Statusengine
CREATE USER 'statusengine'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE IF NOT EXISTS `statusengine` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON `statusengine`.* TO 'statusengine'@'localhost';
Since Statusengine 3.7 Statusengine will manage the database schema automatically
For Statusengine versions < 3.7 load the schema now. So only execute this if you are installing an old version
# -p statusengine does not mean that "statusengine" is the password! This is the name of the database!
mysql -u statusengine -p statusengine < /opt/statusengine/worker/lib/mysql.sql

Redis
Statusengine is also able to store live data, like host and service status to an Redis Server (in-memory database).
Read more about in the In-memory section.

Due to the fact that the Statusengine Worker Config file has a lot of options you may be don't need, I striped this down to the basics you need to change.

Statusengine Worker can also be configured through environment variables. This can be handy if you want to run Statusengine Worker inside of Docker. See the list of environment variables for more information.

If you are going to use CrateDB you can ignore all MySQL settings and vice vera.

Copy example configuration

cp /opt/statusengine/worker/etc/config.yml.example /opt/statusengine/worker/etc/config.yml

  • node_name
    The node_name is used to route external commands to the corresponding monitoring node. It is requred, that the node_name is unique!
  • use_mysql
    Set use_mysql to 1, to enable MySQL Backend.
  • mysql username
    The user Statusengine will use to connect to the MySQL server
  • mysql password
    The password of the given MySQL user
  • mysql database
    Name of the MySQL database
  • use_crate
    Set use_crate to 1, to enable CrateDB Backend.
  • crate nodes
    An array of ip addresses of CrateDB instances. I recommend you to use a load balancer in front of your CrateDB cluster, and only add the address of the load balancer.
  • external_command_file
    Set the right path to your external command file (nagios.cmd/naemon.cmd)
    Since Statusengine 3.7 the C++ Broker Module will be used by default to submit external commands to Naemon/Nagios

Load MySQL database schema

/opt/statusengine/worker/bin/Console.php database --update

Statusengine Worker can store Nagios and Naemon Performance data to different storage backends.

cp /opt/statusengine/worker/lib/statusengine.service /lib/systemd/system/statusengine.service

Pitfall!

If you use Statusengine with CrateDB, add

crate.service
to the After= line in the file /lib/systemd/system/statusengine.service
For Example:
After=syslog.target network.target gearman-job-server.service crate.service

If you use Statusengine with MySQL, add
{[{mysqlShort}]}
to the After= line in the file /lib/systemd/system/statusengine.service
For Example:
{[{mysqlLong}]}

systemctl daemon-reload
systemctl enable statusengine
systemctl start statusengine

cp /opt/statusengine/worker/lib/statusengine.init /etc/init.d/statusengine
chmod +x /etc/init.d/statusengine

Pitfall!

If you use Statusengine with CrateDB, add

crate
to the Required-Start line in the file /etc/init.d/statusengine
For Example:
Required-Start: gearman-job-server crate

If you use Statusengine with MySQL, add
mysql
to the equired-Start line in the file /etc/init.d/statusengine
For Example:
Required-Start: gearman-job-server mysql

update-rc.d statusengine defaults
service statusengine start

The Statusengine Database cleanup cronjob should only run at one node of your cluster. You can run the cron on as many nodes as you want, but this will increase the load of the system.
If you want to run the cronjob on more than one node, you should set different times for scheduling the cron. For example at 01:00AM on node1 and at 01:00PM on node2.

echo "0 1 * * * /opt/statusengine/worker/bin/Console.php cleanup -q" > /etc/cron.d/statusengine
You can define, how long Statusengine will keep your data in /opt/statusengine/worker/etc/config.yml

The parameter -q or --quiet is required, if the cron gets executed via crontab! Otherwise crond will send the output of the script to your mail addres - which could be annoying

For debugging purpose you can also execute the cronjob by your self.

Running Statusengine database cleanup cronjob

If you start a new Statusengine Worker Node, the Node gets automatically added to the table statusengine_nodes.
This is one of the reasons, why the node_name needs to be unique.
How ever. If you want to remove a node from your cluster, you need to do this manually. For this task, you can use the command /opt/statusengine/worker/bin/Console.php cluster

To keep things automatable (e.g. using Ansible), you can pass all data as parameters, like in the following example.
Admins can run the command without the parameter --nodename to get into an interactive shell.

Add or delete a cluster node from Statusengine

As alway --help is your friend

Statusengine Worker provides a simple CLI tool, to get basic metrics about the worker performance. Execute the command /opt/statusengine/worker/bin/Console.php statistics to get a human friendly output

Statusenigne Worker statistics

Run /opt/statusengine/worker/bin/Console.php statistics --watch 3 for automatically refresh every 3 seconds.
Run /opt/statusengine/worker/bin/Console.php statistics --naemon to change to output format to Naemon/Nagios performance data

Statusenigne Worker statistics as nagios or naemon output

Example Grafana Dashboard

Statusengine Performance Grafana Dashboard

As alway --help is your friend

Statusengine Worker is able to store live data to an Redis in-memory database.
Use cases:

  • Notification scripts - Passing special characters, like ', ", &, etc... as parameter can be a challenging task
  • Build a simple API on top
  • Applications with heavy read operations

Statusengine Worker will save the following data to Redis:

  • A list of hosts for every host state (hosts_up, hosts_down and hosts_unreachable)
  • A list of services for every service state (services_ok, services_warning, services_critical and services_unknown)
  • Host status object of every host (hoststatus_<hostname>)
  • Service status object of every service (servicestatus_<hostname>_<service_description>)
  • Internal Statusengine statistics (statusengine_statistics, worker_statistics_<pid>)

Statusengine always require an alternative storage backend to Redis (MySQL or CrateDB), for the cluster status and the external command router. To enable the Redis backend, set use_redis=1 in the Statusengine Worker config.yml.

A list of all Services in state "Ok"

Statusengine live service data in redis

Service status data of a particular service

Statusengine service status object in redis

Internal Statusengine statistics

Internal Statusengine statistics in redis

Updating from a version < 3.7 ?

Statusengine 3.7 contains major changes to the MySQL database schema. Please follow the Update to 3.7 instructions first.

Make sure you have installed the php extension bcmath. You can check if the extension is loaded using this command: php -m |grep bcmath.

If you had installed Statusengine through git, the update will be done quickly.

  1. Stop Statusengine Worker
    {[{commands[selectedOs].stopStatusengineWorker}]}
  2. Create backup of your files
    cp -r /opt/statusengine/worker /some/backup/path
  3. Update your local files
    cd /opt/statusengine/worker
    git pull origin master
    composer update
  4. Update your database (if required)
    MySQL Users:
    Statusengine will handle any schema updates automatically. When executed with --dry-run the update statements will only be print to the CLI but not executed.
    /opt/statusengine/worker/bin/Console.php database --update

    CrateDB Users:
    Check the path /opt/statusengine/worker/lib for any database update files.
    If an database schema update is available, the file name would be like crateDB_3.1.0_TO_3.2.0.sql for example.
    Execute every sql file in the right order!
    The current version of the database schema is stored in the table statusengine_dbversion.
  5. Check for configuration changes
    You old configuration is still loaded. Statusengine Worker defines default values for each configuration variable.
    Nevertheless you should compare your current configuration with the new example configuration file for new interessing configuration options.
    vimdiff /opt/statusengine/worker/etc/config.yml /opt/statusengine/worker/etc/config.yml.example
  6. Start Statusengine Worker
    {[{commands[selectedOs].startStatusengineWorker}]}

This update instructions only affects MySQL users. If you are using CrateDB as storage backend, please continue with the How to update guide. With Statusengine 3.7 new Primary Keys got added to the MySQL database schema. This speeds up select performance by a lot.
Unfortunately it could take a while to add the new columns and primary keys depending on the size of the database.
  1. Stop Statusengine Worker
    {[{commands[selectedOs].stopStatusengineWorker}]}
  2. Create backup of your files
    cp -r /opt/statusengine/worker /some/backup/path
  3. Create backup of your database (Make sure you have enough disk space available)
    mysqldump -u statusengine -p --databases statusengine --flush-privileges --single-transaction --triggers --routines --no-tablespaces --events --hex-blob --ignore-table=statusengine.statusengine_perfdata > /opt/statusengine/statusengine_dump.sql
  4. Update your local files
    cd /opt/statusengine/worker
    git pull origin master
    composer update
  5. Check for configuration changes
    You old configuration is still loaded. Statusengine Worker defines default values for each configuration variable.
    Nevertheless you should compare your current configuration with the new example configuration file for new interessing configuration options.
    vimdiff /opt/statusengine/worker/etc/config.yml /opt/statusengine/worker/etc/config.yml.example
  6. Add new primary keys to database
    This will take a while. running the command via tmux or screen is recommended
    mysql -u statusengine -p statusengine < /opt/statusengine/worker/lib/mysql_update.sql
  7. Update database schema
    This will take a while. running the command via tmux or screen is recommended
    /opt/statusengine/worker/bin/Console.php database --update
  8. Start Statusengine Worker
    {[{commands[selectedOs].startStatusengineWorker}]}

For better debugging, you can run Statusengine in foreground. (Exit with STRG+C)

Statusengine Worker running in foreground mode for debugging