Worker

Overview

Statusengine Worker is a PHP application that will consume the events provided by the Statusengine Broker Module.
All status data are save in a database (CrateDB, MySQL or Redis)
In addition, the Worker is able to parse and process performance data to store them in a time series databases like Graphite.

Scale out to Multiple Nodes

If you work load grows, and you split up your monitoring configuration through multiple monitoring nodes, you shoud deploy a new instance of Statusengine Worker on every monitoring node in your cluster.
Make sure to set a unique node_name for each Statusengine Worker in the cluster!
The default value of node_name is set to Crowbar.

Requirements

• php >= 7.0
• Redis Server
• php redis extension
• php gearman extension
• php composer

Installation

Please select your operating system first. If your operating system is not in the list, pick the operating system version that matches your operating system best. You can still install Statusengine on your system, even if it is not in the list.

1. Load EPEL Repository for PHP 7.2

   yum install epel-release
   yum install http://rpms.remirepo.net/enterprise/remi-release-7.rpm
   yum check-update
   yum install yum-utils
   yum-config-manager --enable remi-php72
   

2. Load EPEL Repository for PHP 7.3

   yum install epel-release
   yum install http://rpms.remirepo.net/enterprise/remi-release-8.rpm
   yum check-update
   
   dnf module reset php
   dnf module install php:remi-7.3
   

3. Install dependencies

   {[{commands[selectedOs].dependencies}]}

   php-gearman

   Unfortunately the php-gearman extension is missing or broken in Ubuntu 20.04 / Ubuntu 16.04 Follow this guide to install it manually

4. Start and enable Redis server

   systemctl start redis
   systemctl enable redis

5. Download and Install

   mkdir -p /opt/statusengine
   cd /opt/statusengine
   git clone https://github.com/statusengine/worker.git worker
   cd /opt/statusengine/worker
   chmod +x bin/*
   composer install
   

Database

At this point you need to decide, which database you prefer to use.

For large environments or high availability you should use CrateDB. For smaller systems or to keep things simple, you should go with MySQL.

CrateDB

MySQL

CREATE USER 'statusengine'@'localhost' IDENTIFIED BY 'password';
CREATE DATABASE IF NOT EXISTS `statusengine` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL PRIVILEGES ON `statusengine`.* TO 'statusengine'@'localhost';

# -p statusengine does not mean that "statusengine" is the password! This is the name of the database!
mysql -u statusengine -p statusengine < /opt/statusengine/worker/lib/mysql.sql

Redis

Configuration

Due to the fact that the Statusengine Worker Config file has a lot of options you may be don't need, I striped this down to the basics you need to change.

Statusengine Worker can also be configured through environment variables. This can be handy if you want to run Statusengine Worker inside of Docker. See the list of environment variables for more information.

If you are going to use CrateDB you can ignore all MySQL settings and vice vera.

Copy example configuration

cp /opt/statusengine/worker/etc/config.yml.example /opt/statusengine/worker/etc/config.yml

• node_name
  The node_name is used to route external commands to the corresponding monitoring node. It is requred, that the node_name is unique!
• use_mysql
  Set use_mysql to 1, to enable MySQL Backend.
• mysql username
  The user Statusengine will use to connect to the MySQL server
• mysql password
  The password of the given MySQL user
• mysql database
  Name of the MySQL database
• use_crate
  Set use_crate to 1, to enable CrateDB Backend.
• crate nodes
  An array of ip addresses of CrateDB instances. I recommend you to use a load balancer in front of your CrateDB cluster, and only add the address of the load balancer.
• external_command_file
  Set the right path to your external command file (nagios.cmd/naemon.cmd)
  Since Statusengine 3.7 the C++ Broker Module will be used by default to submit external commands to Naemon/Nagios

Load MySQL database schema

/opt/statusengine/worker/bin/Console.php database --update

Configure Performance Data

Statusengine Worker can store Nagios and Naemon Performance data to different storage backends.

• CrateDB
• MySQL
• Elasticsearch 5.x / Elasticsearch 6.x / Elasticsearch 7.x
• Graphite

Start Statusengine Worker through Systemd

cp /opt/statusengine/worker/lib/statusengine.service /lib/systemd/system/statusengine.service

crate.service

After=syslog.target network.target gearman-job-server.service crate.service

{[{mysqlShort}]}

{[{mysqlLong}]}

systemctl daemon-reload
systemctl enable statusengine
systemctl start statusengine

Start Statusengine Worker through SysVinit

cp /opt/statusengine/worker/lib/statusengine.init /etc/init.d/statusengine
chmod +x /etc/init.d/statusengine

crate

Required-Start: gearman-job-server crate

mysql

Required-Start: gearman-job-server mysql

update-rc.d statusengine defaults
service statusengine start

Automatically delete old data from database

The Statusengine Database cleanup cronjob should only run at one node of your cluster. You can run the cron on as many nodes as you want, but this will increase the load of the system.
If you want to run the cronjob on more than one node, you should set different times for scheduling the cron. For example at 01:00AM on node1 and at 01:00PM on node2.

echo "0 1 * * * /opt/statusengine/worker/bin/Console.php cleanup -q" > /etc/cron.d/statusengine

The parameter -q or --quiet is required, if the cron gets executed via crontab! Otherwise crond will send the output of the script to your mail addres - which could be annoying

For debugging purpose you can also execute the cronjob by your self.

Manage Cluster Nodes

If you start a new Statusengine Worker Node, the Node gets automatically added to the table statusengine_nodes.
This is one of the reasons, why the node_name needs to be unique.
How ever. If you want to remove a node from your cluster, you need to do this manually. For this task, you can use the command /opt/statusengine/worker/bin/Console.php cluster

To keep things automatable (e.g. using Ansible), you can pass all data as parameters, like in the following example.
Admins can run the command without the parameter --nodename to get into an interactive shell.

Get Metrics

Statusengine Worker provides a simple CLI tool, to get basic metrics about the worker performance. Execute the command /opt/statusengine/worker/bin/Console.php statistics to get a human friendly output

Example Grafana Dashboard

In-memory

Statusengine Worker is able to store live data to an Redis in-memory database.
Use cases:

• Notification scripts - Passing special characters, like ', ", &, etc... as parameter can be a challenging task
• Build a simple API on top
• Applications with heavy read operations

Statusengine Worker will save the following data to Redis:

• A list of hosts for every host state (hosts_up, hosts_down and hosts_unreachable)
• A list of services for every service state (services_ok, services_warning, services_critical and services_unknown)
• Host status object of every host (hoststatus_<hostname>)
• Service status object of every service (servicestatus_<hostname>_<service_description>)
• Internal Statusengine statistics (statusengine_statistics, worker_statistics_<pid>)

Statusengine always require an alternative storage backend to Redis (MySQL or CrateDB), for the cluster status and the external command router. To enable the Redis backend, set use_redis=1 in the Statusengine Worker config.yml.

A list of all Services in state "Ok"

Service status data of a particular service

Internal Statusengine statistics

How to update

Make sure you have installed the php extension bcmath. You can check if the extension is loaded using this command: php -m |grep bcmath.

If you had installed Statusengine through git, the update will be done quickly.

1. Stop Statusengine Worker

   {[{commands[selectedOs].stopStatusengineWorker}]}

2. Create backup of your files

   cp -r /opt/statusengine/worker /some/backup/path

3. Update your local files

   cd /opt/statusengine/worker
   git pull origin master
   composer update

4. Update your database (if required)
   MySQL Users:
   Statusengine will handle any schema updates automatically. When executed with --dry-run the update statements will only be print to the CLI but not executed.

   /opt/statusengine/worker/bin/Console.php database --update

   
   CrateDB Users:
   Check the path /opt/statusengine/worker/lib for any database update files.
   If an database schema update is available, the file name would be like crateDB_3.1.0_TO_3.2.0.sql for example.
   Execute every sql file in the right order!
   The current version of the database schema is stored in the table statusengine_dbversion.
5. Check for configuration changes
   You old configuration is still loaded. Statusengine Worker defines default values for each configuration variable.
   Nevertheless you should compare your current configuration with the new example configuration file for new interessing configuration options.

   vimdiff /opt/statusengine/worker/etc/config.yml /opt/statusengine/worker/etc/config.yml.example

6. Start Statusengine Worker

   {[{commands[selectedOs].startStatusengineWorker}]}

Update to 3.7

1. Stop Statusengine Worker

   {[{commands[selectedOs].stopStatusengineWorker}]}

2. Create backup of your files

   cp -r /opt/statusengine/worker /some/backup/path

3. Create backup of your database (Make sure you have enough disk space available)

   mysqldump -u statusengine -p --databases statusengine --flush-privileges --single-transaction --triggers --routines --no-tablespaces --events --hex-blob --ignore-table=statusengine.statusengine_perfdata > /opt/statusengine/statusengine_dump.sql

4. Update your local files

   cd /opt/statusengine/worker
   git pull origin master
   composer update

5. Check for configuration changes
   You old configuration is still loaded. Statusengine Worker defines default values for each configuration variable.
   Nevertheless you should compare your current configuration with the new example configuration file for new interessing configuration options.

   vimdiff /opt/statusengine/worker/etc/config.yml /opt/statusengine/worker/etc/config.yml.example

6. Add new primary keys to database
   This will take a while. running the command via tmux or screen is recommended
   

   mysql -u statusengine -p statusengine < /opt/statusengine/worker/lib/mysql_update.sql

7. Update database schema
   This will take a while. running the command via tmux or screen is recommended
   

   /opt/statusengine/worker/bin/Console.php database --update

8. Start Statusengine Worker

   {[{commands[selectedOs].startStatusengineWorker}]}

Debugging

For better debugging, you can run Statusengine in foreground. (Exit with STRG+C)