Documentation

What is Statusengine?

Statusengine consists of two parts: An event broker module providing event data and a PHP application that processes it.

The event broker is loaded into the Naemon core and dumps configuration and event data (for instance, state changes, service check results or notification data) into the Gearman queuing engine. All data is encoded as JSON objects to simplify application development.

The PHP application is based on the CakePHP framework and saves the data from the broker module into a MySQL database.

Architecture

Statusengine architecture

Why Statusengine?

As you probably know there's another well known solution to connect Naemon with an MySQL database, so why should I use statusengine?
Let me show you a few advantages of Statusengine:

  • Due to the legacy mode all the software that works with the well known other solution, will work with Statusengine out of the box!
  • Statusengine provides you a full UTF-8 integration and has no problems with double byte characters
  • It's written in PHP so you can realy easy develop patches or what ever you need
  • All the code is open and you can contribute to the project via GitHub
  • Statusengine is made for big environments and don't require strange kernel parameters
  • Only the broker module is written in C (small C code eq less segfaults)
  • You can uninstall or upgrade your MySQL server without stopping your monitoring
  • Every language is able to deal with MySQL
  • Statusengine is able to process your performance data as well
  • Statusengine is more than just a simple Naemon to MySQL gateway. This is just one way to use this solution. The broker module fetches everything you need out of the Naemon core as a json string. You can start developing your own client that does what ever you want with this data.

How does Statusengine work?

Statusengine is developed for Naemon and should work with Nagios 4 as well. There is no support for Nagios 3! If you want to migrate to Naemon check out the official migration guide.

The Statusengine broker module statusengine.o gets loaded by Naemon and is the main part of Statusengine. Every event inside of Naemon will trigger the callback functions inside of statusengine.o. The data gets processed by statusengine.o, json encoded and written to the Gearman Job Server.

Inside of the Gearman Job Server are a lot of queues like statusngin_servicestatus, statusngin_servicechecks or statusngin_statechanges. Every event gets handled by its own queue.

On the other side the StatusengineLegacyShell reads the data out of the job server and process it with different worker processes. Every worker handles one or more queues and writes the data into the database. This part is written in PHP so a lot of people can modify this easily if needed.

Don't worry, PHP is fast enough to get the job done :-)

Due to the Gearman Job Server you can uninstall, upgrade, distroy or whatever you want to do with your MySQL server and your monitoring will still work, as long as the Gearman Job Server is running!

I need an event broker but no database

Statusengine is hackable!

If you are interested in row naemon event data but you don't want to save the data into a database you can simply develop your own worker. The Gearman library supports a lot of different programming languages.

Just load statusenigne.o to your naemon core. The communication is based on standard protocols so all the data is json encoded.

For example, you want to save all your monitored IP addresses in a text file, Statusengine will save your day like this example shows.

Advanced installation PHP5

This part is may be interesting for you if you want to install Statusengine on a different OS than Ubuntu 14.04.

PHP 7 ready!

Statusengine 2.x is tested on PHP 5 and PHP 7

Take a look at the PHP 7 guide for more information

Looking for other Ubuntu or Debian version?

Go to supported operating systems

Packages you need to install (please try to use your package manager as often as possible):

- gearman-job-server
- libgearman-dev
- gearman-tools
- uuid-dev
- php5-gearman
- php5
- php5-cli
- php5-dev
- libjson-c-dev
- manpages-dev
- build-essential

Gearman stuff not in your repository? No problem, install it by hand:
Download the gearmand-1.x.x.tar.gz from https://launchpad.net/gearmand and extract it.


cd gearmand-1.x.x/
./configure
make
make install
cd..

Download the gearman php extension from http://pecl.php.net/package/gearman and extract it.


cd gearman-1.x.x/
phpize
./configure
make
make install
cd..

After this you can load the PHP extension in your /etc/php5/cli/php.ini:

extension="gearman.so"

Compile Statusengine from source:
Check out the repository from https://github.com/nook24/statusengine and unzip it.


git clone https://github.com/nook24/statusengine.git
cd statusengine/
mkdir -p /opt/statusengine
cd statusengine/src
LANG=C gcc -shared -o statusengine.o -fPIC  -Wall -Werror statusengine.c -luuid -levent -lgearman -ljson-c -DNAEMON;
cp statusengine.o /opt/statusengine/
cd ../../
cp -r cakephp /opt/statusengine/

Continue with the installation or migration guide.

Supported operating systems

Basically you can install Statusengine on any Linux OS. Read the Advanced installation guide for more information.

Supported operating systems for Statusengine 1.x

- Ubuntu 14.04 LTS

Supported operating systems for Statusengine 2.x

Ubuntu (recommended)
- 14.04 LTS
- 16.04 LTS
Debian
- 7 (Wheezy)
- 8 (Jessie)
- Raspbian (Jessie only)

What is "Legacy mode"?

If you are searching for a way to dump data out of Naemon to a MySQL database you will see that there is only one common way/application to do this.

The "Legacy mode" of Statusengine acts exactly the same way like this common solution.

With "Legacy mode" Statuengine is compatible to tools like NagVis, openITCOCKPIT and other NDO based applications

Make it scale!

If you run a lot of checks on your system, you maybe see that some queues will grow up, if you run gearadmin --status
Statusengine is able to handle this by simplay add more worker processes to the queues in the file cakephp/app/Config/Command/Statusengine.php

Notice:

1. Please take care of your storage system, if you increase the number of workers!

2. The queue statusngin_objects can only be handled by one worker process!

This short example show you, how you can increase the number of workers for the queue statusngin_servicechecks for example.
Please keep in mind, MySQL servers love multithreaded applications :-)


<?php
'workers' => [
    [
        
'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    
//You can simple add more workers if you servicechecks queue is growing
    //Remember: MySQL love multithreaded applications :)
    /*
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    */

    
[
        
'queues' => [
            
'statusngin_hoststatus' => 'processHoststatus',
            
'statusngin_statechanges' => 'processStatechanges'
        
]
    ],
    [
        
'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    
//You can simple add more workers if you servicechecks queue is growing
    //Remember: MySQL love multithreaded applications :)
    /*
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    */

    
[
        
'queues' => [
            
'statusngin_hostchecks' => 'processHostchecks',
            
'statusngin_logentries' => 'processLogentries'
        
]
    ],
    [
        
'queues' => [
            
'statusngin_notifications' => 'processNotifications',
            
'statusngin_contactstatus' => 'processContactstatus',
            
'statusngin_contactnotificationdata' => 'processContactnotificationdata',
            
'statusngin_contactnotificationmethod' => 'processContactnotificationmethod',
            
'statusngin_acknowledgements' => 'processAcknowledgements',
            
'statusngin_comments' => 'processComments',
            
'statusngin_flappings' => 'processFlappings',
            
'statusngin_downtimes' => 'processDowntimes',
            
'statusngin_externalcommands' => 'processExternalcommands',
            
'statusngin_systemcommands' => 'processSystemcommands',
            
'statusngin_eventhandler' => 'processEventhandler'
        
]
    ]
],

Broker Options

By default the Statusengine Event Broker Module will write every event data to the Gearman Job Server.

Normally there is always data you will not need for example 'statusngin_externalcommands' or 'statusngin_systemcommands'

On large installations this unnecessary data will slow down your database and increase your disk I/O, so you can disable data using the broker options.
To use one or more broker options, just add them in your monitoring config:


broker_module=/opt/statusengine/statusengine-naemon.o use_external_command_data=0 use_system_command_data=1

This is list of all available broker options:

Option name
Affected queue
Description
Recommended to disable
use_host_status_data
statusngin_hoststatus
Hoststatus table
No
use_service_status_data
statusngin_servicestatus
Servicestatus table
No
use_process_data
statusngin_processdata
Notice! There was a bug in Statusengine < 2.0.4. If disabled no objects will be dumped anymore! In later versions you can disable this option if you like
Yes
use_service_check_data
statusngin_servicechecks
Will update the servicechecks table and process performance data information (graphing)
Depends
use_host_check_data
statusngin_hostchecks
Will update the hostchecks table
Depends
use_state_change_data
statusngin_statechanges
If disabled, the table 'statehistory' for hosts and services will get no updates anymore
No
use_log_data
statusngin_logentries
Will update the table logentries
Depends
use_system_command_data
statusngin_systemcommands
Will update the table systemcommands
Yes
use_comment_data
statusngin_comments
If disabled, no comment records will be saved anymore.
No
use_external_command_data
statusngin_externalcommands
If disabled, no external commands will be saved to the database. Disable this if you receive a lot of passive checks.
Yes
use_acknowledgement_data
statusngin_acknowledgements
The message of acknowledgements will not be saved in the database anymore. Will not affect the 'problem_has_been_acknowledged' field in host-/servicestatus tables.
No
use_flapping_data
statusngin_flappings
The table flappinghistory will not be saved anymore. Will not affect the 'is_flapping' field in host-/servicestatus tables
No
use_downtime_data
statusngin_downtimes
Downtime information will be saved in the database anymore. Will not affect the 'scheduled_downtime_depth' field in host-/servicestatus tables.
No
use_notification_data
statusngin_notifications
Notification information will be saved in the database anymore.
No
use_program_status_data
statusngin_programmstatus
Information about the current running Naemon process. Will be updated every n seconds
Yes
use_contact_status_data
statusngin_contactstatus
Information about the last notifications for an contact
No
use_contact_notification_data
statusngin_contactnotificationdata
Notifications a contact received
No
use_contact_notification_method_data
statusngin_contactnotificationmethod
The method that was used to send the notification (notification command)
No
use_event_handler_data
statusngin_eventhandler
Table 'eventhandlers' will not be saved anymore.
Depends
use_object_data
statusngin_objects
If disabled, the objects table will not be saved anymore! Only disable this, if you know what you are doing or for headless OCHP/OCSP sytems!
NO!
enable_ochp
statusngin_ochp
Basically the same as statusngin_hostchecks. You can use this as alternative for the classic OCHP command
Depends
enable_ocsp
statusngin_ocsp
Basically the same as statusngin_servicechecks. You can use this as alternative for the classic OCSP command
Depends

Gearman Job Server (Too many open files)

If you are monitoring large environments you may be use Mod_Gearman to execute checks using multiple worker nodes.

However, unfortunately I had the problem that the gearman job server eats 100% of my CPU if more than 450 workers tried to connect to it and gearadmin --status stuck and did not return information anymore. In the log file /var/log/gearman-job-server/gearman.log I found the following message:
ERROR 2015-04-14 22:02:54.000000 [ main ] accept(Too many open files) -> libgearman-server/gearmand.cc:788
By default the Linux kernel set a limit of 1024 open files which is bad for MySQL servers or the Gearman job server. So I have increased this value, and started the gearman job server again.


root@ubuntu-dev:~# ulimit -n 16384
root@ubuntu-dev:~# /usr/sbin/gearmand --log-file=/var/log/gearman-job-server/gearman.log

Output of gearadmin --status on a large environment

All the statusngin_ queues are - yes your right handled by Statusengine.
The queues service, host, dummy, eventhandler, hostgroup_$HOSTGROUPNAME$ and worker_$HOSTNAME$ are handled by Mod_Gearman.


root@ubuntu-dev:~# uptime
22:36:12 up 26 days,  6:12,  3 users,  load average: 2,08, 2,26, 2,21

As you can see my system handles over 800 workers on the service queue easily now.

How to set the limit:

SysVinit (/etc/init.d/gearman-job-server start)
Edit the file /etc/init.d/gearman-job-server like this:


# Description:       Enable gearman job server
### END INIT INFO

ulimit -n 16384    # <--- Add this line

prefix=/usr
exec_prefix=${prefix}

And restart: /etc/init.d/gearman-job-server restart

Upstart (service gearman-job-server start)
Edit the file /etc/init/gearman-job-server.conf like this:


respawn

limit nofile 16384 16384 # <--- Add this line
exec start-stop-daemon --start --chuid gearman --exec ...

And restart: service gearman-job-server restart

systemd (systemctl start gearman-job-server)
Edit the file /etc/systemd/system/multi-user.target.wants/gearman-job-server.service like this:


PIDFile=/run/gearman/server.pid

LimitNOFILE=16384

ExecStart=/usr/sbin/gearmand --listen=127.0.0.1 ...

And restart:

systemctl daemon-reload
systemctl restart gearman-job-server

To make sure that the new limit is set, you should check the file /proc/$PID$/limits:


root@ubuntu-dev:~# cat /proc/2945/limits | grep -i 'max open files'
Max open files            16384                16384                files

Boot order

Statusengine requires that your MySQL Server and Gearman-Job-Server is running, before you start it. Please make sure that the boot order is set correctly.

For the event broker module statusengine.o it is also required that Gearman-Job-Server is running, before you start your Naemon process!

PHP7 guide

This guide demonstrates how you can install Statusengine with PHP7.

Fair warning:

This guid was tested on Ubuntu 16.04 LTS!

Some PHP extensions are missing in the package manager and need to be compiled and installed manually. Due to this fact Statusengine may be not running as stable as on PHP5!

Let's start with installing basic PHP7 and Statusengine's dependencies


apt-get install mysql-server gearman-job-server libgearman-dev gearman-tools uuid-dev php-gearman php-cli php-dev libjson-c-dev manpages-dev build-essential libglib2.0-dev rrdtool php-rrd

If you want to use Statusengine Web Interface as well, you need to install Apache2


apt-get install apache2 libapache2-mod-php

Install PHP-Gearman extension for PHP7


apt-get install git libgearman-dev

git clone https://github.com/wcgallego/pecl-gearman.git
cd pecl-gearman/
phpize
./configure
make
make install
echo "extension=gearman.so" >> /etc/php/7.0/mods-available/gearman.ini
phpenmod -s ALL gearman

Install PHP-RRDtool extension for PHP7


apt-get install rrdtool php-rrd

ln -s /etc/php/mods-available/rrd.ini /etc/php/7.0/mods-available/rrd.ini
phpenmod -v ALL -s ALL rrd

Check if both extensions are loaded

php -m | grep 'gearman\|rrd'

Continue with the installation or migration guide.
Or continue with the installation guide + Statusengine Web Interfaceinstallation and skip step one.

How to update Statusengine to new Version?

Create a backup of your current version


cp -r /opt/statusengine /opt/statusengine_backup
service statusengine stop

Download the new Statusengine version and untar it to /opt/statusengine
In the next step you sould diff your Configuration files from the Backup with the default config files in /opt/statusengine/cakephp/app/Config

- Cronjob.php
- Graphite.php
- Interface.php
- Perfdata.php
- Statusengine.php
- database.php

Update your database schema.
Truncate the tables *_hoststatus and *_servicestatus!


/opt/statusengine/cakephp/app/Console/cake schema update --plugin Legacy --file legacy_schema_innodb.php --connection legacy

Start Statusengine


service statusengine start

How to install Naemon Core

If you never had installed Naemon manually, you maybe run into some issues.

This little how to will show you, how to install Naemon 1.0.3 on Ubuntu based systems (14.04/16.04 in this case).
This is a "copy pastable" guide ;)
For Naemon > 1.0.3 you also need to install the package libglib2.0-dev
All files will be placed in /opt/naemon


adduser --system naemon
mkdir -p /opt/naemon

apt-get install build-essential automake gperf help2man libtool
wget https://github.com/naemon/naemon-core/archive/v1.0.3.tar.gz
tar xfv v1.0.3.tar.gz

cd naemon-core-1.0.3/
./autogen.sh --prefix=/opt/naemon --with-naemon-user=naemon --with-naemon-group=www-data --with-pluginsdir=/opt/naemon/libexec
make all
make install

cp /opt/naemon/etc/init.d/naemon /etc/init.d/
chmod +x /etc/init.d/naemon

cd ../

mkdir -p /opt/naemon/var/
mkdir -p /opt/naemon/var/cache/naemon/checkresults
mkdir -p /opt/naemon/var/cache/naemon
chown naemon:www-data /opt/naemon/var -R
service naemon start (/etc/init.d/naemon start)


Installing Monitoring Plugins

The Monitoring Plugins are the basic plugins you should install on your system. They provide checks like check_ping, check_disk and so on.
The Monitoring Plugins have a lot of requirements to compile all of the available plugins. In this case just a few basic plugins are enough.
If you want to compile all the Plugins, check the requirements.

All plugins will be installed to /opt/naemon/libexec/


wget https://github.com/monitoring-plugins/monitoring-plugins/archive/v2.1.2.tar.gz
tar xfv v2.1.2.tar.gz
cd monitoring-plugins-2.1.2/

./autogen.sh --prefix=/opt/naemon --with-nagios-user=naemon --with-nagios-group=www-data
make all
make install
cd ../

Alternatively you can install the monitoring plugins using your package manager.


Nagios, NDOUtils and the Nagios logo are trademarks, servicemarks, registered trademarks or registered servicemarks owned by Nagios Enterprises, LLC. All other trademarks, servicemarks, registered trademarks, and registered servicemarks are the property of their respective owner(s).
All other trademarks are property of their respective owners. Other product or company names mentioned may be trademarks or trade names of their respective owner.