Documentation

Statusengine consists of two parts: An event broker module providing event data and a PHP application that processes it.

The event broker is loaded into the Naemon core and dumps configuration and event data (for instance, state changes, service check results or notification data) into the Gearman queuing engine. All data is encoded as JSON objects to simplify application development.

The PHP application is based on the CakePHP framework and saves the data from the broker module into a MySQL database.

As you probably know there's another well known solution to connect Naemon with an MySQL database, so why should I use statusengine?
Let me show you a few advantages of Statusengine:

• Due to the legacy mode all the software that works with the well known other solution, will work with Statusengine out of the box!
• Statusengine provides you a full UTF-8 integration and has no problems with double byte characters
• It's written in PHP so you can realy easy develop patches or what ever you need
• All the code is open and you can contribute to the project via GitHub
• Statusengine is made for big environments and don't require strange kernel parameters
• Only the broker module is written in C (small C code eq less segfaults)
• You can uninstall or upgrade your MySQL server without stopping your monitoring
• Every language is able to deal with MySQL
• Statusengine is able to process your performance data as well
• Statusengine is more than just a simple Naemon to MySQL gateway. This is just one way to use this solution. The broker module fetches everything you need out of the Naemon core as a json string. You can start developing your own client that does what ever you want with this data.

Statusengine is developed for Naemon and should work with Nagios 4 as well. There is no support for Nagios 3! If you want to migrate to Naemon check out the official migration guide.

The Statusengine broker module statusengine.o gets loaded by Naemon and is the main part of Statusengine. Every event inside of Naemon will trigger the callback functions inside of statusengine.o. The data gets processed by statusengine.o, json encoded and written to the Gearman Job Server.

On the other side the StatusengineLegacyShell reads the data out of the job server and process it with different worker processes. Every worker handles one or more queues and writes the data into the database. This part is written in PHP so a lot of people can modify this easily if needed.

Don't worry, PHP is fast enough to get the job done :-)

Due to the Gearman Job Server you can uninstall, upgrade, distroy or whatever you want to do with your MySQL server and your monitoring will still work, as long as the Gearman Job Server is running!

Statusengine is hackable!

If you are interested in row naemon event data but you don't want to save the data into a database you can simply develop your own worker. The Gearman library supports a lot of different programming languages.

Just load statusenigne.o to your naemon core. The communication is based on standard protocols so all the data is json encoded.

This part is may be interesting for you if you want to install Statusengine on a different OS than Ubuntu 14.04.

Packages you need to install (please try to use your package manager as often as possible):

- gearman-job-server

- libgearman-dev

- gearman-tools

- uuid-dev

- php5-gearman

- php5

- php5-cli

- php5-dev

- libjson-c-dev

- manpages-dev

- build-essential

Gearman stuff not in your repository? No problem, install it by hand:
Download the gearmand-1.x.x.tar.gz from https://launchpad.net/gearmand and extract it.

cd gearmand-1.x.x/
./configure
make
make install
cd..

Download the gearman php extension from http://pecl.php.net/package/gearman and extract it.

cd gearman-1.x.x/
phpize
./configure
make
make install
cd..

extension="gearman.so"

Compile Statusengine from source:
Check out the repository from https://github.com/nook24/statusengine and unzip it.

git clone https://github.com/nook24/statusengine.git
cd statusengine/
mkdir -p /opt/statusengine
cd statusengine/src
LANG=C gcc -shared -o statusengine.o -fPIC  -Wall -Werror statusengine.c -luuid -levent -lgearman -ljson-c -DNAEMON;
cp statusengine.o /opt/statusengine/
cd ../../
cp -r cakephp /opt/statusengine/

Basically you can install Statusengine on any Linux OS. Read the Advanced installation guide for more information.

Supported operating systems for Statusengine 1.x

- Ubuntu 14.04 LTS

Supported operating systems for Statusengine 2.x

Ubuntu (recommended)

- 14.04 LTS

- 16.04 LTS

Debian

- 7 (Wheezy)

- 8 (Jessie)

- Raspbian (Jessie only)

If you are searching for a way to dump data out of Naemon to a MySQL database you will see that there is only one common way/application to do this.

The "Legacy mode" of Statusengine acts exactly the same way like this common solution.

With "Legacy mode" Statuengine is compatible to tools like NagVis, openITCOCKPIT and other NDO based applications

If you run a lot of checks on your system, you maybe see that some queues will grow up, if you run gearadmin --status
Statusengine is able to handle this by simplay add more worker processes to the queues in the file cakephp/app/Config/Command/Statusengine.php

<?php
'workers' => [
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    //You can simple add more workers if you servicechecks queue is growing
    //Remember: MySQL love multithreaded applications :)
    /*
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    [
        'queues' => ['statusngin_servicestatus' => 'processServicestatus']
    ],
    */

    [
        'queues' => [
            'statusngin_hoststatus' => 'processHoststatus',
            'statusngin_statechanges' => 'processStatechanges'
        ]
    ],
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    //You can simple add more workers if you servicechecks queue is growing
    //Remember: MySQL love multithreaded applications :)
    /*
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    [
        'queues' => ['statusngin_servicechecks' => 'processServicechecks']
    ],
    */

    [
        'queues' => [
            'statusngin_hostchecks' => 'processHostchecks',
            'statusngin_logentries' => 'processLogentries'
        ]
    ],
    [
        'queues' => [
            'statusngin_notifications' => 'processNotifications',
            'statusngin_contactstatus' => 'processContactstatus',
            'statusngin_contactnotificationdata' => 'processContactnotificationdata',
            'statusngin_contactnotificationmethod' => 'processContactnotificationmethod',
            'statusngin_acknowledgements' => 'processAcknowledgements',
            'statusngin_comments' => 'processComments',
            'statusngin_flappings' => 'processFlappings',
            'statusngin_downtimes' => 'processDowntimes',
            'statusngin_externalcommands' => 'processExternalcommands',
            'statusngin_systemcommands' => 'processSystemcommands',
            'statusngin_eventhandler' => 'processEventhandler'
        ]
    ]
],

By default the Statusengine Event Broker Module will write every event data to the Gearman Job Server.

Normally there is always data you will not need for example 'statusngin_externalcommands' or 'statusngin_systemcommands'

On large installations this unnecessary data will slow down your database and increase your disk I/O, so you can disable data using the broker options.
To use one or more broker options, just add them in your monitoring config:

broker_module=/opt/statusengine/statusengine-naemon.o use_external_command_data=0 use_system_command_data=1

This is list of all available broker options:

If you are monitoring large environments you may be use Mod_Gearman to execute checks using multiple worker nodes.

However, unfortunately I had the problem that the gearman job server eats 100% of my CPU if more than 450 workers tried to connect to it and gearadmin --status stuck and did not return information anymore. In the log file /var/log/gearman-job-server/gearman.log I found the following message:
ERROR 2015-04-14 22:02:54.000000 [ main ] accept(Too many open files) -> libgearman-server/gearmand.cc:788
By default the Linux kernel set a limit of 1024 open files which is bad for MySQL servers or the Gearman job server. So I have increased this value, and started the gearman job server again.

root@ubuntu-dev:~# ulimit -n 16384
root@ubuntu-dev:~# /usr/sbin/gearmand --log-file=/var/log/gearman-job-server/gearman.log

All the statusngin_ queues are - yes your right handled by Statusengine.
The queues service, host, dummy, eventhandler, hostgroup_$HOSTGROUPNAME$ and worker_$HOSTNAME$ are handled by Mod_Gearman.

root@ubuntu-dev:~# uptime
22:36:12 up 26 days,  6:12,  3 users,  load average: 2,08, 2,26, 2,21

How to set the limit:

SysVinit (/etc/init.d/gearman-job-server start)
Edit the file /etc/init.d/gearman-job-server like this:

# Description:       Enable gearman job server
### END INIT INFO

ulimit -n 16384    # <--- Add this line

prefix=/usr
exec_prefix=${prefix}

Upstart (service gearman-job-server start)
Edit the file /etc/init/gearman-job-server.conf like this:

respawn

limit nofile 16384 16384 # <--- Add this line
exec start-stop-daemon --start --chuid gearman --exec ...

systemd (systemctl start gearman-job-server)
Edit the file /etc/systemd/system/multi-user.target.wants/gearman-job-server.service like this:

PIDFile=/run/gearman/server.pid

LimitNOFILE=16384

ExecStart=/usr/sbin/gearmand --listen=127.0.0.1 ...

systemctl daemon-reload
systemctl restart gearman-job-server

To make sure that the new limit is set, you should check the file /proc/$PID$/limits:

root@ubuntu-dev:~# cat /proc/2945/limits | grep -i 'max open files'
Max open files            16384                16384                files

Statusengine requires that your MySQL Server and Gearman-Job-Server is running, before you start it. Please make sure that the boot order is set correctly.

For the event broker module statusengine.o it is also required that Gearman-Job-Server is running, before you start your Naemon process!

This guide demonstrates how you can install Statusengine with PHP7.

Let's start with installing basic PHP7 and Statusengine's dependencies

apt-get install mysql-server gearman-job-server libgearman-dev gearman-tools uuid-dev php-gearman php-cli php-dev libjson-c-dev manpages-dev build-essential libglib2.0-dev rrdtool php-rrd

If you want to use Statusengine Web Interface as well, you need to install Apache2

apt-get install apache2 libapache2-mod-php

Install PHP-Gearman extension for PHP7

apt-get install git libgearman-dev

git clone https://github.com/wcgallego/pecl-gearman.git
cd pecl-gearman/
phpize
./configure
make
make install
echo "extension=gearman.so" >> /etc/php/7.0/mods-available/gearman.ini
phpenmod -s ALL gearman

Install PHP-RRDtool extension for PHP7

apt-get install rrdtool php-rrd

ln -s /etc/php/mods-available/rrd.ini /etc/php/7.0/mods-available/rrd.ini
phpenmod -v ALL -s ALL rrd

php -m | grep 'gearman\|rrd'

Create a backup of your current version

cp -r /opt/statusengine /opt/statusengine_backup
service statusengine stop

Download the new Statusengine version and untar it to /opt/statusengine
In the next step you sould diff your Configuration files from the Backup with the default config files in /opt/statusengine/cakephp/app/Config

- Cronjob.php

- Graphite.php

- Interface.php

- Perfdata.php

- Statusengine.php

- database.php

Update your database schema.
Truncate the tables *_hoststatus and *_servicestatus!

/opt/statusengine/cakephp/app/Console/cake schema update --plugin Legacy --file legacy_schema_innodb.php --connection legacy

Start Statusengine

service statusengine start

If you never had installed Naemon manually, you maybe run into some issues.

This little how to will show you, how to install Naemon 1.0.3 on Ubuntu based systems (14.04/16.04 in this case).
This is a "copy pastable" guide ;)
For Naemon > 1.0.3 you also need to install the package libglib2.0-dev
All files will be placed in /opt/naemon

adduser --system naemon
mkdir -p /opt/naemon

apt-get install build-essential automake gperf help2man libtool
wget https://github.com/naemon/naemon-core/archive/v1.0.3.tar.gz
tar xfv v1.0.3.tar.gz

cd naemon-core-1.0.3/
./autogen.sh --prefix=/opt/naemon --with-naemon-user=naemon --with-naemon-group=www-data --with-pluginsdir=/opt/naemon/libexec
make all
make install

cp /opt/naemon/etc/init.d/naemon /etc/init.d/
chmod +x /etc/init.d/naemon

cd ../

mkdir -p /opt/naemon/var/
mkdir -p /opt/naemon/var/cache/naemon/checkresults
mkdir -p /opt/naemon/var/cache/naemon
chown naemon:www-data /opt/naemon/var -R
service naemon start (/etc/init.d/naemon start)

Installing Monitoring Plugins

The Monitoring Plugins are the basic plugins you should install on your system. They provide checks like check_ping, check_disk and so on.
The Monitoring Plugins have a lot of requirements to compile all of the available plugins. In this case just a few basic plugins are enough.
If you want to compile all the Plugins, check the requirements.

All plugins will be installed to /opt/naemon/libexec/

wget https://github.com/monitoring-plugins/monitoring-plugins/archive/v2.1.2.tar.gz
tar xfv v2.1.2.tar.gz
cd monitoring-plugins-2.1.2/

./autogen.sh --prefix=/opt/naemon --with-nagios-user=naemon --with-nagios-group=www-data
make all
make install
cd ../

Alternatively you can install the monitoring plugins using your package manager.