What is Statusengine? ¶
Statusengine consists of two parts: An event broker module providing event data and a PHP application that processes it.
The event broker is loaded into the Naemon core and dumps configuration and event data (for instance, state changes, service check results or notification data) into the Gearman queuing engine. All data is encoded as JSON objects to simplify application development.
The PHP application is based on the CakePHP framework and saves the data from the broker module into a MySQL database.
Architecture ¶
Why Statusengine? ¶
As you probably know there's another well known solution to connect Naemon with an MySQL database, so why should I use statusengine?
Let me show you a few advantages of Statusengine:
- Due to the legacy mode all the software that works with the well known other solution, will work with Statusengine out of the box!
- Statusengine provides you a full UTF-8 integration and has no problems with double byte characters
- It's written in PHP so you can realy easy develop patches or what ever you need
- All the code is open and you can contribute to the project via GitHub
- Statusengine is made for big environments and don't require strange kernel parameters
- Only the broker module is written in C (small C code eq less segfaults)
- You can uninstall or upgrade your MySQL server without stopping your monitoring
- Every language is able to deal with MySQL
- Statusengine is able to process your performance data as well
- Statusengine is more than just a simple Naemon to MySQL gateway. This is just one way to use this solution. The broker module fetches everything you need out of the Naemon core as a json string. You can start developing your own client that does what ever you want with this data.
How does Statusengine work? ¶
Statusengine is developed for Naemon and should work with Nagios 4 as well. There is no support for Nagios 3! If you want to migrate to Naemon check out the official migration guide.
The Statusengine broker module statusengine.o gets loaded by Naemon and is the main part of Statusengine. Every event inside of Naemon will trigger the callback functions inside of statusengine.o. The data gets processed by statusengine.o, json encoded and written to the Gearman Job Server.
Inside of the Gearman Job Server are a lot of queues like statusngin_servicestatus, statusngin_servicechecks or statusngin_statechanges. Every event gets handled by its own queue.On the other side the StatusengineLegacyShell reads the data out of the job server and process it with different worker processes. Every worker handles one or more queues and writes the data into the database. This part is written in PHP so a lot of people can modify this easily if needed.
Don't worry, PHP is fast enough to get the job done :-)
Due to the Gearman Job Server you can uninstall, upgrade, distroy or whatever you want to do with your MySQL server and your monitoring will still work, as long as the Gearman Job Server is running!
I need an event broker but no database ¶
Statusengine is hackable!
If you are interested in row naemon event data but you don't want to save the data into a database you can simply develop your own worker. The Gearman library supports a lot of different programming languages.
Just load statusenigne.o to your naemon core. The communication is based on standard protocols so all the data is json encoded.
For example, you want to save all your monitored IP addresses in a text file, Statusengine will save your day like this example shows.Advanced installation PHP5¶
This part is may be interesting for you if you want to install Statusengine on a different OS than Ubuntu 14.04.
PHP 7 ready!
Statusengine 2.x is tested on PHP 5 and PHP 7
Take a look at the PHP 7 guide for more information
Looking for other Ubuntu or Debian version?
Packages you need to install (please try to use your package manager as often as possible):
- - gearman-job-server
- - libgearman-dev
- - gearman-tools
- - uuid-dev
- - php5-gearman
- - php5
- - php5-cli
- - php5-dev
- - libjson-c-dev
- - manpages-dev
- - build-essential
Gearman stuff not in your repository? No problem, install it by hand:
Download the gearmand-1.x.x.tar.gz from https://launchpad.net/gearmand and extract it.
cd gearmand-1.x.x/
./configure
make
make install
cd..
Download the gearman php extension from http://pecl.php.net/package/gearman and extract it.
cd gearman-1.x.x/
phpize
./configure
make
make install
cd..
After this you can load the PHP extension in your /etc/php5/cli/php.ini:
extension="gearman.so"
Compile Statusengine from source:
Check out the repository from https://github.com/nook24/statusengine and unzip it.
git clone https://github.com/nook24/statusengine.git
cd statusengine/
mkdir -p /opt/statusengine
cd statusengine/src
LANG=C gcc -shared -o statusengine.o -fPIC -Wall -Werror statusengine.c -luuid -levent -lgearman -ljson-c -DNAEMON;
cp statusengine.o /opt/statusengine/
cd ../../
cp -r cakephp /opt/statusengine/
Continue with the installation or migration guide.
Supported operating systems ¶
Basically you can install Statusengine on any Linux OS. Read the Advanced installation guide for more information.
Supported operating systems for Statusengine 1.x
- - Ubuntu 14.04 LTS
Supported operating systems for Statusengine 2.x
- Ubuntu (recommended)
- - 14.04 LTS
- - 16.04 LTS
- Debian
- - 7 (Wheezy)
- - 8 (Jessie)
- - Raspbian (Jessie only)
What is "Legacy mode"? ¶
If you are searching for a way to dump data out of Naemon to a MySQL database you will see that there is only one common way/application to do this.
The "Legacy mode" of Statusengine acts exactly the same way like this common solution.
With "Legacy mode" Statuengine is compatible to tools like NagVis, openITCOCKPIT and other NDO based applications
Make it scale! ¶
If you run a lot of checks on your system, you maybe see that some queues will grow up, if you run gearadmin --status
Statusengine is able to handle this by simplay add more worker processes to the queues in the file cakephp/app/Config/Command/Statusengine.php
Notice:
1. Please take care of your storage system, if you increase the number of workers!
2. The queue statusngin_objects can only be handled by one worker process!
Please keep in mind, MySQL servers love multithreaded applications :-)
<?php
'workers' => [
[
'queues' => ['statusngin_servicestatus' => 'processServicestatus']
],
//You can simple add more workers if you servicechecks queue is growing
//Remember: MySQL love multithreaded applications :)
/*
[
'queues' => ['statusngin_servicestatus' => 'processServicestatus']
],
[
'queues' => ['statusngin_servicestatus' => 'processServicestatus']
],
[
'queues' => ['statusngin_servicestatus' => 'processServicestatus']
],
*/
[
'queues' => [
'statusngin_hoststatus' => 'processHoststatus',
'statusngin_statechanges' => 'processStatechanges'
]
],
[
'queues' => ['statusngin_servicechecks' => 'processServicechecks']
],
//You can simple add more workers if you servicechecks queue is growing
//Remember: MySQL love multithreaded applications :)
/*
[
'queues' => ['statusngin_servicechecks' => 'processServicechecks']
],
[
'queues' => ['statusngin_servicechecks' => 'processServicechecks']
],
[
'queues' => ['statusngin_servicechecks' => 'processServicechecks']
],
*/
[
'queues' => [
'statusngin_hostchecks' => 'processHostchecks',
'statusngin_logentries' => 'processLogentries'
]
],
[
'queues' => [
'statusngin_notifications' => 'processNotifications',
'statusngin_contactstatus' => 'processContactstatus',
'statusngin_contactnotificationdata' => 'processContactnotificationdata',
'statusngin_contactnotificationmethod' => 'processContactnotificationmethod',
'statusngin_acknowledgements' => 'processAcknowledgements',
'statusngin_comments' => 'processComments',
'statusngin_flappings' => 'processFlappings',
'statusngin_downtimes' => 'processDowntimes',
'statusngin_externalcommands' => 'processExternalcommands',
'statusngin_systemcommands' => 'processSystemcommands',
'statusngin_eventhandler' => 'processEventhandler'
]
]
],
Broker Options ¶
By default the Statusengine Event Broker Module will write every event data to the Gearman Job Server.
Normally there is always data you will not need for example 'statusngin_externalcommands' or 'statusngin_systemcommands'
On large installations this unnecessary data will slow down your database and increase your disk I/O, so you can disable data using the broker options.
To use one or more broker options, just add them in your monitoring config:
broker_module=/opt/statusengine/statusengine-naemon.o use_external_command_data=0 use_system_command_data=1
This is list of all available broker options:
Gearman Job Server (Too many open files) ¶
If you are monitoring large environments you may be use Mod_Gearman to execute checks using multiple worker nodes.
However, unfortunately I had the problem that the gearman job server eats 100% of my CPU if more than 450 workers tried to connect to it and gearadmin --status
stuck and did not return information anymore. In the log file /var/log/gearman-job-server/gearman.log
I found the following message:
ERROR 2015-04-14 22:02:54.000000 [ main ] accept(Too many open files) -> libgearman-server/gearmand.cc:788
By default the Linux kernel set a limit of 1024 open files which is bad for MySQL servers or the Gearman job server. So I have increased this value, and started the gearman job server again.
root@ubuntu-dev:~# ulimit -n 16384
root@ubuntu-dev:~# /usr/sbin/gearmand --log-file=/var/log/gearman-job-server/gearman.log
All the statusngin_ queues are - yes your right handled by Statusengine.
The queues service, host, dummy, eventhandler, hostgroup_$HOSTGROUPNAME$ and worker_$HOSTNAME$ are handled by Mod_Gearman.
root@ubuntu-dev:~# uptime
22:36:12 up 26 days, 6:12, 3 users, load average: 2,08, 2,26, 2,21
As you can see my system handles over 800 workers on the service queue easily now.
How to set the limit:
SysVinit (/etc/init.d/gearman-job-server start)
Edit the file /etc/init.d/gearman-job-server
like this:
# Description: Enable gearman job server
### END INIT INFO
ulimit -n 16384 # <--- Add this line
prefix=/usr
exec_prefix=${prefix}
And restart: /etc/init.d/gearman-job-server restart
Upstart (service gearman-job-server start)
Edit the file /etc/init/gearman-job-server.conf
like this:
respawn
limit nofile 16384 16384 # <--- Add this line
exec start-stop-daemon --start --chuid gearman --exec ...
And restart: service gearman-job-server restart
systemd (systemctl start gearman-job-server)
Edit the file /etc/systemd/system/multi-user.target.wants/gearman-job-server.service
like this:
PIDFile=/run/gearman/server.pid
LimitNOFILE=16384
ExecStart=/usr/sbin/gearmand --listen=127.0.0.1 ...
And restart:
systemctl daemon-reload
systemctl restart gearman-job-server
To make sure that the new limit is set, you should check the file /proc/$PID$/limits:
root@ubuntu-dev:~# cat /proc/2945/limits | grep -i 'max open files'
Max open files 16384 16384 files
Boot order ¶
Statusengine requires that your MySQL Server and Gearman-Job-Server is running, before you start it. Please make sure that the boot order is set correctly.
For the event broker module statusengine.o it is also required that Gearman-Job-Server is running, before you start your Naemon process!
PHP7 guide¶
This guide demonstrates how you can install Statusengine with PHP7.
Fair warning:
This guid was tested on Ubuntu 16.04 LTS!
Some PHP extensions are missing in the package manager and need to be compiled and installed manually. Due to this fact Statusengine may be not running as stable as on PHP5!
Let's start with installing basic PHP7 and Statusengine's dependencies
apt-get install mysql-server gearman-job-server libgearman-dev gearman-tools uuid-dev php-gearman php-cli php-dev libjson-c-dev manpages-dev build-essential libglib2.0-dev rrdtool php-rrd
If you want to use Statusengine Web Interface as well, you need to install Apache2
apt-get install apache2 libapache2-mod-php
Install PHP-Gearman extension for PHP7
apt-get install git libgearman-dev
git clone https://github.com/wcgallego/pecl-gearman.git
cd pecl-gearman/
phpize
./configure
make
make install
echo "extension=gearman.so" >> /etc/php/7.0/mods-available/gearman.ini
phpenmod -s ALL gearman
Install PHP-RRDtool extension for PHP7
apt-get install rrdtool php-rrd
ln -s /etc/php/mods-available/rrd.ini /etc/php/7.0/mods-available/rrd.ini
phpenmod -v ALL -s ALL rrd
Check if both extensions are loaded
php -m | grep 'gearman\|rrd'
Continue with the installation or migration guide.
Or continue with the installation guide + Statusengine Web Interfaceinstallation and skip step one.
How to update Statusengine to new Version? ¶
Create a backup of your current version
cp -r /opt/statusengine /opt/statusengine_backup
service statusengine stop
Download the new Statusengine version and untar it to /opt/statusengine
In the next step you sould diff your Configuration files from the Backup with the default config files in /opt/statusengine/cakephp/app/Config
- - Cronjob.php
- - Graphite.php
- - Interface.php
- - Perfdata.php
- - Statusengine.php
- - database.php
Update your database schema.
Truncate the tables *_hoststatus and *_servicestatus!
/opt/statusengine/cakephp/app/Console/cake schema update --plugin Legacy --file legacy_schema_innodb.php --connection legacy
Start Statusengine
service statusengine start
How to install Naemon Core ¶
If you never had installed Naemon manually, you maybe run into some issues.
This little how to will show you, how to install Naemon 1.0.3 on Ubuntu based systems (14.04/16.04 in this case).
This is a "copy pastable" guide ;)
For Naemon > 1.0.3 you also need to install the package libglib2.0-dev
All files will be placed in /opt/naemon
adduser --system naemon
mkdir -p /opt/naemon
apt-get install build-essential automake gperf help2man libtool
wget https://github.com/naemon/naemon-core/archive/v1.0.3.tar.gz
tar xfv v1.0.3.tar.gz
cd naemon-core-1.0.3/
./autogen.sh --prefix=/opt/naemon --with-naemon-user=naemon --with-naemon-group=www-data --with-pluginsdir=/opt/naemon/libexec
make all
make install
cp /opt/naemon/etc/init.d/naemon /etc/init.d/
chmod +x /etc/init.d/naemon
cd ../
mkdir -p /opt/naemon/var/
mkdir -p /opt/naemon/var/cache/naemon/checkresults
mkdir -p /opt/naemon/var/cache/naemon
chown naemon:www-data /opt/naemon/var -R
service naemon start (/etc/init.d/naemon start)
Installing Monitoring Plugins
The Monitoring Plugins are the basic plugins you should
install on your system. They provide checks like check_ping, check_disk and so on.
The Monitoring Plugins have a lot of requirements to compile all of the available plugins. In this case just a few basic plugins are enough.
If you want to compile all the Plugins, check the requirements.
All plugins will be installed to /opt/naemon/libexec/
wget https://github.com/monitoring-plugins/monitoring-plugins/archive/v2.1.2.tar.gz
tar xfv v2.1.2.tar.gz
cd monitoring-plugins-2.1.2/
./autogen.sh --prefix=/opt/naemon --with-nagios-user=naemon --with-nagios-group=www-data
make all
make install
cd ../
Alternatively you can install the monitoring plugins using your package manager.
Nagios, NDOUtils and the Nagios logo are trademarks, servicemarks, registered trademarks or registered servicemarks owned by Nagios Enterprises, LLC. All other trademarks, servicemarks, registered trademarks, and registered servicemarks are the property of their respective owner(s).
All other trademarks are property of their respective owners. Other product or company names mentioned may be trademarks or trade names of their respective owner.