The Statusengine Broker Module is a small C++ dynamic library (so) that gets loaded into
your Naemon or Nagios Core.
It will grab all status information, encode them as JSON,
and put them into RabbitMQ Queues or the Gearman Job Server. Furthermore it can read commands
and check results from the Queues, so you can easily exchange status information between nodes,
or as an alternative to run external commands.
Due to the queuing engine (RabbitMQ/Gearman) your Monitoring Core will not get blocked by an slow
database or disk io issues.
It is highly recommended to run the Queue (RabbitMQ/Gearman) on the same node as the monitoring core to keep the latencies low
I would recommend you, to split your monitoring node if you reach 50.000 services.
This highly depends on your hardware and check interval but just as a rough idea.
Depending of your monitoring configuration this can be a challenging task.
For large environments I would recommend to use a config generator (like openITCOCKPIT) or something similar.
Once you have done splitting your configuration, you can deploy a new node with Naemon or Nagios
, RabbitMQ (or Gearman) as Queue and load the Statusengine Broker Module.
The Statusengine UI comes with support for multiple nodes, so you will see all monitored devices in one interface.
Naemon:
Please select your operating system and monitoring core first. If your operating system is not in the list, pick the operating system version that matches your operating system best. You can still install Statusengine on your system, even if it is not in the list.
yum install epel-release yum check-update yum group install "Development Tools"
yum install epel-release yum check-update yum group install "Development Tools" dnf -y config-manager --set-enabled powertools dnf install json-c-develyum check-update
{[{commands[selectedOs].dependencies}]} {[{commands[selectedOs].queueDep[selectedQueue]}]}
{[{commands[selectedOs].pip}]}
systemctl enable gearmand systemctl start gearmand
systemctl enable rabbitmq-server systemctl start rabbitmq-server
cd /tmp git clone https://github.com/statusengine/broker.git broker cd broker/ export PKG_CONFIG_PATH=/opt/naemon/lib/pkgconfig/ meson setup --buildtype=release build ninja -C build mkdir -p /opt/statusengine/lib cp build/src/libstatusengine.so /opt/statusengine/lib/libstatusengine.so
Create statusengine.toml configuration file
You can find an example configuration with comments working with the Statusengine worker here: https://github.com/statusengine/broker/blob/master/statusengine.toml
You should copy the file to /opt/statusengine/etc/statusengine.toml or somewhere else where Naemon/Nagios can read it.
Download example statusengine.toml
mkdir -p /opt/statusengine/etc
curl https://raw.githubusercontent.com/statusengine/broker/master/statusengine.toml > /opt/statusengine/etc/statusengine.toml
Configure Naemon/Nagios to load the broker
To activate the broker you have to add a configuration line to yournaemon.cfg or nagios.cfg:
broker_module=/opt/statusengine/lib/libstatusengine.so /opt/statusengine/etc/statusengine.toml
Restart Naemon/Nagios
systemctl restart naemon
The default OCSP and OCHP allows you to run a command or script after a service or host check was executed.
This will highly affect the performance of your monitoring core in a bad manner!
To avoid this issue, Statusengine's "OCSP"/"OCHP" will create you a special Queue
and store a copy of every host - and service check event.
You can consume the events inside of the queue with a little script to get the data you need.
Data Example:
This example show all fields, you can receive via the Statusengine OCHP and OCSP queues.
{
"messages": [
{
"type": 701,
"flags": 0,
"attr": 0,
"timestamp": 1614507120,
"timestamp_usec": 41434,
"servicecheck": {
"host_name": "linksys-srw224p",
"service_description": "PING",
"command_line": "$USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5",
"command_name": "check_ping!200.0,20%!600.0,60%",
"output": "PING CRITICAL - Packet loss = 100%",
"long_output": "",
"perf_data": "rta=600.000000ms;200.000000;600.000000;0.000000 pl=100%;20;60;0",
"check_type": 0,
"current_attempt": 3,
"max_attempts": 3,
"state_type": 1,
"state": 2,
"timeout": 60,
"start_time": 1614507110,
"end_time": 1614507120,
"early_timeout": 0,
"execution_time": 10.004374,
"latency": 0.03640900179743767,
"return_code": 2
}
},
{
"type": 701,
"flags": 0,
"attr": 0,
"timestamp": 1614507129,
"timestamp_usec": 131330,
"servicecheck": {
"host_name": "localhost",
"service_description": "PING",
"command_line": "$USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5",
"command_name": "check_ping!100.0,20%!500.0,60%",
"output": "PING OK - Packet loss = 0%, RTA = 0.05 ms",
"long_output": "",
"perf_data": "rta=0.055000ms;100.000000;500.000000;0.000000 pl=0%;20;60;0",
"check_type": 0,
"current_attempt": 1,
"max_attempts": 4,
"state_type": 1,
"state": 0,
"timeout": 60,
"start_time": 1614507125,
"end_time": 1614507129,
"early_timeout": 0,
"execution_time": 4.093545,
"latency": 0.037303999066352844,
"return_code": 0
}
}
],
"format": "none"
}
PHP Example Script:
The given PHP example script will fetch all jobs out of the queue
and print the data to the shell.
Press STRG+C (^C) to exit.
<?php
$StatusengineOcsp = new StatusengineOcspProcessor();
$StatusengineOcsp->loop();
//Example Class
class StatusengineOcspProcessor{
/**
* @var \GearmanWorker
*/
private $GearmanWorker;
public function __construct(){
//Create the GearmanWorker PHP Object
$this->GearmanWorker = new \GearmanWorker();
//Connect to the Gearman-Job-Server
$this->GearmanWorker->addServer('127.0.0.1', 4730);
//Consume data from the queue statusngin_ocsp and pass it
//to the php method handleOcsp of the this class
$this->GearmanWorker->addFunction('statusngin_ocsp', [$this, 'handleOcsp']);
}
public function loop(){
//Start infinite loop to consume incoming jobs
while(true){
$this->GearmanWorker->work();
}
}
public function handleOcsp($job){
//Print service check data to stdout
print_r(json_decode($job->workload()));
}
}
Bash Example Script:
apt-get install jqThis example will print one job to the shell and exit
#!/bin/bash
gearman -w -c 1 -f statusngin_ocsp | jq .
More languages are available in
the official Gearman documentation
.
If you want to update to a new version of the Statusengine event broker, create a backup of your currently
installed version first.
Stop your monitoring engine like: systemctl stop naemon.
Than repeat the installation steps.
To check if the Statusengine Broker Module save all events to the Gearman Job Server
you can use the gearadmin --status command.
This will display you all existing queues, waiting jobs, active workers,
and how much workers are connected to the queue.
# Jobs Active Worker root@naemon:~# gearadmin --status statusngin_contactstatus 2 0 0 statusngin_servicestatus 52 0 0 statusngin_hoststatus 10 0 0 statusngin_servicechecks 3 0 0 statusngin_ocsp 3 0 0 statusngin_statechanges 0 0 0 statusngin_hostchecks 1 0 0 statusngin_logentries 4 0 0In this example broker module put data into the queue but no process is consuming the provided data (0 connected workers).
In addition, you can also run your monitoring core in foreground to debug issues or get more information about what's going on.
root@focal:~# sudo -u naemon /bin/bash
naemon@focal:/tmp/broker$ /opt/naemon/bin/naemon /opt/naemon/etc/naemon/naemon.cfg
Naemon Core 1.2.3
Copyright (c) 2013-present Naemon Core Development Team and Community Contributors
Copyright (c) 2009-2013 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
License: GPL
Website: http://www.naemon.org
Naemon 1.2.3 starting... (PID=71902)
Local time is Sun Feb 28 10:15:14 UTC 2021
qh: Socket '/opt/naemon/var/naemon.qh' successfully initialized
nerd: Channel hostchecks registered successfully
nerd: Channel servicechecks registered successfully
nerd: Fully initialized and ready to rock!
Statusengine: Logstream initalized
Statusengine: the missing event broker
Statusengine: This is the c++ version of statusengine event broker
Statusengine: License:
statusengine - the missing event broker
Copyright (C) 2019 The statusengine team
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License along
with this program; if not, write to the Free Software Foundation, Inc.,
51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Statusengine: Finished loading config
Statusengine: Gearman Clients: 1
Statusengine: Gearman 1 queue name: statusngin_hoststatus
Statusengine: Gearman 1 queue name: statusngin_hostchecks
Statusengine: Gearman 1 queue name: statusngin_servicestatus
Statusengine: Gearman 1 queue name: statusngin_servicechecks
Statusengine: Gearman 1 queue name: statusngin_service_perfdata
Statusengine: Gearman 1 queue name: statusngin_statechanges
Statusengine: Gearman 1 queue name: statusngin_acknowledgements
Statusengine: Gearman 1 queue name: statusngin_flappings
Statusengine: Gearman 1 queue name: statusngin_downtimes
Statusengine: Gearman 1 queue name: statusngin_contactnotificationmethod
Statusengine: Gearman 1 queue name: statusngin_core_restart
Statusengine: Gearman 1 queue name: statusngin_ocsp
Statusengine: Gearman 1 queue name: statusngin_ochp
Statusengine: Rabbitmq Clients: 0
Event broker module '/opt/statusengine/lib/libstatusengine.so' initialized successfully.
Successfully launched command file worker with pid 71907
^C
Caught 'Interrupt', shutting down...
Retention data successfully saved.
Successfully shutdown... (PID=71902)
Event broker module 'NERD' deinitialized successfully.
Event broker module '/opt/statusengine/lib/libstatusengine.so' deinitialized successfully.
Successfully reaped command worker (PID = 71907)
naemon@focal:/tmp/broker$