You are reading the documentation for Statusengine 3.x - Switch to Version 2.x for old stable

Broker Module

C++ Broker Module

This documentation is for the new C++ Broker Module. It has several advantages over the old legacy plain C broker module such as support for Gearman and RabbitMQ or the ability of processing external commands.

The Statusengine Broker Module is a small C++ dynamic library (so) that gets loaded into your Naemon or Nagios Core.

It will grab all status information, encode them as JSON, and put them into RabbitMQ Queues or the Gearman Job Server. Furthermore it can read commands and check results from the Queues, so you can easily exchange status information between nodes, or as an alternative to run external commands.
Due to the queuing engine (RabbitMQ/Gearman) your Monitoring Core will not get blocked by an slow database or disk io issues.
It is highly recommended to run the Queue (RabbitMQ/Gearman) on the same node as the monitoring core to keep the latencies low

I would recommend you, to split your monitoring node if you reach 50.000 services. This highly depends on your hardware and check interval but just as a rough idea.
Depending of your monitoring configuration this can be a challenging task.
For large environments I would recommend to use a config generator (like openITCOCKPIT) or something similar.
Once you have done splitting your configuration, you can deploy a new node with Naemon or Nagios , RabbitMQ (or Gearman) as Queue and load the Statusengine Broker Module.
The Statusengine UI comes with support for multiple nodes, so you will see all monitored devices in one interface.

Naemon:

  • Naemon >= 1.0.8
Nagios:
  • Nagios >= 4.3.2
Usually the latest broker version only supports the latest Naemon/Nagios version.
Did not have a running installation yet? Check the tutorials for Naemon or Nagios

Please select your operating system and monitoring core first. If your operating system is not in the list, pick the operating system version that matches your operating system best. You can still install Statusengine on your system, even if it is not in the list.

  1. Load EPEL Repository and install CentOS dependencies
    yum install epel-release
    yum check-update
    
    yum group install "Development Tools"
  2. Load EPEL Repository and install CentOS dependencies
    yum install epel-release
    yum check-update
    
    yum group install "Development Tools"
    
    dnf --enablerepo=PowerTools install json-c-devel
    yum check-update
  3. Install dependencies
    {[{commands[selectedOs].dependencies}]} {[{commands[selectedOs].queueDep[selectedQueue]}]}
    {[{commands[selectedOs].pip}]}
  4. Start Gearman-Job-Server on system boot
    systemctl enable gearmand
    systemctl start gearmand
  5. Start RabbitMQ on system boot
    systemctl enable rabbitmq-server
    systemctl start rabbitmq-server
  6. Download and Install Statusengine Broker Module
    cd /tmp
    git clone https://github.com/statusengine/broker.git broker
    cd broker/
    
    export PKG_CONFIG_PATH=/opt/naemon/lib/pkgconfig/
    
    meson setup --buildtype=release build
    ninja -C build
    
    mkdir -p /opt/statusengine/lib
    cp build/src/libstatusengine.so /opt/statusengine/lib/libstatusengine.so
    

Create statusengine.toml configuration file

You can find an example configuration with comments working with the Statusengine worker here: https://github.com/statusengine/broker/blob/master/statusengine.toml

You should copy the file to /opt/statusengine/etc/statusengine.toml or somewhere else where Naemon/Nagios can read it.

Download example statusengine.toml

mkdir -p /opt/statusengine/etc
curl https://raw.githubusercontent.com/statusengine/broker/master/statusengine.toml > /opt/statusengine/etc/statusengine.toml

Configure Naemon/Nagios to load the broker

To activate the broker you have to add a configuration line to your naemon.cfg or nagios.cfg:
broker_module=/opt/statusengine/lib/libstatusengine.so /opt/statusengine/etc/statusengine.toml

Restart Naemon/Nagios

systemctl restart naemon

The default OCSP and OCHP allows you to run a command or script after a service or host check was executed.
This will highly affect the performance of your monitoring core in a bad manner!
To avoid this issue, Statusengine's "OCSP"/"OCHP" will create you a special Queue and store a copy of every host - and service check event.
You can consume the events inside of the queue with a little script to get the data you need.

Data Example:
This example show all fields, you can receive via the Statusengine OCHP and OCSP queues.


{
  "messages": [
    {
      "type": 701,
      "flags": 0,
      "attr": 0,
      "timestamp": 1614507120,
      "timestamp_usec": 41434,
      "servicecheck": {
        "host_name": "linksys-srw224p",
        "service_description": "PING",
        "command_line": "$USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5",
        "command_name": "check_ping!200.0,20%!600.0,60%",
        "output": "PING CRITICAL - Packet loss = 100%",
        "long_output": "",
        "perf_data": "rta=600.000000ms;200.000000;600.000000;0.000000 pl=100%;20;60;0",
        "check_type": 0,
        "current_attempt": 3,
        "max_attempts": 3,
        "state_type": 1,
        "state": 2,
        "timeout": 60,
        "start_time": 1614507110,
        "end_time": 1614507120,
        "early_timeout": 0,
        "execution_time": 10.004374,
        "latency": 0.03640900179743767,
        "return_code": 2
      }
    },
    {
      "type": 701,
      "flags": 0,
      "attr": 0,
      "timestamp": 1614507129,
      "timestamp_usec": 131330,
      "servicecheck": {
        "host_name": "localhost",
        "service_description": "PING",
        "command_line": "$USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5",
        "command_name": "check_ping!100.0,20%!500.0,60%",
        "output": "PING OK - Packet loss = 0%, RTA = 0.05 ms",
        "long_output": "",
        "perf_data": "rta=0.055000ms;100.000000;500.000000;0.000000 pl=0%;20;60;0",
        "check_type": 0,
        "current_attempt": 1,
        "max_attempts": 4,
        "state_type": 1,
        "state": 0,
        "timeout": 60,
        "start_time": 1614507125,
        "end_time": 1614507129,
        "early_timeout": 0,
        "execution_time": 4.093545,
        "latency": 0.037303999066352844,
        "return_code": 0
      }
    }
  ],
  "format": "none"
}

PHP Example Script:
The given PHP example script will fetch all jobs out of the queue and print the data to the shell.
Press STRG+C (^C) to exit.


<?php
$StatusengineOcsp = new StatusengineOcspProcessor();
$StatusengineOcsp->loop();

//Example Class
class StatusengineOcspProcessor{

    /**
     * @var \GearmanWorker
     */
    private $GearmanWorker;

    public function __construct(){
        //Create the GearmanWorker PHP Object
        $this->GearmanWorker = new \GearmanWorker();

        //Connect to the Gearman-Job-Server
        $this->GearmanWorker->addServer('127.0.0.1', 4730);

        //Consume data from the queue statusngin_ocsp and pass it
        //to the php method handleOcsp of the this class
        $this->GearmanWorker->addFunction('statusngin_ocsp', [$this, 'handleOcsp']);
    }

    public function loop(){
        //Start infinite loop to consume incoming jobs
        while(true){
            $this->GearmanWorker->work();
        }
    }

    public function handleOcsp($job){
        //Print service check data to stdout
        print_r(json_decode($job->workload()));
    }

}

Bash Example Script:

apt-get install jq
This example will print one job to the shell and exit
#!/bin/bash
gearman -w -c 1 -f statusngin_ocsp | jq .


More languages are available in the official Gearman documentation .

If you want to update to a new version of the Statusengine event broker, create a backup of your currently installed version first.

Stop your monitoring engine like: systemctl stop naemon.

Than repeat the installation steps.

To check if the Statusengine Broker Module save all events to the Gearman Job Server you can use the gearadmin --status command.
This will display you all existing queues, waiting jobs, active workers, and how much workers are connected to the queue.

#                              Jobs    Active   Worker
root@naemon:~# gearadmin --status
statusngin_contactstatus         2       0       0
statusngin_servicestatus        52       0       0
statusngin_hoststatus           10       0       0
statusngin_servicechecks         3       0       0
statusngin_ocsp                  3       0       0
statusngin_statechanges          0       0       0
statusngin_hostchecks            1       0       0
statusngin_logentries            4       0       0
In this example broker module put data into the queue but no process is consuming the provided data (0 connected workers).
So the broker modules works fine.

In addition, you can also run your monitoring core in foreground to debug issues or get more information about what's going on.

root@focal:~# sudo -u naemon /bin/bash
naemon@focal:/tmp/broker$ /opt/naemon/bin/naemon /opt/naemon/etc/naemon/naemon.cfg

Naemon Core 1.2.3
Copyright (c) 2013-present Naemon Core Development Team and Community Contributors
Copyright (c) 2009-2013 Nagios Core Development Team and Community Contributors
Copyright (c) 1999-2009 Ethan Galstad
License: GPL

Website: http://www.naemon.org
Naemon 1.2.3 starting... (PID=71902)
Local time is Sun Feb 28 10:15:14 UTC 2021
qh: Socket '/opt/naemon/var/naemon.qh' successfully initialized
nerd: Channel hostchecks registered successfully
nerd: Channel servicechecks registered successfully
nerd: Fully initialized and ready to rock!
Statusengine: Logstream initalized
Statusengine: the missing event broker
Statusengine: This is the c++ version of statusengine event broker
Statusengine: License:
    statusengine - the missing event broker
    Copyright (C) 2019  The statusengine team

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License along
    with this program; if not, write to the Free Software Foundation, Inc.,
    51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
Statusengine: Finished loading config
Statusengine: Gearman Clients: 1
Statusengine: Gearman 1 queue name: statusngin_hoststatus
Statusengine: Gearman 1 queue name: statusngin_hostchecks
Statusengine: Gearman 1 queue name: statusngin_servicestatus
Statusengine: Gearman 1 queue name: statusngin_servicechecks
Statusengine: Gearman 1 queue name: statusngin_service_perfdata
Statusengine: Gearman 1 queue name: statusngin_statechanges
Statusengine: Gearman 1 queue name: statusngin_acknowledgements
Statusengine: Gearman 1 queue name: statusngin_flappings
Statusengine: Gearman 1 queue name: statusngin_downtimes
Statusengine: Gearman 1 queue name: statusngin_contactnotificationmethod
Statusengine: Gearman 1 queue name: statusngin_core_restart
Statusengine: Gearman 1 queue name: statusngin_ocsp
Statusengine: Gearman 1 queue name: statusngin_ochp
Statusengine: Rabbitmq Clients: 0
Event broker module '/opt/statusengine/lib/libstatusengine.so' initialized successfully.
Successfully launched command file worker with pid 71907
^C
Caught 'Interrupt', shutting down...
Retention data successfully saved.
Successfully shutdown... (PID=71902)
Event broker module 'NERD' deinitialized successfully.
Event broker module '/opt/statusengine/lib/libstatusengine.so' deinitialized successfully.
Successfully reaped command worker (PID = 71907)
naemon@focal:/tmp/broker$