Worker

The Statusengine Worker is the other half of the pipeline: it drains the queue that the broker module fills. It is written in Go and is the successor to the original PHP worker.

What it does

Requirements

There is no CrateDB, Elasticsearch, or Redis backend. If a guide tells you otherwise, it is describing the old PHP worker .

Build and install

git clone https://github.com/statusengine/statusengine-worker
cd statusengine-worker

make build

make build produces every binary in bin/, stamped with version information. A plain go build skips that stamping, so the resulting binary reports its version as dev:

$ ./bin/worker -version
statusengine-worker 1.4.0 (commit 8706d2f41f28, built 2026-08-19T17:33:16Z, go1.26.5)

Useful targets:

A full install

sudo useradd --system --no-create-home --shell /usr/sbin/nologin statusengine
sudo mkdir -p /etc/statusengine
sudo cp config.example.yaml /etc/statusengine/config.yml

sudo make install            # binaries into /usr/local/bin under their service names
sudo make install-systemd    # unit files, without enabling anything

sudo systemctl enable --now statusengine-worker
sudo systemctl enable --now statusengine-db-cleanup.timer

Configuration

Every setting can be given four ways, and they resolve in this order:

explicit CLI flag  >  environment variable  >  config file  >  built-in default

The easiest way to provide the configuration is via a config file, typically /etc/statusengine/config.yml. Please copy the example configuration ↗ to /etc/statusengine/config.yml.

The default configuration contains all available settings with a description. It is a good starting point for new users.

Configure through environment variables

The Statusengine 4 Worker can also be configured via environment variables. This is especially useful when running in a containerized setup. All variables start with the STATUSENGINE_ prefix. A full list of environment variables can be found in the source code ↗ .

STATUSENGINE_API_KEYS=key-one,key-two

Database

Statusengine 4 does currently not manage the database schema automatically. The schema must be loaded manually before starting the worker.

A plain SQL file is shipped with the worker: packaging/mysql_schema.sql.

Create the database and the user

CREATE DATABASE IF NOT EXISTS `statusengine` DEFAULT CHARACTER SET utf8mb4;
CREATE USER 'statusengine'@'localhost' IDENTIFIED BY 'a-password-of-your-own';
GRANT ALL PRIVILEGES ON `statusengine`.* TO 'statusengine'@'localhost';
FLUSH PRIVILEGES;

Load the schema

mysql -u statusengine -p statusengine < packaging/mysql_schema.sql

Point the worker at it

mysql_dsn: "statusengine:a-password-of-your-own@tcp(127.0.0.1:3306)/statusengine?parseTime=true"

The format is go-sql-driver/mysql’s ↗ , not a URL: user:password@tcp(host:port)/database. Keep parseTime=true.

GRANT SELECT, INSERT, UPDATE, DELETE, DROP ON `statusengine`.* TO 'statusengine'@'localhost';

Performance data

The Statusengine Worker will handle processing of service performance data. It supports MySQL and Graphite as backends for storing this data.

perfdata_route: mysql        # mysql | graphite | both

Into MySQL

mysql and both write one row per metric into statusengine_perfdata table. The age_perfdata option trims. 90 days unless you say otherwise.

Into Graphite

graphite and both ship metrics to a Carbon plaintext receiver, graphite_addr, 127.0.0.1:2003 by default. The path of each metric is four segments:

<graphite_prefix>.<hostname>.<service_description>.<label>
statusengine.web01.HTTP.time

graphite_prefix is statusengine by default.

Metrics are buffered and flushed when the batch fills or after 250 ms, whichever comes first:

Running as a systemd service

make install-systemd copies three units into /etc/systemd/system but does not enable anything. statusengine-worker.service, plus the statusengine-db-cleanup service and timer.

systemctl enable --now statusengine-worker

EnvironmentFile=-/etc/statusengine/worker.env

STATUSENGINE_API_KEYS=key-one,key-two
STATUSENGINE_API_COMMAND_KEYS=another-key

Data retention

Nothing in the worker deletes anything. A monitoring core producing a few thousand checks a minute fills statusengine_hostchecks and statusengine_servicechecks faster than anything else in the schema, and trimming them is the job of a second binary.

It goes by two names, which is worth knowing before you go looking for it: make build produces it as bin/db_cleanup, and make install puts it in /usr/local/bin as statusengine-db-cleanup — the name used below and in the systemd unit.

It reads the same configuration file as the worker. Each binary ignores the other’s keys, so retention is configured next to everything else:

statusengine-db-cleanup -config /etc/statusengine/config.yml

# straight out of a build tree, before installing:
./bin/db_cleanup -config /etc/statusengine/config.yml

make install-systemd ships a timer for it:

systemctl enable --now statusengine-db-cleanup.timer

Daily, with Persistent=true so a missed run is caught up rather than skipped. Retention that quietly stops running is noticed when the disk fills and RandomizedDelaySec=1h keeps it off the top of the hour.

How long to keep what

Every value is a number of days, and every key is optional. Omit one and its default below applies. The key names are the ones the PHP worker used, so an existing config.yml can be carried over value for value, including its convention:

0 disables cleanup of that table entirely. Only an explicit 0 switches a table off; a missing key falls back to the default.

age_hostchecks: 5
age_servicechecks: 5
age_host_statehistory: 365
age_service_statehistory: 365
age_perfdata: 0          # perfdata_route: graphite - let Graphite handle it

Deleting without hurting

Rows go in batches, each its own transaction, and two settings shape that.

cleanup_batch_size is the number of rows per DELETE, 5000 by default. Smaller batches hold locks for shorter, keep the undo log small and produce binlog events a replica can digest - at the cost of more round-trips.

cleanup_batch_pause is a duration between two batches of the same table, 0s by default. No pause deletes as fast as the database allows. Set 50ms or so if the cleanup has to share the database with live check results.

The tool stops cleanly between batches on SIGTERM, so it never has to be killed mid-statement. The unit allows it 300 seconds to do that, which matters for the first run against a database that has never been cleaned. That one can take a while, and it is the run most likely to be interrupted.

DELETE FROM statusengine_hostchecks
WHERE start_time < 1788869899
LIMIT 5000;

Monitoring the worker

The worker exports 30 Prometheus series on its own port, metrics_listen_addr, :9105 by default. That server has no authentication of its own! It is meant to be reached by a trusted scraper, which means keeping the port off any public network rather than putting a key on it.

Every series exists from the first scrape, before anything has happened, so an alerting rule never has to cope with a metric that is missing until the first event arrives. The one deliberate exception is statusengine_queue_connected, which appears once a consumer has actually connected: a pre-created 0 would claim an outage for every queue during startup.

The Worker API reference documents all metrics with the reasoning behind each. These are the ones worth an alert:

statusengine_db_available is 0. Bulk inserts are not reaching MySQL. Nothing is lost while this is zero. The batch is held and the backlog waits at the broker (queue), but nothing is draining either, so the catch-up afterwards takes as long as it takes. This also means that the queues are filling up which could eventually lead to Out of Memory (OOM) conditions if not addressed.

statusengine_queue_connected is 0 for a queue_name. That consumer has lost its connection. This is the only series that separates stopped from idle: both leave messages_received_total flat and jobs_in_flight at zero. On RabbitMQ one connection carries every queue, so all of them move together; on Gearman each queue has its own and they move independently.

statusengine_graphite_metrics_dropped_total is rising. Every increment is a metric that is lost. Graphite fails differently from MySQL on purpose: an unreachable Carbon is dropped rather than retried, because retrying would stall the database path or grow the buffer without bound.

statusengine_queue_downtime_updates_unmatched_total is rising. A downtime’s START or STOP found no row to update, so the ADD that should have created it never arrived and the event is lost - silently, because MySQL reports the UPDATE as successful. Expect a few right after a fresh installation and none afterwards.

statusengine_db_batch_retries_total is climbing steadily. Bulk inserts re-run after a deadlock or a lock wait timeout. The occasional one is harmless; a steady climb means another writer is contending for the same rows, in practice statusengine-db-cleanup running against a busy table.

statusengine_websocket_messages_dropped_total looks alarming and usually is not: it counts events a client was too slow to accept. That is a problem with that client, not with the worker, and the pipeline is deliberately unaffected — see When a client cannot keep up .

Is it keeping up?

Three series answer that together, and none of them is an alert on its own.

statusengine_queue_jobs_in_flight sitting at the consumer’s concurrency cap for a queue — 8 per queue on Gearman, 100 on RabbitMQ, both configurable — means the broker is feeding that queue faster than the pipeline drains it. That is working as designed: the surplus waits at the broker instead of accumulating in this process. Read it per queue. One queue at its cap while the others idle is normal during a backlog and does not mean the others are blocked.

statusengine_queue_handler_duration_seconds shows where that time goes. A handler blocks in the bulk-insert buffer once it is full, so this is where MySQL backpressure becomes visible from the ingestion side.

statusengine_db_batch_size_at_flush pinned at the configured batch size means flushes are triggered by the batch filling up rather than by the 250 ms ticker. This is the same saturation, seen from the database side.

WebSocket event stream

Everything the worker writes to MySQL it also broadcasts live on /ws. Nothing is stored for a client that is not connected, this is a stream, not a backlog, which makes it the right tool for a dashboard, a chat notifier or anything that wants to react to a state change.

The server listens on listen_addr, 127.0.0.1:8080 by default. The full protocol, with a captured example for every event topic, is in the Worker API reference ; what follows is what you need to run it.

Authentication is always on

There is no setting that turns it off. Leave api_keys empty and the worker generates a 256-bit key for that run and logs it:

WARN websocket: no API key configured, generated a random one for this run
     api_key=8f3c…  hint="set -api-keys/STATUSENGINE_API_KEYS (or api_keys in the config file) for a stable key"

That is a safety net, not a setup: the key changes on every restart, so any client that reconnects on its own needs a configured API key.

api_keys:
  - "a-long-random-string"

A client presents it as Authorization: Bearer <key> or X-Api-Key: <key>. There is a third way, ?api_key=<key> in the URL, and it exists only because a browser’s WebSocket constructor cannot set headers — a key in a URL ends up in proxy logs and browser history, so use a header everywhere else.

Topics

A topic is a queue name. Subscribe at connect time:

ws://host/ws?topics=statusngin_hoststatus,statusngin_servicestatus

or change it later by sending a control frame at any time:

{"subscribe": ["statusngin_logentries"]}
{"unsubscribe": ["statusngin_hoststatus"]}

Subscribing to nothing means subscribing to everything. A client that connects without ?topics= receives all topics the worker consumes, which on a large installation is a great deal of traffic. Only subscribe to the topics you actually need is the recommended approach.

Frame format

One frame per job, not per event:

{"topic": "statusngin_hoststatus", "payload": [ … ]}

payload is always an array, even for the four queues the broker never bulks — those send an array of one — so a client never has to branch on the shape. The array is one bulk message as the broker sent it, so a frame typically carries a hundred events rather than one.

When a client cannot keep up

The hub never blocks the pipeline to wait for a slow reader. It buffers 1024 frames inbound and 256 frames per client, and when a buffer is full the frame is dropped for that client. The database write and every other client are unaffected. Drops are counted per client, reported when it disconnects, and exported as Prometheus counters.

A client that drops is a client that is too slow, or a link that is too thin. Subscribing to fewer topics is usually the fix; reading the socket in a loop that does no work of its own is the other.

The /commands API

The queue runs both ways. /commands takes Naemon external commands over HTTP, publishes them to the statusngin_cmd queue, and the broker module applies them to the running core . This way a dashboard could acknowledge a problem or force a check without a shell on the monitoring host.

It is off unless you switch it on

Unlike /ws, an unconfigured key here means the endpoint is not served at all:

INFO command: endpoint disabled, no API key configured
     hint="set -command-api-keys/STATUSENGINE_API_COMMAND_KEYS to enable /commands"

Enabling it takes one key, and it is deliberately a different list from api_keys:

command_api_keys:
  - "another-long-random-string"
command_listen_addr: 127.0.0.1:8081

An api_keys entry grants reading the event stream. A command_api_keys entry grants controlling the monitoring core. Those are not the same privilege, so they are not the same key. Also the endpoint gets its own port, so exposing the stream on the network does not also expose the write endpoint.

Sending commands

The body is the broker’s own envelope, unchanged. The same JSON a client would publish to Gearman or RabbitMQ directly, so the four commands and their fields are the ones documented under Submitting data to the core : check_result, schedule_check, delete_downtime and raw.

curl -X POST http://127.0.0.1:8081/commands \
  -H "Authorization: Bearer another-long-random-string" \
  -H "Content-Type: application/json" \
  -d '{"Command": "raw", "Data": "SCHEDULE_FORCED_SVC_CHECK;localhost;PING;1700000000"}'

A bulk sends many at once and may mix types freely:

{"messages": [
  {"Command": "check_result", "Data": {"host_name": "localhost", "output": "OK"}},
  {"Command": "raw", "Data": "ENABLE_HOST_FLAP_DETECTION;localhost"}
]}

One request carries at most 1000 commands and 8 MiB of body. Both limits exist so a rejection names the actual problem rather than surfacing as a truncated read — but the number that matters in practice is much smaller: the broker applies a bulk inside the monitoring core’s event loop and cannot be interrupted part-way through one, so keep a bulk to about 50 .

What it refuses

Five external commands are rejected even with a valid key:

Notably absent are CHANGE_*_CHECK_COMMAND and CHANGE_*_EVENT_HANDLER, the obvious route to running arbitrary code. Naemon disables those internally, so there is nothing left to deny.

Development tooling

make build produces five more binaries beside the worker and the cleanup. make install does not install them — they stay in bin/ and they read their fixtures from .claude/specs/ in the repository, so they run out of a source checkout and nowhere else.

simulator

Replays the recorded queue payloads through the same Router and bulk inserters cmd/app wires up, at a rate you choose — so the batching behaviour can be watched against a real MySQL rather than reasoned about.

./bin/simulator -rate 5000 -duration 15s -mysql-dsn "user:pass@tcp(127.0.0.1:3306)/statusengine-dev"

It hands fixture payloads straight to the handlers and never touches a broker (queue). That is a mock only in the sense that Gearman and RabbitMQ are absent — the decode, persist and broadcast path is the production one.

gearman_publisher and rabbitmq_publisher

The same idea from the other side: synthetic events for one queue, published to a real broker (Gearman or RabbitMQ), so the whole ingestion path runs — broker, consumer, router, inserter — with the worker as a separate process. Start the worker against the same broker and watch it consume what these publish.

./bin/gearman_publisher -queue statusngin_hoststatus -count 1000 -server localhost:4730
./bin/rabbitmq_publisher -queue statusngin_hoststatus -count 1000 \
    -server amqp://statusengine:statusengine@127.0.0.1:5672/

Each queue’s wire format comes from its recorded payload, used as a template. The four queues the broker never bulks get one job per event; everything else is cloned into batches and submitted as one {"messages": […]} job per batch — the shape a real consumer expects.

-count 1 is the useful special case: a single event never reaches the batch threshold, so it can only be written by the 250 ms ticker. If one event lands in the database, the ticker works.

losstest

Proves — or disproves — that a worker stopped under load loses nothing. That is the property a graceful shutdown exists to provide, and the one that cannot be established by reading the code.

Every event gets its own hostname, lt-<run-id>-<seq>, which is also the first column of statusengine_hostchecks’ primary key. Two things follow, and both are the point: a missing sequence number is proof of a lost event, because nothing in the pipeline can collapse two of those rows into one; and a redelivered job collides on that key, so MySQL rejects the whole multi-row INSERT with error 1062 and the batch is dropped, which shows up as a contiguous gap. The second is not a measurement artefact either — it is exactly what a redelivery would do to production data.

Three commands with an interruption in the middle:

./bin/losstest -mode publish -run-id r1 -count 50000
# start the worker, let it chew through the backlog, SIGTERM it mid-run,
# then start it again and let it drain the rest
./bin/losstest -mode verify -run-id r1 -count 50000
./bin/losstest -mode cleanup -run-id r1

Publishing everything up front rather than trickling it in is deliberate: it leaves a backlog at the job server, which is both the realistic restart scenario and the one that exercises the interesting window — jobs handed over while the consumer is already shutting down.

Read the exit code, not just the output. verify exits 1 when anything is missing, so it can drive a script.

db_verifier

A shadow-testing tool from the rewrite. It connects to two databases — one fed by the legacy PHP worker, one by this Go worker, both consuming the same event stream — and diffs their most recent rows table by table, column by column, to show the two pipelines persist identical data.

./bin/db_verifier \
    -dsn-php "user:pass@tcp(127.0.0.1:3306)/statusengine_php" \
    -dsn-go  "user:pass@tcp(127.0.0.1:3306)/statusengine" \
    -limit 5000

It never writes to either database, which is what makes it safe to point at a live one. -tables narrows the comparison; the default covers the status, check, history, notification, acknowledgement and downtime tables.