A service’s check decides how each backend is tested, and its
check_options table configures that checker. Every check ends in up,
down or error (chapter 1).
check | Tests | Good for |
|---|---|---|
tcp | A TCP connection, optionally TLS and a send/expect exchange | Any TCP service; greeting-based protocols such as SMTP |
http | One GET request and its status code | Web services and health endpoints |
exec | A program run once per check | Existing Nagios-style scripts, infrequent checks |
| a plugin name | A long-running plugin program | redis, MySQL, PostgreSQL, LDAP, DNS, RADIUS, SIP, Diameter, SMTP, AMQP, MongoDB, gRPC, QUIC and HTTP/3 (emfor-check), or your own |
A check that has not finished within the service’s timeout is down
with reason timeout. The one exception is a plugin: a plugin that does not
answer in time gives error, because the plugin, not the backend, is late.
tcp
Without options, a backend is up when it accepts a connection; emfor
closes the connection at once. A refused or unreachable connection is
down, also when this host has no route to the backend or its own
firewall blocks the connection: the applications on this host could not
reach the backend either. Only a shortage on this host (open files,
buffers, memory, local ports) makes it error.
| Option | Type | Default | Meaning |
|---|---|---|---|
send | string | none | Bytes written once after connecting. TOML escapes such as \r\n are sent as those bytes. Needs expect. |
expect | string | none | Bytes that must appear, case-sensitively, in what the backend sends (read up to 16384 bytes). At most 16384 bytes. |
tls | boolean | false | Do a TLS handshake right after connecting; never falls back to plaintext. |
tls_verify | boolean | true | false accepts any certificate (still encrypted). Needs tls = true. |
server_name | string | the backend IP | Name (or IP) the certificate must match; sent as SNI when it is a name. Needs tls = true. |
ca_file | absolute path | system CAs | PEM file of CA certificates to trust instead. When the config is loaded it must exist and hold at least one certificate usable as a CA. Needs tls = true. |
[services."smtp.svc.emfor"]
check = "tcp"
backends = ["10.0.6.11:25", "10.0.6.12:25"]
check_options = { expect = "220" }
[services."memcache.svc.emfor"]
check = "tcp"
backends = ["10.0.7.11:11211"]
check_options = { send = "stats\r\n", expect = "STAT pid" }
[services."imap.svc.emfor"]
check = "tcp"
backends = ["10.0.8.11:993"]
check_options = { tls = true, server_name = "imap.internal", expect = "* OK" }
Typical reasons: expect: got "421 too busy\r\n" (the first 64 bytes
received, escaped), send: ..., tls: ...,
tls: no answer to the TLS handshake (does the port speak TLS?), and
tls: connection closed during the TLS handshake (does the server have a certificate?) when the server closes the connection instead of
answering, as Active Directory does on port 636 without a certificate.
http
One GET per check to the backend’s IP and port. Redirects are not
followed.
| Option | Type | Default | Meaning |
|---|---|---|---|
scheme | "http" or "https" | "http" | Protocol. |
path | string starting with / | "/" | Request path. #, \ and control characters are refused, because the request URL would cut or change the path there; a space or other special character is sent percent-encoded. |
host_header | string | none | Host header to send: a host name or IP address with an optional :port, IPv6 in brackets (api.internal, api.internal:8443, [2001:db8::1]). The connection still goes to the backend; with https a host name is also the TLS server name. |
expect_status | non-empty list | ["200-399"] | Status codes that count as up: codes (200 or "200") or inclusive ranges ("200-299"), all within 100–599. |
tls_verify | boolean | true | For https: false accepts any certificate. |
expect_body | string | none | Bytes that must appear, case-sensitively, in the body (read up to 65536 bytes). At most 16384 bytes. |
[services."api.svc.emfor".check_options]
path = "/healthz"
host_header = "api.internal"
expect_status = ["200-299"]
expect_body = '"status":"UP"'
A response with another status is down with the status code in the
reason, and its body is not read. With expect_body, a response whose
status matches is read until the text is found, the body ends, or 65536
bytes have been read; without the text it is down with
expect_body: got "<the first 64 bytes, escaped>", and a body that cannot
be read is down with a reason starting body: . Use it for health
endpoints that answer 200 either way and put the state in the body. The
latency fastest uses then includes reading the body.
exec
Runs a program once per check, without a shell. The backend is passed in
the environment variables EMFOR_IP, EMFOR_PORT and EMFOR_SERVICE.
[services."report.svc.emfor"]
check = "exec"
backends = ["10.0.3.1:5432"]
interval = "10s"
timeout = "5s"
check_options = { command = ["/usr/local/libexec/check_pg_replication", "--max-lag", "60"] }
| Exit | Result |
|---|---|
| 0 | up |
| 1 or 2 | down; the reason is the first line of stdout, or exit code N |
| any other code, killed by a signal, or failed to start | error |
still running at timeout | down (timeout); the program and all its children are killed |
command is the only option: a non-empty list of strings without NUL whose first element is an
absolute path to an executable. To run an existing Nagios plugin, which wants
-H and -p on its command line, see
Watching emfor from Nagios. Starting a process per check is costly; for
frequent checks of many backends, use a plugin.
Plugins and emfor-check
A plugin is a long-running program that the daemon starts once and sends check requests to, as JSON lines on its stdin and stdout. One process serves every service that uses it, and it can keep connections and state between checks.
emfor ships one plugin program, emfor-check. Its argument selects the
check:
[plugins."pgsql"]
command = ["/usr/local/libexec/emfor-check", "pgsql"]
[services."db.svc.emfor"]
check = "pgsql"
balance = "failover"
backends = ["10.0.3.1:5432", "10.0.3.2:5432"]
check_options = { user = "emfor", database = "postgres", role = "primary", password_file = "/etc/emfor/pg.pass" }
| Argument | Up when |
|---|---|
redis | PING answers PONG (after AUTH when a password is set) |
mysql | The login succeeds and SELECT 1 returns 1 (tested with MariaDB, and MySQL 8.0 and 8.4) |
pgsql | The login succeeds and SELECT 1 returns 1 |
ldap | A simple bind succeeds |
dns | One query gets an expected response code (and records, with expect_answer) |
radius | A Status-Server (or Access-Request) gets an accept signed with the shared secret |
sip | An OPTIONS request gets a final status code in expect_status (default 200 to 499) |
diameter | A capabilities exchange (CER / CEA) and a device watchdog (DWR / DWA) succeed |
smtp | The greeting is 220 and EHLO answers 250, over STARTTLS or TLS when asked for |
amqp | An AMQP 0-9-1 login succeeds and the vhost opens |
mongodb | hello answers, the SCRAM-SHA-256 login succeeds when user is set, and the role matches |
grpc | The standard health service (grpc.health.v1.Health/Check) answers SERVING |
quic | A QUIC handshake over UDP completes with a verified certificate and an ALPN in alpn; with http3 = true, an HTTP/3 GET also gets a status in expect_status |
Main options: user, password_file (never the password itself; read on
every check), database (pgsql), role (master/replica for redis,
primary/replica for MySQL and PostgreSQL; primary lets failover
follow the primary, and replica also needs replication to be running;
primary/secondary for MongoDB),
max_lag (with role = "replica", the seconds a replica may be behind),
and tls, starttls (ldap), tls_verify, server_name, ca_file.
The dns plugin takes name, type, transport, recursion_desired,
expect_rcode and expect_answer instead, and the radius plugin
secret_file, request, application, user, password_file,
reject_is_up, nas_identifier and require_message_authenticator, and
the sip plugin transport, tls (with the TLS keys), uri, from and
expect_status, and the diameter plugin origin_host, origin_realm,
the application ids, supported_vendor_ids, host_ip_address and tls
(with the TLS keys and the client certificate tls_cert_file and
tls_key_file), and the smtp plugin helo, starttls and tls (with
the TLS keys), and the amqp plugin user, password_file, vhost and
tls (with the TLS keys), and the mongodb plugin user,
password_file, auth_source, role and tls (with the TLS keys), and the
grpc plugin service and tls (with the TLS keys), and the quic plugin
alpn, tls_verify, server_name, ca_file (always TLS, so no tls
key), http3, path, host_header, expect_status and expect_body.
docs/deploy/plugins.md lists every option and
reason.
emfor check-config and every reload start the plugins and have them
validate each service’s check_options, so mistakes show up before the
config is applied. Validation builds what a check builds: a ca_file must
hold a certificate usable as a CA, a tls_key_file must be the key of
tls_cert_file, and a value the protocol sends as a NUL-terminated or
NUL-separated string (the mysql and pgsql user, the pgsql
database and password, the amqp user and password) must not contain
NUL. A message that must fit a size the protocol sets is built once: the
amqp login must fit in the 4096-byte first frame, and the diameter
Capabilities-Exchange-Request in 16777215 bytes.
When a plugin exits, its pending checks become error (plugin exited),
and the daemon restarts it after 1 s, then 2 s, 4 s … up to 30 s; until
then its checks are error (plugin <name> unavailable). A crashing
plugin never marks backends down: an error does not count as a failed
check. But a backend whose checks have given nothing but errors for longer
than fall × interval goes back to unknown (see
chapter 1), so a plugin that keeps crashing leaves its
backends unknown rather than healthy. A check that panics in a plugin
built on the emfor-plugin crate is answered error with reason
check panicked: <message>, and the plugin keeps running.
Lines a plugin writes to stderr are logged at info, at most 100 at once
and 10 a second after that; the log says how many were dropped. Each
plugin runs in a process group of its own: when it exits or is stopped,
the processes it started are killed too.
To write your own plugin, see
docs/plugins/protocol-v1.md; the emfor-plugin
crate implements the protocol for Rust.
Scheduling
Each backend is checked on its own schedule, and never by two checks at once:
- The first check after a start or reload begins at a random moment within
min(interval, 200ms). - While the backend is
unknownand checks returnupordown, the next check followsmin(interval, 200ms)after the previous one ends. With the defaults (rise = 2), backends are known within about half a second. This fast pace lastsrise + fallchecks at most: a backend whose results keep changing so that neither count is reached staysunknownand is then checked everyinterval. - Otherwise the next check starts
intervalafter the previous one ends, shifted at random by up tojitter(10% by default) either way.
Because the interval is counted from the end of a check, a slow backend is
not checked more often than every interval.
Rise, fall and errors
up: the success counter goes up and the failure counter resets; atrisethe backend turnshealthy.down: the failure counter goes up and the success counter resets; atfallthe backend turnsunhealthy.error: no counter and no state changes. When a backend has had only errors for longer thanfall × interval, it goes back tounknown.
Time to detect a failure is about fall × interval (6 s with the
defaults); time to recover is about rise × interval (4 s). Every state
change is logged at info with the service, backend, old and new state,
and the reason.
Reasons
A down or error result carries a reason, shown by emfor status, in
logs and in the state history. Before it is stored, the reason is made
safe: tab, CR and LF become \t, \r and \n; other control characters
become \u{1b} and the like; and a reason longer than 512 characters is
cut and ends in ….
Latency
For every backend the daemon measures how long each up check takes and
keeps a smoothed value (each new sample weighs 5%). emfor status and
/metrics show it, and balance = "fastest" uses it. Choose a check that
resembles real traffic when you rely on it.