4. Health checks

A service’s check decides how each backend is tested, and its check_options table configures that checker. Every check ends in up, down or error (chapter 1).

checkTestsGood for
tcpA TCP connection, optionally TLS and a send/expect exchangeAny TCP service; greeting-based protocols such as SMTP
httpOne GET request and its status codeWeb services and health endpoints
execA program run once per checkExisting Nagios-style scripts, infrequent checks
a plugin nameA long-running plugin programredis, MySQL, PostgreSQL, LDAP, DNS, RADIUS, SIP, Diameter, SMTP, AMQP, MongoDB, gRPC, QUIC and HTTP/3 (emfor-check), or your own

A check that has not finished within the service’s timeout is down with reason timeout. The one exception is a plugin: a plugin that does not answer in time gives error, because the plugin, not the backend, is late.

tcp

Without options, a backend is up when it accepts a connection; emfor closes the connection at once. A refused or unreachable connection is down, also when this host has no route to the backend or its own firewall blocks the connection: the applications on this host could not reach the backend either. Only a shortage on this host (open files, buffers, memory, local ports) makes it error.

OptionTypeDefaultMeaning
sendstringnoneBytes written once after connecting. TOML escapes such as \r\n are sent as those bytes. Needs expect.
expectstringnoneBytes that must appear, case-sensitively, in what the backend sends (read up to 16384 bytes). At most 16384 bytes.
tlsbooleanfalseDo a TLS handshake right after connecting; never falls back to plaintext.
tls_verifybooleantruefalse accepts any certificate (still encrypted). Needs tls = true.
server_namestringthe backend IPName (or IP) the certificate must match; sent as SNI when it is a name. Needs tls = true.
ca_fileabsolute pathsystem CAsPEM file of CA certificates to trust instead. When the config is loaded it must exist and hold at least one certificate usable as a CA. Needs tls = true.
[services."smtp.svc.emfor"]
check = "tcp"
backends = ["10.0.6.11:25", "10.0.6.12:25"]
check_options = { expect = "220" }

[services."memcache.svc.emfor"]
check = "tcp"
backends = ["10.0.7.11:11211"]
check_options = { send = "stats\r\n", expect = "STAT pid" }

[services."imap.svc.emfor"]
check = "tcp"
backends = ["10.0.8.11:993"]
check_options = { tls = true, server_name = "imap.internal", expect = "* OK" }

Typical reasons: expect: got "421 too busy\r\n" (the first 64 bytes received, escaped), send: ..., tls: ..., tls: no answer to the TLS handshake (does the port speak TLS?), and tls: connection closed during the TLS handshake (does the server have a certificate?) when the server closes the connection instead of answering, as Active Directory does on port 636 without a certificate.

http

One GET per check to the backend’s IP and port. Redirects are not followed.

OptionTypeDefaultMeaning
scheme"http" or "https""http"Protocol.
pathstring starting with /"/"Request path. #, \ and control characters are refused, because the request URL would cut or change the path there; a space or other special character is sent percent-encoded.
host_headerstringnoneHost header to send: a host name or IP address with an optional :port, IPv6 in brackets (api.internal, api.internal:8443, [2001:db8::1]). The connection still goes to the backend; with https a host name is also the TLS server name.
expect_statusnon-empty list["200-399"]Status codes that count as up: codes (200 or "200") or inclusive ranges ("200-299"), all within 100–599.
tls_verifybooleantrueFor https: false accepts any certificate.
expect_bodystringnoneBytes that must appear, case-sensitively, in the body (read up to 65536 bytes). At most 16384 bytes.
[services."api.svc.emfor".check_options]
path = "/healthz"
host_header = "api.internal"
expect_status = ["200-299"]
expect_body = '"status":"UP"'

A response with another status is down with the status code in the reason, and its body is not read. With expect_body, a response whose status matches is read until the text is found, the body ends, or 65536 bytes have been read; without the text it is down with expect_body: got "<the first 64 bytes, escaped>", and a body that cannot be read is down with a reason starting body: . Use it for health endpoints that answer 200 either way and put the state in the body. The latency fastest uses then includes reading the body.

exec

Runs a program once per check, without a shell. The backend is passed in the environment variables EMFOR_IP, EMFOR_PORT and EMFOR_SERVICE.

[services."report.svc.emfor"]
check = "exec"
backends = ["10.0.3.1:5432"]
interval = "10s"
timeout = "5s"
check_options = { command = ["/usr/local/libexec/check_pg_replication", "--max-lag", "60"] }
ExitResult
0up
1 or 2down; the reason is the first line of stdout, or exit code N
any other code, killed by a signal, or failed to starterror
still running at timeoutdown (timeout); the program and all its children are killed

command is the only option: a non-empty list of strings without NUL whose first element is an absolute path to an executable. To run an existing Nagios plugin, which wants -H and -p on its command line, see Watching emfor from Nagios. Starting a process per check is costly; for frequent checks of many backends, use a plugin.

Plugins and emfor-check

A plugin is a long-running program that the daemon starts once and sends check requests to, as JSON lines on its stdin and stdout. One process serves every service that uses it, and it can keep connections and state between checks.

emfor ships one plugin program, emfor-check. Its argument selects the check:

[plugins."pgsql"]
command = ["/usr/local/libexec/emfor-check", "pgsql"]

[services."db.svc.emfor"]
check = "pgsql"
balance = "failover"
backends = ["10.0.3.1:5432", "10.0.3.2:5432"]
check_options = { user = "emfor", database = "postgres", role = "primary", password_file = "/etc/emfor/pg.pass" }
ArgumentUp when
redisPING answers PONG (after AUTH when a password is set)
mysqlThe login succeeds and SELECT 1 returns 1 (tested with MariaDB, and MySQL 8.0 and 8.4)
pgsqlThe login succeeds and SELECT 1 returns 1
ldapA simple bind succeeds
dnsOne query gets an expected response code (and records, with expect_answer)
radiusA Status-Server (or Access-Request) gets an accept signed with the shared secret
sipAn OPTIONS request gets a final status code in expect_status (default 200 to 499)
diameterA capabilities exchange (CER / CEA) and a device watchdog (DWR / DWA) succeed
smtpThe greeting is 220 and EHLO answers 250, over STARTTLS or TLS when asked for
amqpAn AMQP 0-9-1 login succeeds and the vhost opens
mongodbhello answers, the SCRAM-SHA-256 login succeeds when user is set, and the role matches
grpcThe standard health service (grpc.health.v1.Health/Check) answers SERVING
quicA QUIC handshake over UDP completes with a verified certificate and an ALPN in alpn; with http3 = true, an HTTP/3 GET also gets a status in expect_status

Main options: user, password_file (never the password itself; read on every check), database (pgsql), role (master/replica for redis, primary/replica for MySQL and PostgreSQL; primary lets failover follow the primary, and replica also needs replication to be running; primary/secondary for MongoDB), max_lag (with role = "replica", the seconds a replica may be behind), and tls, starttls (ldap), tls_verify, server_name, ca_file. The dns plugin takes name, type, transport, recursion_desired, expect_rcode and expect_answer instead, and the radius plugin secret_file, request, application, user, password_file, reject_is_up, nas_identifier and require_message_authenticator, and the sip plugin transport, tls (with the TLS keys), uri, from and expect_status, and the diameter plugin origin_host, origin_realm, the application ids, supported_vendor_ids, host_ip_address and tls (with the TLS keys and the client certificate tls_cert_file and tls_key_file), and the smtp plugin helo, starttls and tls (with the TLS keys), and the amqp plugin user, password_file, vhost and tls (with the TLS keys), and the mongodb plugin user, password_file, auth_source, role and tls (with the TLS keys), and the grpc plugin service and tls (with the TLS keys), and the quic plugin alpn, tls_verify, server_name, ca_file (always TLS, so no tls key), http3, path, host_header, expect_status and expect_body. docs/deploy/plugins.md lists every option and reason.

emfor check-config and every reload start the plugins and have them validate each service’s check_options, so mistakes show up before the config is applied. Validation builds what a check builds: a ca_file must hold a certificate usable as a CA, a tls_key_file must be the key of tls_cert_file, and a value the protocol sends as a NUL-terminated or NUL-separated string (the mysql and pgsql user, the pgsql database and password, the amqp user and password) must not contain NUL. A message that must fit a size the protocol sets is built once: the amqp login must fit in the 4096-byte first frame, and the diameter Capabilities-Exchange-Request in 16777215 bytes.

When a plugin exits, its pending checks become error (plugin exited), and the daemon restarts it after 1 s, then 2 s, 4 s … up to 30 s; until then its checks are error (plugin <name> unavailable). A crashing plugin never marks backends down: an error does not count as a failed check. But a backend whose checks have given nothing but errors for longer than fall × interval goes back to unknown (see chapter 1), so a plugin that keeps crashing leaves its backends unknown rather than healthy. A check that panics in a plugin built on the emfor-plugin crate is answered error with reason check panicked: <message>, and the plugin keeps running. Lines a plugin writes to stderr are logged at info, at most 100 at once and 10 a second after that; the log says how many were dropped. Each plugin runs in a process group of its own: when it exits or is stopped, the processes it started are killed too.

To write your own plugin, see docs/plugins/protocol-v1.md; the emfor-plugin crate implements the protocol for Rust.

Scheduling

Each backend is checked on its own schedule, and never by two checks at once:

  • The first check after a start or reload begins at a random moment within min(interval, 200ms).
  • While the backend is unknown and checks return up or down, the next check follows min(interval, 200ms) after the previous one ends. With the defaults (rise = 2), backends are known within about half a second. This fast pace lasts rise + fall checks at most: a backend whose results keep changing so that neither count is reached stays unknown and is then checked every interval.
  • Otherwise the next check starts interval after the previous one ends, shifted at random by up to jitter (10% by default) either way.

Because the interval is counted from the end of a check, a slow backend is not checked more often than every interval.

Rise, fall and errors

  • up: the success counter goes up and the failure counter resets; at rise the backend turns healthy.
  • down: the failure counter goes up and the success counter resets; at fall the backend turns unhealthy.
  • error: no counter and no state changes. When a backend has had only errors for longer than fall × interval, it goes back to unknown.

Time to detect a failure is about fall × interval (6 s with the defaults); time to recover is about rise × interval (4 s). Every state change is logged at info with the service, backend, old and new state, and the reason.

Reasons

A down or error result carries a reason, shown by emfor status, in logs and in the state history. Before it is stored, the reason is made safe: tab, CR and LF become \t, \r and \n; other control characters become \u{1b} and the like; and a reason longer than 512 characters is cut and ends in ….

Latency

For every backend the daemon measures how long each up check takes and keeps a smoothed value (each new sample weighs 5%). emfor status and /metrics show it, and balance = "fastest" uses it. Choose a check that resembles real traffic when you rely on it.

Built with Hugo
Theme Stack designed by Jimmy