emfor is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.
0.6.0 was released on 2026-11-17. 0.5.0 was about what happens when something goes wrong around the daemon; this release is about running emfor next to what you already have: taking a backend out for a deploy, keeping the config in several files, asking emfor what an application would get and why, and plugging its DNS answers into nginx, HAProxy, Caddy, Prometheus, pf and the clients that look up SRV records themselves. It also ships deb and rpm packages, and a manual chapter for people who come from HAProxy or Consul.
1. Before you upgrade
The store format (version 2) and the plugin protocol (1) are unchanged: clients and plugins of 0.5.0 keep working with this daemon, and a config that 0.5.0 accepted is still accepted. Three things can still affect a running setup:
- Upgrade
libemforand the clients before you usedrain = true. A drained backend is written to the store asunhealthy, so an older client skips it like any unhealthy one. The exception is the last tier of anopenservice when every backend is down: there an older client can still choose it. - A
[peers]secret_filethat group or others can write is now refused, as a config file with a secret in it should be. A group that has such a file stops loading its config after the upgrade:emfor check-configfinds it first. (The files of the newinclude_dirmust not be writable by group or others either, and must be owned by root or the user that runs emfor; runemfor check-configas that user.) libemforis named for 0.6 (libemfor.so.0.6), and the example FreeBSD port moved fromexamples/freebsd/topackages/freebsd/(the same files): update a link or a script that names the old path.
2. drain = true: take a backend out for a deploy
A backend table can set drain = true. After a reload the backend is never chosen: by no balance method (sticky and failover included), not in DNS A, AAAA and SRV answers, not in a fastest pool. Its checks keep running, so emfor status shows drained (checks: healthy), and removing drain brings it back at once, in the state the checks gave, without counting rise again. emfor lookup --explain lists it as drained, and the metric emfor_backend_drain is 1 for it.
When every backend of a service is drained, lookups fail with ALL_DOWN and DNS answers SERVFAIL, open or not. drain belongs to the host: it is not sent to peers and not part of the service definition they compare. There is no run-time command: you edit the config and reload.
Chapter 8 of the manual has a rolling-deploy recipe around it, and examples/deploy/emfor-drain.sh does the steps for one backend: set drain in a conf.d file, check the config, reload, wait until emfor shows the backend drained, and later wait until its checks have said healthy before it is put back. With nginx in front, a request in flight finishes after a drain and is cut after a plain stop. wait-ready does not trust the first healthy: a backend that has just stopped keeps that state for fall x interval (6 seconds by default), so it waits for a check newer than the command and healthy for ten seconds in a row.
3. The config in several files, and check-config --show
include_dir = "conf.d" in emfor.toml makes the daemon read the main file and then every *.toml file of that directory in filename order (byte order: write 01-, 10-, 99-), a later file overriding an earlier one. Tables merge key by key; anything else, lists such as backends included, is replaced by the later value. A later file cannot remove a service or backend. [store] and include_dir only work in the main file, because emfor status and emfor lookup read only that file.
The directory must exist, or the config is refused (at start the daemon exits 2; on a reload it keeps the old config). Errors name their file, emfor check-config lists the files it read, and the daemon logs the list at start and reload. The packages below ship a config that only sets include_dir = "conf.d" plus an empty directory.
emfor check-config --show checks the config as before and prints the merged config as TOML on standard output, each value followed by # <file>, the file that last set it. Built-in defaults are not added and nothing is redacted. The comment format is for people, not for scripts; without --show nothing changes.
4. emfor lookup, and a clearer emfor status
emfor lookup <service> [--key <k>] [--explain] [--json] [--dns A|AAAA] prints the backend a client library would give, for shell scripts and cron jobs without a client. --explain shows whether the store is stale, the tier the candidates come from, the reason every other backend is out (unhealthy, unknown, last check failed, backup held back, not in the pool, not at this site, stale), slow-start steps and how balance chooses, without advancing any rotation. --dns A|AAAA explains the answer of the DNS server from the store. --json only gains fields; exit status 0 is an answer, 1 no backend, 2 no such service or bad usage, 3 an unusable store.
The text output of emfor status now marks a backup backend, marks a backend that is still healthy but whose last check failed (last check failed, which lookups usually skip), and says too few live members to vote on the peers: line when the group has fewer than three live members, so every reason is this host’s own.
5. Recipes for the tools you already run
Chapter 7 of the manual is now two: the DNS reference, and chapter 7b, the recipes. Each recipe has a test script that runs against the real program in a container (tests/<name>/run.sh); the Integrations workflow runs most of them every week, and the pf one was run by hand on FreeBSD. What the tests found, and the manual now says:
- nginx (1.27.3 or later):
server api.svc.emfor service=http resolve;withresolver 127.0.0.1:5300takes the upstream from emfor’s SRV answers; health,drainand weights reach it within seconds, without a reload. It works instreamtoo. After a failed lookup nginx asks again only afterresolver_timeout, 30 seconds by default, so the recipe setsresolver_timeout 1s. - HAProxy: a
server-templatefills its servers from the SRV answers. HAProxy ignores SRV priorities and weights, sofailover,weighted,localityandfastestdo not behave as for nginx; afterSERVFAILit drops its servers afterhold other(30 seconds) where nginx keeps its last list; with TLS to the backends setsni. - Caddy:
reverse_proxywithdynamic srv. It needsresolversin bothdynamic srvandtransport http, agrace_period(without it every request fails when emfor answersSERVFAIL) andlb_try_duration(without it a backend that has just stopped loses requests); it follows priorities only withlb_policy first. - Prometheus:
dns_sd_configsof typeSRVgives the healthy backends as scrape targets. A backend that fails its checks is no longer a target, so itsupseries ends andup == 0never fires: alert on emfor’s own metrics. - pf (FreeBSD):
examples/pf/emfor-pf-table.shkeeps a pf table equal to the SRV targets with the lowest priority, so anrdrrule spreads connections over the backends emfor offers, with no proxy. - Clients that ask for SRV records themselves: MIT Kerberos (
dns_lookup_kdc), OpenLDAP (ldapsearch -H 'ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor'; the commas must be%2C) and PyMongo (mongodb+srv://) find their servers through emfor, and leave a server that fails its checks or is drained. - Nagios, Icinga and Zabbix:
examples/nagios/check_emfor.shgives the Nagios status of whatemfor status --jsonreports, andexamples/nagios/emfor-exec.shruns an existing Nagios plugin as anexeccheck.
Each section ends with what was not tried (real KDC, LDAP and MongoDB servers, nginx Plus, HTTPS upstreams and others); read it before you rely on a recipe. Chapter 10 is new: it translates an HAProxy backend and a Consul service with an HTTP check into emfor.toml, option by option, says what has no equivalent (least connections, retries, passive checks, registration), and describes how to try emfor next to what is running.
6. deb and rpm packages
packages/build.sh builds a deb and an rpm in a container with an old glibc (Debian 11 and later, Ubuntu 20.04 and later, RHEL 9 and relatives): emfor and emfor-check under /usr/bin, the systemd unit, the example config, and the emfor user (the deb enables and starts the service, the rpm does not). They are built from the repository and not published, and they are not signed. They depend on ca-certificates: the http check builds its client with the system’s root certificates even for scheme = "http", and fails without them.
7. Fixes from a review of the whole system
A review of the whole system, read against the code and the tests, found and fixed:
- a reload or shutdown no longer counts the plugin requests it cancels as “no answer”, so a healthy plugin shared by several services is not killed and restarted when one of them is reloaded away;
emfor check-configandemfor lookupreport a write error on standard output with a status instead of panicking; a closed pipe is not an error;- a plugin’s
validateanswer is bounded, so that a rejected reload cannot make the daemon kill the running plugin; bootin a peer message must be lowercase hex, as specified; a peer’sdownreason is cut and escaped before it is logged; the DNS rate limiter sweeps a full table once a second, not once per packet.
Chapters 7, 8 and 10 of the manual were corrected where they said more than the tests had shown. Not changed, on purpose: the RADIUS secret_file, password_file and tls_key_file still have no permission check, because adding one would stop installs that work today from starting.
8. Known limits
- FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests, the rc.d script, the port with
make stage check-plist, the fork tests, and the nginx, HAProxy and pf recipes by hand. Not tested: poudriere, 13.x, 14.x, amd64. - HAProxy 3.4.4 on FreeBSD answered 503 for a while after a
server-templateslot came up with port 0, also with one name server; the cause is not known, and the same test passes on Linux. - The deb and rpm have been installed, upgraded and removed in containers (arm64) without systemd (the deb also under systemd); not tested on x86-64 or the rpm under systemd, and they are not signed or published.
- Unchanged from earlier releases: peer messages have no forward secrecy; the
ldapplugin has been checked against Active Directory only without a certificate; lookup throughput across threads levels off at about two million a second on Linux; with[store] sync = false(the default), a host crash can damage the store; a host announces at most 80 services to the peer group.
9. How to upgrade
- Run
emfor check-configas the daemon’s user with the 0.6.0 binary on the config of every host before you upgrade it (section 1). - Install
libemforand the daemon from 0.6.0, host by host, clients first if you plan to usedrain = true. - To use the new things: add
include_dirand move services into files, tryemfor lookup --explainon a host, and read chapter 7b for the proxy or tool you run.
The user manual on this site is updated to 0.6.0. The full list of changes is in the CHANGELOG, and the source is in the v0.6.0 release on GitHub.
💡 Technical collaboration and architecture consulting
Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends? We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.
