emfor is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.
0.5.0 was released on 2026-11-15. 0.4.0 changed which backend an application gets back; this release is about what happens around it when something goes wrong: a store cut short by a power loss, a plugin that stops answering, a disk that is slow to commit, a DNS port that anyone can reach, a forked child that hangs on FreeBSD. It also adds a second encrypted format for peer groups, and emfor check-config now refuses more options that could never work.
1. Before you upgrade: three things that can affect a running setup
1. A config that 0.4.0 accepted can now be refused. emfor check-config, and so every start and reload, refuses options that could never work and that 0.4.0 let through, to fail later. The ones you are most likely to meet:
intervalabove24hortimeoutbelow10ms;- an http
host_headerthat is not a host with an optional port (spaces, a path or a scheme), and an httppathwith#,\or a control character; - with
[dns], a service name too long to leave room for the SRV target names of its backends; - an exec or plugin
commandwith a NUL, and aca_filewith no certificate usable as a CA; - in
emfor-check: a NUL in a mysql, pgsql or amqp user or password, an amqp password too long for AMQP’s first frame, a diameter option list too long for one message, a sipuriorfromwith<or>, and a mongodbpassword_filewhose password is not printable ASCII.
Run emfor check-config with the new binary on each host’s config before you upgrade that host.
2. The DNS server limits UDP responses per client network. By default each IPv4 /24 (IPv6 /56) gets 100 responses a second ([dns] rate_limit, 0 turns it off). An SRV answer can be many times the size of its query, so a server that other hosts can reach could be used to flood a third party with forged-source queries. Past the limit every other response is an empty truncated one, which makes a real client retry over TCP, and the rest are dropped; emfor_dns_rate_limited_total counts them. Loopback clients and TCP are never limited. If the resolvers that forward your zone send more than 100 queries a second from one network, raise the limit. At start the daemon also warns about every [dns] listen address that is not a loopback address.
3. libemfor is named for 0.5: libemfor.so.0.5 and libemfor.0.5.dylib. As with every release, install libemfor from the same release as the daemon; the Python package looks for the file named for its own release first.
The store format (version 2) and the plugin protocol (1) are unchanged.
2. Peer groups
A second encrypted format, EMP3. 0.4.0 encrypts peer messages with ChaCha20-Poly1305 (EMP2) and a random 12-byte nonce per message. That is safe for about 2^32 messages under one secret; by an estimate, not a measurement, a large group can send that many within days, and a repeated nonce would let an observer read and forge messages. EMP3 uses XChaCha20-Poly1305 with a 24-byte random nonce and a key of its own, which has no practical limit.
The format is chosen per member. Every EMP2 message of 0.5.0 carries "emp3": true inside its encrypted payload, and a member is sent EMP3 once the latest message accepted from it is EMP3 or such an EMP2; otherwise it is sent EMP2. So 0.5.0 members use EMP3 between themselves and EMP2 with 0.4.0 members, and a group upgrades, or goes back, in any order without a configuration change. A host that goes back to 0.4.0 drops the EMP3 messages for about its first second. The EMP3 bytes are checked against a second implementation, written separately, which itself passes the test vectors of the XChaCha20 draft.
More in the peer group:
[peers] accept_plaintext = falsedrops every unencryptedEMP1message without checking it, counted under the new reasonplaintext, so a group whose members all encrypt no longer has to accept signed plaintext. It needsencrypt = true.- New
[peers] previous_secret_file: the secretsecret_fileheld before its last change. A daemon restarted within 120 seconds of that change keeps signing with and accepting the old secret as if it had not restarted; before, members whose file had not changed yet rejected it asauth. - A second change of
secret_filewithin 60 seconds of the first no longer signs for a minute with the intermediate secret, which members whose file had not changed did not know. - A voting fix. A member kept another member’s latest result for 3 of the sender’s intervals and counted it at every one of its own check times in that span. With
interval = "1s"here and"30s"at the assigned checkers, onedownfrom each of two checkers could movefallup to about 90 times, and mark down a backend that the checkers themselves still saw healthy. Now each result counts at one check time. - Messages that arrive out of order are accepted once instead of being dropped as
replay.
3. Staying up when something goes wrong
- A store cut short no longer ends the process with
SIGBUS. A store file shorter than LMDB records it (changed in place, or its tail lost in a power loss) is moved to<path>.corruptat start like other damage, with a warning that saystruncated. Before, reading it to check it ended the daemon withSIGBUS, again at every restart.emfor statusprintscannot open store: truncatedand exits 2, and a client fails the lookup withUNAVAILABLEinstead of ending the application. After every commit the daemon now extends the file to the length LMDB records, so a healthy store is never taken for a truncated one. A store changed in place while it is open can still do this to a running client: replace it withmv, nevercpover it ortruncateit. - A plugin that stops answering is restarted. Until now only a plugin that exited or closed its stdout was restarted; one that kept running but answered nothing left every check of its backends
errorfor good, and the requests for it queued up without bound. When the callers of 3 requests in a row give up and no answer arrives in between, the daemon logs it, kills the plugin and restarts it. A plugin that answers withintimeout_msis never affected. - A slow commit no longer freezes DNS answers and
/metrics. While the state store waited for the disk, the daemon stopped applying check results, so DNS kept answering with old states. Results are now applied as they arrive; the changes made meanwhile are written together by the next commit. - Bounded queues. A peer task that falls more than 4096 messages behind drops new
downandresultmessages (counted inemfor_peer_queue_dropped_total); a plugin that stops reading its stdin gets at most 1024 waiting requests, and further checks areerrorat once withplugin <name> busy. - DNS. TCP serves at most 128 connections and closes one with no query for 5 seconds; if a DNS listener stops, the daemon logs it and exits with status 70 so the service manager restarts it, where before DNS went silent; BADVERS for EDNS versions above 0 and REFUSED for classes other than IN.
- A single key or value in the store that is not UTF-8 no longer blocks the daemon; a reload that changes
intervalapplies to the wait already under way (going from10mto2sused to take up to 11 minutes); a client keeps reading after the daemon is restarted with a larger[store] map_size.
4. emfor-check: hangs and crashes with a badly behaved server
A review of every plugin against servers that answer wrongly found a number of hangs and crashes. All are now down or error with a reason, and the check’s timeout ends them:
mongodb: a server asking for 4294967295 SCRAM iterations held a worker thread for many minutes (more than 1000000 is now refused), and ahelloreply nested about 8000 deep overflowed the stack and aborted the plugin;sipover TCP or TLS: a Content-Length near the largest 64-bit number made the plugin loop forever at full CPU on the same provisional response;redis: a reply over 2 MiB;mysql: a NULL answer toSELECT 1panicked the check, and a check cut off by its timeout leaked a task and a socket every time;- a reason longer than 512 characters is cut before it is sent: a 70 KB redis
-ERRmade the answer line longer than the daemon’s limit, so the daemon restarted the plugin at every check of that backend; - a check that panics is answered
errorwithcheck panicked: <message>instead of no answer.
The mysql plugin’s live tests now also run against MySQL 8.0 and 8.4 (tests/plugins/mysql.sh), and pass; before, only MariaDB was checked.
5. Clients, and a hang on FreeBSD
The clients fail a lookup on a truncated store and keep reading after a store grows (above), try a store that exists but cannot be opened at most once a second, and return a status instead of aborting when emfor_lookup is called from a thread-local destructor. A backend state or on_all_down value written by a newer daemon no longer makes the lookup fail with INCOMPATIBLE: an unknown state counts as unhealthy, an unknown on_all_down as open.
A forked child could hang for ever on FreeBSD. Our fork test, in which other threads look up while the process forks, failed in every run on FreeBSD 15.1 (arm64): a child waited for ever in its first lookup, within the first few dozen of 300 forks. 0.4.0 has the same hang there: built from its tag with the same tests, the Rust test failed in 10 of 10 runs and the C test in 5 of 5.
The cause is in FreeBSD’s threading library, libthr, not in emfor. LMDB keeps its reader mutex in the store’s lock file, as a process-shared mutex, and libthr looks such mutexes up behind a lock of its own. Its fork takes that lock only for reading, so the read hold of a thread that is inside such a mutex at that moment stays in the child, where nothing releases it; the child’s first use of a newly mapped mutex then waits for it. On a hung child we read the lock: one reader left and one writer waiting. libthr has a second problem next to this one: a parent that forks while another thread maps a process-shared mutex for the first time can deadlock.
The client now works around both: on FreeBSD it makes fork wait until no lookup is inside the store (pthread_atfork handlers), and libemfor can no longer be unloaded there, because the handlers stay registered. Lookup throughput did not change (measured with 1, 4 and 8 threads). It does not cover other users of process-shared mutexes in the same process, a fork from a signal handler that interrupted a lookup of the same thread, or a libemfor.a linked statically into a module that is unloaded.
We also wrote a patch for libthr and put it, with a small reproduction that does not use LMDB, in docs/freebsd/. We built libthr from the 15.1 source twice, unpatched and patched: unpatched, 40 to 55 children of 300 hang in each of 5 runs and the deadlocking parent never finishes; patched, no child hangs in 5 of 5 runs and the parent finishes. An LMDB client without our workaround passes on the patched library. We tried it on FreeBSD 15.1 (arm64) only, and have not reported it to FreeBSD yet.
6. Other changes
emfor status: a truncated store,emfor status | head -1keeping its exit status,--watchopening the store again when the file is replaced, and records from a newer daemon being shown instead of failing.- The example systemd unit sets
LimitNOFILE=65536. - Every fuzz target runs for 10 minutes before a release. For 0.5.0 the six targets (plugin answers, peer messages, the DNS server, metrics, the plugin protocol and the config; the DNS one is new) ran about 16 million inputs between them without a crash. They now keep overflow checks on and run every input on a 2 MiB stack like a tokio worker’s, so an overflow or a too-deep recursion is a crash, as it would be in the daemon.
- The user manual on this site is updated to 0.5.0.
7. Known limits
- FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests, the rc.d script, the port with
make stage check-plist install, and the fork tests. Not tested: poudriere, 13.x, 14.x, amd64. - The FreeBSD fork guard does not cover the cases above, and the libthr patch has been tried on 15.1 only.
- Peer messages have no forward secrecy: whoever later gets
secret_filecan read recorded messages. Rotate the secret regularly. - The
ldapplugin has been checked against Active Directory only without a certificate and without a successful login. - Unchanged from earlier releases: lookup throughput across threads levels off at about two million a second on Linux; with
[store] sync = false(the default), a host crash can leave the store damaged and the state history is lost; a host announces at most 80 services left out of the peer group.
8. How to upgrade
- Run
emfor check-configwith the 0.5.0 binary on the config of every host before you upgrade it (section 1). - Install
libemforand the daemon from 0.5.0, host by host. A peer group of 0.4.0 and 0.5.0 members works in any order, with no change to[peers]. - If your DNS clients send many queries from one network, watch
emfor_dns_rate_limited_totalafter the upgrade; if it grows, raise[dns] rate_limit.
The full list of changes is in the CHANGELOG, and the source is in the v0.5.0 release on GitHub.
💡 Technical collaboration and architecture consulting
Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends? We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.
