Featured image of post emfor 0.4.0: Backup Backends, Slow Start, Encrypted Peer Groups and Five New Plugins

emfor 0.4.0: Backup Backends, Slow Start, Encrypted Peer Groups and Five New Plugins

emfor 0.4.0 makes applications choose backends more carefully: backup backends, a gradual return for recovered backends, and stepping around a backend at its first failed check. Peer group messages are now encrypted, and emfor-check adds SMTP, AMQP, MongoDB, gRPC and QUIC/HTTP/3.

emfor is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.

0.4.0 was released on 2026-11-14. Its focus is the address an application gets back: when to fall back to a backup, whether a backend that just recovered should take its full share at once, and how quickly to step around a backend that starts to fail. Peer group messages are now encrypted, and emfor-check speaks five more protocols.

1. Before you upgrade: three breaking changes

1. The store format is version 2. The daemon writes meta/schema_version 2, which adds the backend field backup. Clients and emfor status from 0.4.0 read versions 1 and 2; older clients fail every lookup on a version 2 store with INCOMPATIBLE. So upgrade the clients on a host (libemfor and the language packages) before the daemon.

2. Peer messages are encrypted by default. Members still on 0.3.0 cannot read them and drop them as malformed. To upgrade a group host by host, add encrypt = false to [peers] on each host as you upgrade it; once every member runs 0.4.0, remove the line and reload.

3. FreeBSD: examples/rc.d/emfor is gone. The only rc.d script is now the port’s examples/freebsd/files/emfor.in; without the port, fill in %%PREFIX%% as chapter 2 of the manual shows and install it. Its emfor_user defaults to emfor rather than root. The example port also keeps the store in /var/db/emfor, where FreeBSD keeps daemon state, so applications on the host set EMFOR_STORE=/var/db/emfor/state.mdb.

2. Choosing a backend more carefully

Backup backends

A backend can be marked as a backup:

[services."db.svc.emfor"]
check = "tcp"
balance = "failover"
backends = [
  "10.0.1.5:5432",
  "10.0.1.6:5432",
  { addr = "10.1.1.5:5432", backup = true },   # another data centre
]

As long as any backend that is not a backup passes its checks, the backups are not used. The order is: primaries whose last check passed → backups whose last check passed → healthy primaries → healthy backups, and balance works within the chosen set. The typical use is a backend in another data centre, or a more expensive one. SRV answers list only the chosen set.

Step around a backend at its first failed check

A backend has to fail fall checks in a row before it turns unhealthy, about 6 seconds by default. Until then it is still healthy, and 0.3.0 clients kept connecting to it.

0.4.0 clients and the DNS server leave out a healthy backend whose last check failed, as long as another healthy backend passed its last check. In other words, a backend stops being chosen at its first failed check instead of after fall of them; its state still follows rise and fall, and it comes back once it passes. When every healthy backend is failing, all of them are used as before, so nothing is refused because of this.

With on_all_down = "open" and no healthy backend, there is a similar change: the unknown backends (not decided yet, or whose checks only give errors) are used before the unhealthy ones that failed fall checks.

Slow start

A backend that has just recovered has a cold cache and an empty connection pool, and its full share of traffic at once often knocks it over again. The new slow_start gives it a growing share over a set time, in 10 steps:

[defaults]
slow_start = "30s"     # default "0s", off

It applies to client lookups and DNS A/AAAA answers for every method but failover and sticky with a key, and to SRV weights.

Sticky honours weight

balance = "sticky" with an affinity key now follows weight: a backend of weight w gets w shares of the keys. The computation uses only integers and MD5, so every host and every language still picks the same backend for a key. With every weight 1 nothing changes; a sticky service that already sets other weights moves some keys when its clients are upgraded, so upgrade the clients of such a service together.

3. Peer groups

Encrypted messages. 0.3.0 signed peer messages but sent them in the clear, backend names, addresses and reasons included. 0.4.0 sends EMP2: a key derived from secret_file with HKDF-SHA256, ChaCha20-Poly1305, and a random nonce per message. Secret rotation works as before. There is no forward secrecy: whoever later gets the secret can read recorded messages, so rotate it regularly.

A failure only a non-assigned member sees is noticed sooner. The known limit of 0.3.0 was that a failure only a member which is not one of a backend’s assigned checkers sees took up to about five intervals plus fall checks to notice. In 0.4.0 such a member checks the backend itself at least every third interval, and when its own check fails while the assigned checkers say up, it starts a fast recheck run (every 200 ms, up to fall checks), as it does after a peer’s down report. Measured over 12 failures with interval = "2s" and fall = 3, that host marked the backend down after 0.8 to 5.8 seconds, 3.3 seconds median; 0.3.0 took up to about 14 seconds by the schedule. In exchange a backend gets 3 + (N − 3) / 3 checks per interval instead of 3 + (N − 3) / 5.

4. emfor-check: five new plugins

emfor-check is the plugin that ships with emfor for services whose checks need to speak their protocol. 0.4.0 adds:

PluginWhat it checks
smtpThe 220 greeting and EHLO, optionally with STARTTLS or TLS on port 465; it never logs in or sends mail
amqpAMQP 0-9-1 brokers such as RabbitMQ: a PLAIN login and opening vhost
mongodbhello on MongoDB 4.4 and later, with an optional SCRAM-SHA-256 login; role = "primary" lets failover follow the replica set’s primary
grpcThe standard grpc.health.v1.Health/Check; up only on SERVING
quicA QUIC version 1 handshake; with http3 = true also an HTTP/3 GET, judged on status and body like the built-in http check

Database checks are stricter too: role = "replica" now means a replica that is replicating: a redis replica whose link to its master is connected, a MySQL or MariaDB replica whose IO and SQL threads both run, a PostgreSQL standby with a WAL receiver. With max_lag you can bound how far behind a replica may be. Note that a redis replica that was up with role = "replica" in 0.3.0 while its link was down is down in 0.4.0.

LDAP reasons read better against Active Directory: the NUL byte AD puts at the end of its diagnostic message is gone, and a server that drops the connection during the TLS handshake (as AD does without an LDAPS certificate) is down with tls: connection closed during the TLS handshake (does the server have a certificate?).

5. Other changes

  • The built-in http check takes expect_body: text the response body must contain, for health endpoints that answer 200 either way.
  • On macOS, exec checks and plugins no longer inherit the daemon’s sockets and pipes.
  • FreeBSD: the rc.d script starts daemon(8) with -f, so the SSH session that runs service emfor start no longer hangs. The daemon, the rc.d script, the port and the tests have run on FreeBSD 15.1-RELEASE (arm64) for the first time.
  • Every fuzz target runs for 10 minutes before a release. For 0.4.0 the five targets (plugin answers, peer messages, metrics, the plugin protocol and the config) ran about 18 million inputs between them without a crash.
  • The user manual is now on this site too: emfor User Manual.

6. Known limits

  • FreeBSD has been tested on 15.1-RELEASE arm64 only; not with poudriere, on 14.x or on amd64.
  • The ldap plugin has been checked against Active Directory only without a certificate and without a successful login; the mysql plugin, at release, only against MariaDB (since checked against MySQL 8.0 and 8.4, which pass).
  • With [store] sync = false (the default), a host crash can leave the store damaged; the daemon then sets it aside at the next start and the state history is lost. Chapter 8 of the manual shows how to keep the store on a tmpfs.

7. How to upgrade

  1. On every host, upgrade the clients first: libemfor and the language packages from 0.4.0.
  2. Upgrade and restart the daemon host by host. With a peer group, add encrypt = false to [peers] on each host as you upgrade it (0.3.0 does not know the key, so do not add it ahead).
  3. Once every member runs 0.4.0, remove encrypt = false and reload each daemon.

The full list of changes is in the CHANGELOG, and the source is in the v0.4.0 release on GitHub.


💡 Technical collaboration and architecture consulting

Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends? We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.

📧 Book a technical consultation

Built with Hugo
Theme Stack designed by Jimmy