<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>Rust on Embark Systems</title>
        <link>https://embark.systems/en/tags/rust/</link>
        <description>Recent content in Rust on Embark Systems</description>
        <generator>Hugo -- gohugo.io</generator>
        <language>en</language>
        <lastBuildDate>Tue, 17 Nov 2026 07:00:00 +0800</lastBuildDate><atom:link href="https://embark.systems/en/tags/rust/index.xml" rel="self" type="application/rss+xml" /><item>
            <title>emfor 0.6.0: Drain a Backend for a Deploy, a Config in Several Files, and Recipes for the Tools You Already Run</title>
            <link>https://embark.systems/en/posts/emfor-0.6.0/</link>
            <pubDate>Tue, 17 Nov 2026 07:00:00 +0800</pubDate>
            <guid>https://embark.systems/en/posts/emfor-0.6.0/</guid>
            <description>&lt;img src=&#34;https://embark.systems/images/emfor-0.6.0.png&#34; alt=&#34;Featured image of post emfor 0.6.0: Drain a Backend for a Deploy, a Config in Several Files, and Recipes for the Tools You Already Run&#34; /&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;emfor&lt;/a&gt; is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.&lt;/p&gt;&#xA;&lt;p&gt;0.6.0 was released on 2026-11-17. 0.5.0 was about what happens when something goes wrong around the daemon; this release is about &lt;strong&gt;running emfor next to what you already have&lt;/strong&gt;: taking a backend out for a deploy, keeping the config in several files, asking emfor what an application would get and why, and plugging its DNS answers into nginx, HAProxy, Caddy, Prometheus, pf and the clients that look up SRV records themselves. It also ships deb and rpm packages, and a manual chapter for people who come from HAProxy or Consul.&lt;/p&gt;&#xA;&lt;h2 id=&#34;1-before-you-upgrade&#34;&gt;1. Before you upgrade&#xA;&lt;/h2&gt;&lt;p&gt;The store format (version 2) and the plugin protocol (1) are unchanged: clients and plugins of 0.5.0 keep working with this daemon, and a config that 0.5.0 accepted is still accepted. Three things can still affect a running setup:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;&lt;strong&gt;Upgrade &lt;code&gt;libemfor&lt;/code&gt; and the clients before you use &lt;code&gt;drain = true&lt;/code&gt;.&lt;/strong&gt; A drained backend is written to the store as &lt;code&gt;unhealthy&lt;/code&gt;, so an older client skips it like any unhealthy one. The exception is the last tier of an &lt;code&gt;open&lt;/code&gt; service when every backend is down: there an older client can still choose it.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A &lt;code&gt;[peers]&lt;/code&gt; &lt;code&gt;secret_file&lt;/code&gt; that group or others can write is now refused&lt;/strong&gt;, as a config file with a secret in it should be. A group that has such a file stops loading its config after the upgrade: &lt;code&gt;emfor check-config&lt;/code&gt; finds it first. (The files of the new &lt;code&gt;include_dir&lt;/code&gt; must not be writable by group or others either, and must be owned by root or the user that runs emfor; run &lt;code&gt;emfor check-config&lt;/code&gt; as that user.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;libemfor&lt;/code&gt; is named for 0.6&lt;/strong&gt; (&lt;code&gt;libemfor.so.0.6&lt;/code&gt;), and the example FreeBSD port moved from &lt;code&gt;examples/freebsd/&lt;/code&gt; to &lt;code&gt;packages/freebsd/&lt;/code&gt; (the same files): update a link or a script that names the old path.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2 id=&#34;2-drain--true-take-a-backend-out-for-a-deploy&#34;&gt;2. &lt;code&gt;drain = true&lt;/code&gt;: take a backend out for a deploy&#xA;&lt;/h2&gt;&lt;p&gt;A backend table can set &lt;code&gt;drain = true&lt;/code&gt;. After a reload the backend is never chosen: by no &lt;code&gt;balance&lt;/code&gt; method (&lt;code&gt;sticky&lt;/code&gt; and &lt;code&gt;failover&lt;/code&gt; included), not in DNS A, AAAA and SRV answers, not in a &lt;code&gt;fastest&lt;/code&gt; pool. Its checks keep running, so &lt;code&gt;emfor status&lt;/code&gt; shows &lt;code&gt;drained (checks: healthy)&lt;/code&gt;, and removing &lt;code&gt;drain&lt;/code&gt; brings it back at once, in the state the checks gave, without counting &lt;code&gt;rise&lt;/code&gt; again. &lt;code&gt;emfor lookup --explain&lt;/code&gt; lists it as &lt;code&gt;drained&lt;/code&gt;, and the metric &lt;code&gt;emfor_backend_drain&lt;/code&gt; is 1 for it.&lt;/p&gt;&#xA;&lt;p&gt;When every backend of a service is drained, lookups fail with &lt;code&gt;ALL_DOWN&lt;/code&gt; and DNS answers &lt;code&gt;SERVFAIL&lt;/code&gt;, &lt;code&gt;open&lt;/code&gt; or not. &lt;code&gt;drain&lt;/code&gt; belongs to the host: it is not sent to peers and not part of the service definition they compare. There is no run-time command: you edit the config and reload.&lt;/p&gt;&#xA;&lt;p&gt;Chapter 8 of the manual has a rolling-deploy recipe around it, and &lt;code&gt;examples/deploy/emfor-drain.sh&lt;/code&gt; does the steps for one backend: set &lt;code&gt;drain&lt;/code&gt; in a &lt;code&gt;conf.d&lt;/code&gt; file, check the config, reload, wait until emfor shows the backend drained, and later wait until its checks have said healthy before it is put back. With nginx in front, a request in flight finishes after a drain and is cut after a plain stop. &lt;code&gt;wait-ready&lt;/code&gt; does not trust the first &lt;code&gt;healthy&lt;/code&gt;: a backend that has just stopped keeps that state for &lt;code&gt;fall&lt;/code&gt; x &lt;code&gt;interval&lt;/code&gt; (6 seconds by default), so it waits for a check newer than the command and &lt;code&gt;healthy&lt;/code&gt; for ten seconds in a row.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-the-config-in-several-files-and-check-config---show&#34;&gt;3. The config in several files, and &lt;code&gt;check-config --show&lt;/code&gt;&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;include_dir = &amp;quot;conf.d&amp;quot;&lt;/code&gt; in &lt;code&gt;emfor.toml&lt;/code&gt; makes the daemon read the main file and then every &lt;code&gt;*.toml&lt;/code&gt; file of that directory in filename order (byte order: write &lt;code&gt;01-&lt;/code&gt;, &lt;code&gt;10-&lt;/code&gt;, &lt;code&gt;99-&lt;/code&gt;), a later file overriding an earlier one. Tables merge key by key; anything else, lists such as &lt;code&gt;backends&lt;/code&gt; included, is replaced by the later value. A later file cannot remove a service or backend. &lt;code&gt;[store]&lt;/code&gt; and &lt;code&gt;include_dir&lt;/code&gt; only work in the main file, because &lt;code&gt;emfor status&lt;/code&gt; and &lt;code&gt;emfor lookup&lt;/code&gt; read only that file.&lt;/p&gt;&#xA;&lt;p&gt;The directory must exist, or the config is refused (at start the daemon exits 2; on a reload it keeps the old config). Errors name their file, &lt;code&gt;emfor check-config&lt;/code&gt; lists the files it read, and the daemon logs the list at start and reload. The packages below ship a config that only sets &lt;code&gt;include_dir = &amp;quot;conf.d&amp;quot;&lt;/code&gt; plus an empty directory.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;emfor check-config --show&lt;/code&gt; checks the config as before and prints the merged config as TOML on standard output, each value followed by &lt;code&gt;# &amp;lt;file&amp;gt;&lt;/code&gt;, the file that last set it. Built-in defaults are not added and nothing is redacted. The comment format is for people, not for scripts; without &lt;code&gt;--show&lt;/code&gt; nothing changes.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-emfor-lookup-and-a-clearer-emfor-status&#34;&gt;4. &lt;code&gt;emfor lookup&lt;/code&gt;, and a clearer &lt;code&gt;emfor status&lt;/code&gt;&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor lookup &amp;lt;service&amp;gt; [--key &amp;lt;k&amp;gt;] [--explain] [--json] [--dns A|AAAA]&lt;/code&gt; prints the backend a client library would give, for shell scripts and cron jobs without a client. &lt;code&gt;--explain&lt;/code&gt; shows whether the store is stale, the tier the candidates come from, the reason every other backend is out (&lt;code&gt;unhealthy&lt;/code&gt;, &lt;code&gt;unknown&lt;/code&gt;, &lt;code&gt;last check failed&lt;/code&gt;, &lt;code&gt;backup held back&lt;/code&gt;, &lt;code&gt;not in the pool&lt;/code&gt;, &lt;code&gt;not at this site&lt;/code&gt;, &lt;code&gt;stale&lt;/code&gt;), slow-start steps and how &lt;code&gt;balance&lt;/code&gt; chooses, without advancing any rotation. &lt;code&gt;--dns A|AAAA&lt;/code&gt; explains the answer of the DNS server from the store. &lt;code&gt;--json&lt;/code&gt; only gains fields; exit status 0 is an answer, 1 no backend, 2 no such service or bad usage, 3 an unusable store.&lt;/p&gt;&#xA;&lt;p&gt;The text output of &lt;code&gt;emfor status&lt;/code&gt; now marks a &lt;code&gt;backup&lt;/code&gt; backend, marks a backend that is still &lt;code&gt;healthy&lt;/code&gt; but whose last check failed (&lt;code&gt;last check failed&lt;/code&gt;, which lookups usually skip), and says &lt;code&gt;too few live members to vote&lt;/code&gt; on the &lt;code&gt;peers:&lt;/code&gt; line when the group has fewer than three live members, so every reason is this host&amp;rsquo;s own.&lt;/p&gt;&#xA;&lt;h2 id=&#34;5-recipes-for-the-tools-you-already-run&#34;&gt;5. Recipes for the tools you already run&#xA;&lt;/h2&gt;&lt;p&gt;Chapter 7 of the manual is now two: the DNS reference, and chapter 7b, the recipes. Each recipe has a test script that runs against the real program in a container (&lt;code&gt;tests/&amp;lt;name&amp;gt;/run.sh&lt;/code&gt;); the Integrations workflow runs most of them every week, and the pf one was run by hand on FreeBSD. What the tests found, and the manual now says:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;nginx&lt;/strong&gt; (1.27.3 or later): &lt;code&gt;server api.svc.emfor service=http resolve;&lt;/code&gt; with &lt;code&gt;resolver 127.0.0.1:5300&lt;/code&gt; takes the upstream from emfor&amp;rsquo;s SRV answers; health, &lt;code&gt;drain&lt;/code&gt; and weights reach it within seconds, without a reload. It works in &lt;code&gt;stream&lt;/code&gt; too. After a failed lookup nginx asks again only after &lt;code&gt;resolver_timeout&lt;/code&gt;, 30 seconds by default, so the recipe sets &lt;code&gt;resolver_timeout 1s&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;HAProxy&lt;/strong&gt;: a &lt;code&gt;server-template&lt;/code&gt; fills its servers from the SRV answers. HAProxy ignores SRV &lt;strong&gt;priorities and weights&lt;/strong&gt;, so &lt;code&gt;failover&lt;/code&gt;, &lt;code&gt;weighted&lt;/code&gt;, &lt;code&gt;locality&lt;/code&gt; and &lt;code&gt;fastest&lt;/code&gt; do not behave as for nginx; after &lt;code&gt;SERVFAIL&lt;/code&gt; it drops its servers after &lt;code&gt;hold other&lt;/code&gt; (30 seconds) where nginx keeps its last list; with TLS to the backends set &lt;code&gt;sni&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Caddy&lt;/strong&gt;: &lt;code&gt;reverse_proxy&lt;/code&gt; with &lt;code&gt;dynamic srv&lt;/code&gt;. It needs &lt;code&gt;resolvers&lt;/code&gt; in both &lt;code&gt;dynamic srv&lt;/code&gt; and &lt;code&gt;transport http&lt;/code&gt;, a &lt;code&gt;grace_period&lt;/code&gt; (without it every request fails when emfor answers &lt;code&gt;SERVFAIL&lt;/code&gt;) and &lt;code&gt;lb_try_duration&lt;/code&gt; (without it a backend that has just stopped loses requests); it follows priorities only with &lt;code&gt;lb_policy first&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Prometheus&lt;/strong&gt;: &lt;code&gt;dns_sd_configs&lt;/code&gt; of type &lt;code&gt;SRV&lt;/code&gt; gives the healthy backends as scrape targets. A backend that fails its checks is no longer a target, so its &lt;code&gt;up&lt;/code&gt; series ends and &lt;code&gt;up == 0&lt;/code&gt; never fires: alert on emfor&amp;rsquo;s own metrics.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;pf (FreeBSD)&lt;/strong&gt;: &lt;code&gt;examples/pf/emfor-pf-table.sh&lt;/code&gt; keeps a pf table equal to the SRV targets with the lowest priority, so an &lt;code&gt;rdr&lt;/code&gt; rule spreads connections over the backends emfor offers, with no proxy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Clients that ask for SRV records themselves&lt;/strong&gt;: MIT Kerberos (&lt;code&gt;dns_lookup_kdc&lt;/code&gt;), OpenLDAP (&lt;code&gt;ldapsearch -H &#39;ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor&#39;&lt;/code&gt;; the commas must be &lt;code&gt;%2C&lt;/code&gt;) and PyMongo (&lt;code&gt;mongodb+srv://&lt;/code&gt;) find their servers through emfor, and leave a server that fails its checks or is drained.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Nagios, Icinga and Zabbix&lt;/strong&gt;: &lt;code&gt;examples/nagios/check_emfor.sh&lt;/code&gt; gives the Nagios status of what &lt;code&gt;emfor status --json&lt;/code&gt; reports, and &lt;code&gt;examples/nagios/emfor-exec.sh&lt;/code&gt; runs an existing Nagios plugin as an &lt;code&gt;exec&lt;/code&gt; check.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Each section ends with what was not tried (real KDC, LDAP and MongoDB servers, nginx Plus, HTTPS upstreams and others); read it before you rely on a recipe. Chapter 10 is new: it translates an HAProxy backend and a Consul service with an HTTP check into &lt;code&gt;emfor.toml&lt;/code&gt;, option by option, says what has no equivalent (least connections, retries, passive checks, registration), and describes how to try emfor next to what is running.&lt;/p&gt;&#xA;&lt;h2 id=&#34;6-deb-and-rpm-packages&#34;&gt;6. deb and rpm packages&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;packages/build.sh&lt;/code&gt; builds a deb and an rpm in a container with an old glibc (Debian 11 and later, Ubuntu 20.04 and later, RHEL 9 and relatives): &lt;code&gt;emfor&lt;/code&gt; and &lt;code&gt;emfor-check&lt;/code&gt; under &lt;code&gt;/usr/bin&lt;/code&gt;, the systemd unit, the example config, and the &lt;code&gt;emfor&lt;/code&gt; user (the deb enables and starts the service, the rpm does not). They are built from the repository and not published, and they are not signed. They depend on &lt;code&gt;ca-certificates&lt;/code&gt;: the http check builds its client with the system&amp;rsquo;s root certificates even for &lt;code&gt;scheme = &amp;quot;http&amp;quot;&lt;/code&gt;, and fails without them.&lt;/p&gt;&#xA;&lt;h2 id=&#34;7-fixes-from-a-review-of-the-whole-system&#34;&gt;7. Fixes from a review of the whole system&#xA;&lt;/h2&gt;&lt;p&gt;A review of the whole system, read against the code and the tests, found and fixed:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a reload or shutdown no longer counts the plugin requests it cancels as &amp;ldquo;no answer&amp;rdquo;, so a healthy plugin shared by several services is not killed and restarted when one of them is reloaded away;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;emfor check-config&lt;/code&gt; and &lt;code&gt;emfor lookup&lt;/code&gt; report a write error on standard output with a status instead of panicking; a closed pipe is not an error;&lt;/li&gt;&#xA;&lt;li&gt;a plugin&amp;rsquo;s &lt;code&gt;validate&lt;/code&gt; answer is bounded, so that a rejected reload cannot make the daemon kill the running plugin;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;boot&lt;/code&gt; in a peer message must be lowercase hex, as specified; a peer&amp;rsquo;s &lt;code&gt;down&lt;/code&gt; reason is cut and escaped before it is logged; the DNS rate limiter sweeps a full table once a second, not once per packet.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Chapters 7, 8 and 10 of the manual were corrected where they said more than the tests had shown. Not changed, on purpose: the RADIUS &lt;code&gt;secret_file&lt;/code&gt;, &lt;code&gt;password_file&lt;/code&gt; and &lt;code&gt;tls_key_file&lt;/code&gt; still have no permission check, because adding one would stop installs that work today from starting.&lt;/p&gt;&#xA;&lt;h2 id=&#34;8-known-limits&#34;&gt;8. Known limits&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests, the rc.d script, the port with &lt;code&gt;make stage check-plist&lt;/code&gt;, the fork tests, and the nginx, HAProxy and pf recipes by hand. Not tested: poudriere, 13.x, 14.x, amd64.&lt;/li&gt;&#xA;&lt;li&gt;HAProxy 3.4.4 on FreeBSD answered 503 for a while after a &lt;code&gt;server-template&lt;/code&gt; slot came up with port 0, also with one name server; the cause is not known, and the same test passes on Linux.&lt;/li&gt;&#xA;&lt;li&gt;The deb and rpm have been installed, upgraded and removed in containers (arm64) without systemd (the deb also under systemd); not tested on x86-64 or the rpm under systemd, and they are not signed or published.&lt;/li&gt;&#xA;&lt;li&gt;Unchanged from earlier releases: peer messages have no forward secrecy; the &lt;code&gt;ldap&lt;/code&gt; plugin has been checked against Active Directory only without a certificate; lookup throughput across threads levels off at about two million a second on Linux; with &lt;code&gt;[store] sync = false&lt;/code&gt; (the default), a host crash can damage the store; a host announces at most 80 services to the peer group.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;9-how-to-upgrade&#34;&gt;9. How to upgrade&#xA;&lt;/h2&gt;&lt;ol&gt;&#xA;&lt;li&gt;Run &lt;code&gt;emfor check-config&lt;/code&gt; as the daemon&amp;rsquo;s user with the 0.6.0 binary on the config of every host before you upgrade it (section 1).&lt;/li&gt;&#xA;&lt;li&gt;Install &lt;code&gt;libemfor&lt;/code&gt; and the daemon from 0.6.0, host by host, clients first if you plan to use &lt;code&gt;drain = true&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;To use the new things: add &lt;code&gt;include_dir&lt;/code&gt; and move services into files, try &lt;code&gt;emfor lookup --explain&lt;/code&gt; on a host, and read chapter 7b for the proxy or tool you run.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The &lt;a class=&#34;link&#34; href=&#34;https://embark.systems/en/emfor/&#34; &gt;user manual&lt;/a&gt; on this site is updated to 0.6.0. The full list of changes is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/CHANGELOG.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;CHANGELOG&lt;/a&gt;, and the source is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/releases/tag/v0.6.0&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;v0.6.0 release on GitHub&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h3 id=&#34;-technical-collaboration-and-architecture-consulting&#34;&gt;💡 Technical collaboration and architecture consulting&#xA;&lt;/h3&gt;&lt;p&gt;Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends?&#xA;We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.&lt;/p&gt;&#xA;&lt;p&gt;📧 &lt;a class=&#34;link&#34; href=&#34;mailto:service@embark.systems&#34; &gt;Book a technical consultation&lt;/a&gt;&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>emfor 0.5.0: A Second Encrypted Peer Format, Stricter Config Validation and a Round of Stability Fixes</title>
            <link>https://embark.systems/en/posts/emfor-0.5.0/</link>
            <pubDate>Sun, 15 Nov 2026 15:00:00 +0800</pubDate>
            <guid>https://embark.systems/en/posts/emfor-0.5.0/</guid>
            <description>&lt;img src=&#34;https://embark.systems/images/emfor-0.5.0.png&#34; alt=&#34;Featured image of post emfor 0.5.0: A Second Encrypted Peer Format, Stricter Config Validation and a Round of Stability Fixes&#34; /&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;emfor&lt;/a&gt; is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.&lt;/p&gt;&#xA;&lt;p&gt;0.5.0 was released on 2026-11-15. 0.4.0 changed which backend an application gets back; this release is about what happens around it when something goes wrong: a store cut short by a power loss, a plugin that stops answering, a disk that is slow to commit, a DNS port that anyone can reach, a forked child that hangs on FreeBSD. It also adds a second encrypted format for peer groups, and &lt;code&gt;emfor check-config&lt;/code&gt; now refuses more options that could never work.&lt;/p&gt;&#xA;&lt;h2 id=&#34;1-before-you-upgrade-three-things-that-can-affect-a-running-setup&#34;&gt;1. Before you upgrade: three things that can affect a running setup&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;1. A config that 0.4.0 accepted can now be refused.&lt;/strong&gt; &lt;code&gt;emfor check-config&lt;/code&gt;, and so every start and reload, refuses options that could never work and that 0.4.0 let through, to fail later. The ones you are most likely to meet:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;interval&lt;/code&gt; above &lt;code&gt;24h&lt;/code&gt; or &lt;code&gt;timeout&lt;/code&gt; below &lt;code&gt;10ms&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;an http &lt;code&gt;host_header&lt;/code&gt; that is not a host with an optional port (spaces, a path or a scheme), and an http &lt;code&gt;path&lt;/code&gt; with &lt;code&gt;#&lt;/code&gt;, &lt;code&gt;\&lt;/code&gt; or a control character;&lt;/li&gt;&#xA;&lt;li&gt;with &lt;code&gt;[dns]&lt;/code&gt;, a service name too long to leave room for the SRV target names of its backends;&lt;/li&gt;&#xA;&lt;li&gt;an exec or plugin &lt;code&gt;command&lt;/code&gt; with a NUL, and a &lt;code&gt;ca_file&lt;/code&gt; with no certificate usable as a CA;&lt;/li&gt;&#xA;&lt;li&gt;in &lt;code&gt;emfor-check&lt;/code&gt;: a NUL in a mysql, pgsql or amqp user or password, an amqp password too long for AMQP&amp;rsquo;s first frame, a diameter option list too long for one message, a sip &lt;code&gt;uri&lt;/code&gt; or &lt;code&gt;from&lt;/code&gt; with &lt;code&gt;&amp;lt;&lt;/code&gt; or &lt;code&gt;&amp;gt;&lt;/code&gt;, and a mongodb &lt;code&gt;password_file&lt;/code&gt; whose password is not printable ASCII.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Run &lt;code&gt;emfor check-config&lt;/code&gt; with the new binary on each host&amp;rsquo;s config before you upgrade that host.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;2. The DNS server limits UDP responses per client network.&lt;/strong&gt; By default each IPv4 /24 (IPv6 /56) gets 100 responses a second (&lt;code&gt;[dns] rate_limit&lt;/code&gt;, &lt;code&gt;0&lt;/code&gt; turns it off). An SRV answer can be many times the size of its query, so a server that other hosts can reach could be used to flood a third party with forged-source queries. Past the limit every other response is an empty truncated one, which makes a real client retry over TCP, and the rest are dropped; &lt;code&gt;emfor_dns_rate_limited_total&lt;/code&gt; counts them. Loopback clients and TCP are never limited. If the resolvers that forward your zone send more than 100 queries a second from one network, raise the limit. At start the daemon also warns about every &lt;code&gt;[dns] listen&lt;/code&gt; address that is not a loopback address.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;3. &lt;code&gt;libemfor&lt;/code&gt; is named for 0.5&lt;/strong&gt;: &lt;code&gt;libemfor.so.0.5&lt;/code&gt; and &lt;code&gt;libemfor.0.5.dylib&lt;/code&gt;. As with every release, install &lt;code&gt;libemfor&lt;/code&gt; from the same release as the daemon; the Python package looks for the file named for its own release first.&lt;/p&gt;&#xA;&lt;p&gt;The store format (version 2) and the plugin protocol (1) are unchanged.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-peer-groups&#34;&gt;2. Peer groups&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;A second encrypted format, &lt;code&gt;EMP3&lt;/code&gt;.&lt;/strong&gt; 0.4.0 encrypts peer messages with ChaCha20-Poly1305 (&lt;code&gt;EMP2&lt;/code&gt;) and a random 12-byte nonce per message. That is safe for about 2^32 messages under one secret; by an estimate, not a measurement, a large group can send that many within days, and a repeated nonce would let an observer read and forge messages. &lt;code&gt;EMP3&lt;/code&gt; uses XChaCha20-Poly1305 with a 24-byte random nonce and a key of its own, which has no practical limit.&lt;/p&gt;&#xA;&lt;p&gt;The format is chosen per member. Every &lt;code&gt;EMP2&lt;/code&gt; message of 0.5.0 carries &lt;code&gt;&amp;quot;emp3&amp;quot;: true&lt;/code&gt; inside its encrypted payload, and a member is sent &lt;code&gt;EMP3&lt;/code&gt; once the latest message accepted from it is &lt;code&gt;EMP3&lt;/code&gt; or such an &lt;code&gt;EMP2&lt;/code&gt;; otherwise it is sent &lt;code&gt;EMP2&lt;/code&gt;. So 0.5.0 members use &lt;code&gt;EMP3&lt;/code&gt; between themselves and &lt;code&gt;EMP2&lt;/code&gt; with 0.4.0 members, and a group upgrades, or goes back, in any order without a configuration change. A host that goes back to 0.4.0 drops the &lt;code&gt;EMP3&lt;/code&gt; messages for about its first second. The &lt;code&gt;EMP3&lt;/code&gt; bytes are checked against a second implementation, written separately, which itself passes the test vectors of the XChaCha20 draft.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;More in the peer group:&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;[peers] accept_plaintext = false&lt;/code&gt; drops every unencrypted &lt;code&gt;EMP1&lt;/code&gt; message without checking it, counted under the new reason &lt;code&gt;plaintext&lt;/code&gt;, so a group whose members all encrypt no longer has to accept signed plaintext. It needs &lt;code&gt;encrypt = true&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;New &lt;code&gt;[peers] previous_secret_file&lt;/code&gt;: the secret &lt;code&gt;secret_file&lt;/code&gt; held before its last change. A daemon restarted within 120 seconds of that change keeps signing with and accepting the old secret as if it had not restarted; before, members whose file had not changed yet rejected it as &lt;code&gt;auth&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;A second change of &lt;code&gt;secret_file&lt;/code&gt; within 60 seconds of the first no longer signs for a minute with the intermediate secret, which members whose file had not changed did not know.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A voting fix.&lt;/strong&gt; A member kept another member&amp;rsquo;s latest result for 3 of the sender&amp;rsquo;s intervals and counted it at every one of its own check times in that span. With &lt;code&gt;interval = &amp;quot;1s&amp;quot;&lt;/code&gt; here and &lt;code&gt;&amp;quot;30s&amp;quot;&lt;/code&gt; at the assigned checkers, one &lt;code&gt;down&lt;/code&gt; from each of two checkers could move &lt;code&gt;fall&lt;/code&gt; up to about 90 times, and mark down a backend that the checkers themselves still saw healthy. Now each result counts at one check time.&lt;/li&gt;&#xA;&lt;li&gt;Messages that arrive out of order are accepted once instead of being dropped as &lt;code&gt;replay&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;3-staying-up-when-something-goes-wrong&#34;&gt;3. Staying up when something goes wrong&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;A store cut short no longer ends the process with &lt;code&gt;SIGBUS&lt;/code&gt;.&lt;/strong&gt; A store file shorter than LMDB records it (changed in place, or its tail lost in a power loss) is moved to &lt;code&gt;&amp;lt;path&amp;gt;.corrupt&lt;/code&gt; at start like other damage, with a warning that says &lt;code&gt;truncated&lt;/code&gt;. Before, reading it to check it ended the daemon with &lt;code&gt;SIGBUS&lt;/code&gt;, again at every restart. &lt;code&gt;emfor status&lt;/code&gt; prints &lt;code&gt;cannot open store: truncated&lt;/code&gt; and exits 2, and a client fails the lookup with &lt;code&gt;UNAVAILABLE&lt;/code&gt; instead of ending the application. After every commit the daemon now extends the file to the length LMDB records, so a healthy store is never taken for a truncated one. A store changed in place while it is open can still do this to a running client: replace it with &lt;code&gt;mv&lt;/code&gt;, never &lt;code&gt;cp&lt;/code&gt; over it or &lt;code&gt;truncate&lt;/code&gt; it.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A plugin that stops answering is restarted.&lt;/strong&gt; Until now only a plugin that exited or closed its stdout was restarted; one that kept running but answered nothing left every check of its backends &lt;code&gt;error&lt;/code&gt; for good, and the requests for it queued up without bound. When the callers of 3 requests in a row give up and no answer arrives in between, the daemon logs it, kills the plugin and restarts it. A plugin that answers within &lt;code&gt;timeout_ms&lt;/code&gt; is never affected.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A slow commit no longer freezes DNS answers and &lt;code&gt;/metrics&lt;/code&gt;.&lt;/strong&gt; While the state store waited for the disk, the daemon stopped applying check results, so DNS kept answering with old states. Results are now applied as they arrive; the changes made meanwhile are written together by the next commit.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Bounded queues.&lt;/strong&gt; A peer task that falls more than 4096 messages behind drops new &lt;code&gt;down&lt;/code&gt; and &lt;code&gt;result&lt;/code&gt; messages (counted in &lt;code&gt;emfor_peer_queue_dropped_total&lt;/code&gt;); a plugin that stops reading its stdin gets at most 1024 waiting requests, and further checks are &lt;code&gt;error&lt;/code&gt; at once with &lt;code&gt;plugin &amp;lt;name&amp;gt; busy&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;DNS.&lt;/strong&gt; TCP serves at most 128 connections and closes one with no query for 5 seconds; if a DNS listener stops, the daemon logs it and exits with status 70 so the service manager restarts it, where before DNS went silent; BADVERS for EDNS versions above 0 and REFUSED for classes other than IN.&lt;/li&gt;&#xA;&lt;li&gt;A single key or value in the store that is not UTF-8 no longer blocks the daemon; a reload that changes &lt;code&gt;interval&lt;/code&gt; applies to the wait already under way (going from &lt;code&gt;10m&lt;/code&gt; to &lt;code&gt;2s&lt;/code&gt; used to take up to 11 minutes); a client keeps reading after the daemon is restarted with a larger &lt;code&gt;[store] map_size&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;4-emfor-check-hangs-and-crashes-with-a-badly-behaved-server&#34;&gt;4. emfor-check: hangs and crashes with a badly behaved server&#xA;&lt;/h2&gt;&lt;p&gt;A review of every plugin against servers that answer wrongly found a number of hangs and crashes. All are now &lt;code&gt;down&lt;/code&gt; or &lt;code&gt;error&lt;/code&gt; with a reason, and the check&amp;rsquo;s timeout ends them:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;mongodb&lt;/code&gt;: a server asking for 4294967295 SCRAM iterations held a worker thread for many minutes (more than 1000000 is now refused), and a &lt;code&gt;hello&lt;/code&gt; reply nested about 8000 deep overflowed the stack and aborted the plugin;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sip&lt;/code&gt; over TCP or TLS: a Content-Length near the largest 64-bit number made the plugin loop forever at full CPU on the same provisional response;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;redis&lt;/code&gt;: a reply over 2 MiB; &lt;code&gt;mysql&lt;/code&gt;: a NULL answer to &lt;code&gt;SELECT 1&lt;/code&gt; panicked the check, and a check cut off by its timeout leaked a task and a socket every time;&lt;/li&gt;&#xA;&lt;li&gt;a reason longer than 512 characters is cut before it is sent: a 70 KB redis &lt;code&gt;-ERR&lt;/code&gt; made the answer line longer than the daemon&amp;rsquo;s limit, so the daemon restarted the plugin at every check of that backend;&lt;/li&gt;&#xA;&lt;li&gt;a check that panics is answered &lt;code&gt;error&lt;/code&gt; with &lt;code&gt;check panicked: &amp;lt;message&amp;gt;&lt;/code&gt; instead of no answer.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The &lt;code&gt;mysql&lt;/code&gt; plugin&amp;rsquo;s live tests now also run against MySQL 8.0 and 8.4 (&lt;code&gt;tests/plugins/mysql.sh&lt;/code&gt;), and pass; before, only MariaDB was checked.&lt;/p&gt;&#xA;&lt;h2 id=&#34;5-clients-and-a-hang-on-freebsd&#34;&gt;5. Clients, and a hang on FreeBSD&#xA;&lt;/h2&gt;&lt;p&gt;The clients fail a lookup on a truncated store and keep reading after a store grows (above), try a store that exists but cannot be opened at most once a second, and return a status instead of aborting when &lt;code&gt;emfor_lookup&lt;/code&gt; is called from a thread-local destructor. A backend &lt;code&gt;state&lt;/code&gt; or &lt;code&gt;on_all_down&lt;/code&gt; value written by a newer daemon no longer makes the lookup fail with &lt;code&gt;INCOMPATIBLE&lt;/code&gt;: an unknown state counts as &lt;code&gt;unhealthy&lt;/code&gt;, an unknown &lt;code&gt;on_all_down&lt;/code&gt; as &lt;code&gt;open&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A forked child could hang for ever on FreeBSD.&lt;/strong&gt; Our fork test, in which other threads look up while the process forks, failed in every run on FreeBSD 15.1 (arm64): a child waited for ever in its first lookup, within the first few dozen of 300 forks. &lt;strong&gt;0.4.0 has the same hang there&lt;/strong&gt;: built from its tag with the same tests, the Rust test failed in 10 of 10 runs and the C test in 5 of 5.&lt;/p&gt;&#xA;&lt;p&gt;The cause is in FreeBSD&amp;rsquo;s threading library, libthr, not in emfor. LMDB keeps its reader mutex in the store&amp;rsquo;s lock file, as a process-shared mutex, and libthr looks such mutexes up behind a lock of its own. Its &lt;code&gt;fork&lt;/code&gt; takes that lock only for reading, so the read hold of a thread that is inside such a mutex at that moment stays in the child, where nothing releases it; the child&amp;rsquo;s first use of a newly mapped mutex then waits for it. On a hung child we read the lock: one reader left and one writer waiting. libthr has a second problem next to this one: a parent that forks while another thread maps a process-shared mutex for the first time can deadlock.&lt;/p&gt;&#xA;&lt;p&gt;The client now works around both: on FreeBSD it makes &lt;code&gt;fork&lt;/code&gt; wait until no lookup is inside the store (&lt;code&gt;pthread_atfork&lt;/code&gt; handlers), and &lt;code&gt;libemfor&lt;/code&gt; can no longer be unloaded there, because the handlers stay registered. Lookup throughput did not change (measured with 1, 4 and 8 threads). It does not cover other users of process-shared mutexes in the same process, a &lt;code&gt;fork&lt;/code&gt; from a signal handler that interrupted a lookup of the same thread, or a &lt;code&gt;libemfor.a&lt;/code&gt; linked statically into a module that is unloaded.&lt;/p&gt;&#xA;&lt;p&gt;We also wrote a patch for libthr and put it, with a small reproduction that does not use LMDB, in &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.5.0/docs/freebsd/libthr-pshared-fork.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/freebsd/&lt;/a&gt;. We built libthr from the 15.1 source twice, unpatched and patched: unpatched, 40 to 55 children of 300 hang in each of 5 runs and the deadlocking parent never finishes; patched, no child hangs in 5 of 5 runs and the parent finishes. An LMDB client without our workaround passes on the patched library. We tried it on FreeBSD 15.1 (arm64) only, and have not reported it to FreeBSD yet.&lt;/p&gt;&#xA;&lt;h2 id=&#34;6-other-changes&#34;&gt;6. Other changes&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;emfor status&lt;/code&gt;: a truncated store, &lt;code&gt;emfor status | head -1&lt;/code&gt; keeping its exit status, &lt;code&gt;--watch&lt;/code&gt; opening the store again when the file is replaced, and records from a newer daemon being shown instead of failing.&lt;/li&gt;&#xA;&lt;li&gt;The example systemd unit sets &lt;code&gt;LimitNOFILE=65536&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Every fuzz target runs for 10 minutes before a release. For 0.5.0 the six targets (plugin answers, peer messages, the DNS server, metrics, the plugin protocol and the config; the DNS one is new) ran about 16 million inputs between them without a crash. They now keep overflow checks on and run every input on a 2 MiB stack like a tokio worker&amp;rsquo;s, so an overflow or a too-deep recursion is a crash, as it would be in the daemon.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;a class=&#34;link&#34; href=&#34;https://embark.systems/en/emfor/&#34; &gt;user manual&lt;/a&gt; on this site is updated to 0.5.0.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;7-known-limits&#34;&gt;7. Known limits&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;FreeBSD has been tested on 15.1-RELEASE (arm64, in a VM): the tests, the rc.d script, the port with &lt;code&gt;make stage check-plist install&lt;/code&gt;, and the fork tests. Not tested: poudriere, 13.x, 14.x, amd64.&lt;/li&gt;&#xA;&lt;li&gt;The FreeBSD fork guard does not cover the cases above, and the libthr patch has been tried on 15.1 only.&lt;/li&gt;&#xA;&lt;li&gt;Peer messages have no forward secrecy: whoever later gets &lt;code&gt;secret_file&lt;/code&gt; can read recorded messages. Rotate the secret regularly.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;code&gt;ldap&lt;/code&gt; plugin has been checked against Active Directory only without a certificate and without a successful login.&lt;/li&gt;&#xA;&lt;li&gt;Unchanged from earlier releases: lookup throughput across threads levels off at about two million a second on Linux; with &lt;code&gt;[store] sync = false&lt;/code&gt; (the default), a host crash can leave the store damaged and the state history is lost; a host announces at most 80 services left out of the peer group.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;8-how-to-upgrade&#34;&gt;8. How to upgrade&#xA;&lt;/h2&gt;&lt;ol&gt;&#xA;&lt;li&gt;Run &lt;code&gt;emfor check-config&lt;/code&gt; with the 0.5.0 binary on the config of every host before you upgrade it (section 1).&lt;/li&gt;&#xA;&lt;li&gt;Install &lt;code&gt;libemfor&lt;/code&gt; and the daemon from 0.5.0, host by host. A peer group of 0.4.0 and 0.5.0 members works in any order, with no change to &lt;code&gt;[peers]&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If your DNS clients send many queries from one network, watch &lt;code&gt;emfor_dns_rate_limited_total&lt;/code&gt; after the upgrade; if it grows, raise &lt;code&gt;[dns] rate_limit&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The full list of changes is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.5.0/CHANGELOG.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;CHANGELOG&lt;/a&gt;, and the source is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/releases/tag/v0.5.0&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;v0.5.0 release on GitHub&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h3 id=&#34;-technical-collaboration-and-architecture-consulting&#34;&gt;💡 Technical collaboration and architecture consulting&#xA;&lt;/h3&gt;&lt;p&gt;Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends?&#xA;We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.&lt;/p&gt;&#xA;&lt;p&gt;📧 &lt;a class=&#34;link&#34; href=&#34;mailto:service@embark.systems&#34; &gt;Book a technical consultation&lt;/a&gt;&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>emfor 0.4.0: Backup Backends, Slow Start, Encrypted Peer Groups and Five New Plugins</title>
            <link>https://embark.systems/en/posts/emfor-0.4.0/</link>
            <pubDate>Sat, 14 Nov 2026 21:30:00 +0800</pubDate>
            <guid>https://embark.systems/en/posts/emfor-0.4.0/</guid>
            <description>&lt;img src=&#34;https://embark.systems/images/emfor-0.4.0.png&#34; alt=&#34;Featured image of post emfor 0.4.0: Backup Backends, Slow Start, Encrypted Peer Groups and Five New Plugins&#34; /&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;emfor&lt;/a&gt; is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use and writes which ones are healthy into a local file; applications ask a client library which backend to connect to and get an answer in about a microsecond, without a network round trip.&lt;/p&gt;&#xA;&lt;p&gt;0.4.0 was released on 2026-11-14. Its focus is &lt;strong&gt;the address an application gets back&lt;/strong&gt;: when to fall back to a backup, whether a backend that just recovered should take its full share at once, and how quickly to step around a backend that starts to fail. Peer group messages are now encrypted, and &lt;code&gt;emfor-check&lt;/code&gt; speaks five more protocols.&lt;/p&gt;&#xA;&lt;h2 id=&#34;1-before-you-upgrade-three-breaking-changes&#34;&gt;1. Before you upgrade: three breaking changes&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;1. The store format is version 2.&lt;/strong&gt; The daemon writes &lt;code&gt;meta/schema_version&lt;/code&gt; 2, which adds the backend field &lt;code&gt;backup&lt;/code&gt;. Clients and &lt;code&gt;emfor status&lt;/code&gt; from 0.4.0 read versions 1 and 2; older clients fail every lookup on a version 2 store with &lt;code&gt;INCOMPATIBLE&lt;/code&gt;. So &lt;strong&gt;upgrade the clients on a host (&lt;code&gt;libemfor&lt;/code&gt; and the language packages) before the daemon&lt;/strong&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;2. Peer messages are encrypted by default.&lt;/strong&gt; Members still on 0.3.0 cannot read them and drop them as &lt;code&gt;malformed&lt;/code&gt;. To upgrade a group host by host, add &lt;code&gt;encrypt = false&lt;/code&gt; to &lt;code&gt;[peers]&lt;/code&gt; on each host as you upgrade it; once every member runs 0.4.0, remove the line and reload.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;3. FreeBSD: &lt;code&gt;examples/rc.d/emfor&lt;/code&gt; is gone.&lt;/strong&gt; The only rc.d script is now the port&amp;rsquo;s &lt;code&gt;examples/freebsd/files/emfor.in&lt;/code&gt;; without the port, fill in &lt;code&gt;%%PREFIX%%&lt;/code&gt; as chapter 2 of the manual shows and install it. Its &lt;code&gt;emfor_user&lt;/code&gt; defaults to &lt;code&gt;emfor&lt;/code&gt; rather than &lt;code&gt;root&lt;/code&gt;. The example port also keeps the store in &lt;code&gt;/var/db/emfor&lt;/code&gt;, where FreeBSD keeps daemon state, so applications on the host set &lt;code&gt;EMFOR_STORE=/var/db/emfor/state.mdb&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-choosing-a-backend-more-carefully&#34;&gt;2. Choosing a backend more carefully&#xA;&lt;/h2&gt;&lt;h3 id=&#34;backup-backends&#34;&gt;Backup backends&#xA;&lt;/h3&gt;&lt;p&gt;A backend can be marked as a backup:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;db.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;balance&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;failover&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.1.5:5432&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.1.6:5432&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.1.1.5:5432&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;backup&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt; },   &lt;span style=&#34;color:#75715e&#34;&gt;# another data centre&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;As long as any backend that is not a backup passes its checks, the backups are not used. The order is: primaries whose last check passed → backups whose last check passed → healthy primaries → healthy backups, and &lt;code&gt;balance&lt;/code&gt; works within the chosen set. The typical use is a backend in another data centre, or a more expensive one. SRV answers list only the chosen set.&lt;/p&gt;&#xA;&lt;h3 id=&#34;step-around-a-backend-at-its-first-failed-check&#34;&gt;Step around a backend at its first failed check&#xA;&lt;/h3&gt;&lt;p&gt;A backend has to fail &lt;code&gt;fall&lt;/code&gt; checks in a row before it turns &lt;code&gt;unhealthy&lt;/code&gt;, about 6 seconds by default. Until then it is still healthy, and 0.3.0 clients kept connecting to it.&lt;/p&gt;&#xA;&lt;p&gt;0.4.0 clients and the DNS server leave out a healthy backend whose last check failed, as long as another healthy backend passed its last check. In other words, &lt;strong&gt;a backend stops being chosen at its first failed check&lt;/strong&gt; instead of after &lt;code&gt;fall&lt;/code&gt; of them; its state still follows &lt;code&gt;rise&lt;/code&gt; and &lt;code&gt;fall&lt;/code&gt;, and it comes back once it passes. When every healthy backend is failing, all of them are used as before, so nothing is refused because of this.&lt;/p&gt;&#xA;&lt;p&gt;With &lt;code&gt;on_all_down = &amp;quot;open&amp;quot;&lt;/code&gt; and no healthy backend, there is a similar change: the &lt;code&gt;unknown&lt;/code&gt; backends (not decided yet, or whose checks only give errors) are used before the &lt;code&gt;unhealthy&lt;/code&gt; ones that failed &lt;code&gt;fall&lt;/code&gt; checks.&lt;/p&gt;&#xA;&lt;h3 id=&#34;slow-start&#34;&gt;Slow start&#xA;&lt;/h3&gt;&lt;p&gt;A backend that has just recovered has a cold cache and an empty connection pool, and its full share of traffic at once often knocks it over again. The new &lt;code&gt;slow_start&lt;/code&gt; gives it a growing share over a set time, in 10 steps:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;defaults&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;slow_start&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;30s&amp;#34;&lt;/span&gt;     &lt;span style=&#34;color:#75715e&#34;&gt;# default &amp;#34;0s&amp;#34;, off&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It applies to client lookups and DNS A/AAAA answers for every method but &lt;code&gt;failover&lt;/code&gt; and &lt;code&gt;sticky&lt;/code&gt; with a key, and to SRV weights.&lt;/p&gt;&#xA;&lt;h3 id=&#34;sticky-honours-weight&#34;&gt;Sticky honours weight&#xA;&lt;/h3&gt;&lt;p&gt;&lt;code&gt;balance = &amp;quot;sticky&amp;quot;&lt;/code&gt; with an affinity key now follows &lt;code&gt;weight&lt;/code&gt;: a backend of weight w gets w shares of the keys. The computation uses only integers and MD5, so every host and every language still picks the same backend for a key. With every weight 1 nothing changes; a sticky service that already sets other weights moves some keys when its clients are upgraded, so upgrade the clients of such a service together.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-peer-groups&#34;&gt;3. Peer groups&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Encrypted messages.&lt;/strong&gt; 0.3.0 signed peer messages but sent them in the clear, backend names, addresses and reasons included. 0.4.0 sends &lt;code&gt;EMP2&lt;/code&gt;: a key derived from &lt;code&gt;secret_file&lt;/code&gt; with HKDF-SHA256, ChaCha20-Poly1305, and a random nonce per message. Secret rotation works as before. There is no forward secrecy: whoever later gets the secret can read recorded messages, so rotate it regularly.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;A failure only a non-assigned member sees is noticed sooner.&lt;/strong&gt; The known limit of 0.3.0 was that a failure only a member which is not one of a backend&amp;rsquo;s assigned checkers sees took up to about five intervals plus &lt;code&gt;fall&lt;/code&gt; checks to notice. In 0.4.0 such a member checks the backend itself at least every third interval, and when its own check fails while the assigned checkers say up, it starts a fast recheck run (every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks), as it does after a peer&amp;rsquo;s &lt;code&gt;down&lt;/code&gt; report. Measured over 12 failures with &lt;code&gt;interval = &amp;quot;2s&amp;quot;&lt;/code&gt; and &lt;code&gt;fall = 3&lt;/code&gt;, that host marked the backend down after 0.8 to 5.8 seconds, 3.3 seconds median; 0.3.0 took up to about 14 seconds by the schedule. In exchange a backend gets &lt;code&gt;3 + (N − 3) / 3&lt;/code&gt; checks per interval instead of &lt;code&gt;3 + (N − 3) / 5&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-emfor-check-five-new-plugins&#34;&gt;4. emfor-check: five new plugins&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor-check&lt;/code&gt; is the plugin that ships with emfor for services whose checks need to speak their protocol. 0.4.0 adds:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Plugin&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What it checks&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;smtp&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The 220 greeting and &lt;code&gt;EHLO&lt;/code&gt;, optionally with &lt;code&gt;STARTTLS&lt;/code&gt; or TLS on port 465; it never logs in or sends mail&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;amqp&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;AMQP 0-9-1 brokers such as RabbitMQ: a &lt;code&gt;PLAIN&lt;/code&gt; login and opening &lt;code&gt;vhost&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;mongodb&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;hello&lt;/code&gt; on MongoDB 4.4 and later, with an optional SCRAM-SHA-256 login; &lt;code&gt;role = &amp;quot;primary&amp;quot;&lt;/code&gt; lets &lt;code&gt;failover&lt;/code&gt; follow the replica set&amp;rsquo;s primary&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;grpc&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The standard &lt;code&gt;grpc.health.v1.Health/Check&lt;/code&gt;; up only on &lt;code&gt;SERVING&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;quic&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A QUIC version 1 handshake; with &lt;code&gt;http3 = true&lt;/code&gt; also an HTTP/3 &lt;code&gt;GET&lt;/code&gt;, judged on status and body like the built-in &lt;code&gt;http&lt;/code&gt; check&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Database checks are stricter too: &lt;code&gt;role = &amp;quot;replica&amp;quot;&lt;/code&gt; now means &lt;strong&gt;a replica that is replicating&lt;/strong&gt;: a redis replica whose link to its master is &lt;code&gt;connected&lt;/code&gt;, a MySQL or MariaDB replica whose IO and SQL threads both run, a PostgreSQL standby with a WAL receiver. With &lt;code&gt;max_lag&lt;/code&gt; you can bound how far behind a replica may be. Note that a redis replica that was up with &lt;code&gt;role = &amp;quot;replica&amp;quot;&lt;/code&gt; in 0.3.0 while its link was down is down in 0.4.0.&lt;/p&gt;&#xA;&lt;p&gt;LDAP reasons read better against Active Directory: the NUL byte AD puts at the end of its diagnostic message is gone, and a server that drops the connection during the TLS handshake (as AD does without an LDAPS certificate) is down with &lt;code&gt;tls: connection closed during the TLS handshake (does the server have a certificate?)&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;5-other-changes&#34;&gt;5. Other changes&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;The built-in &lt;code&gt;http&lt;/code&gt; check takes &lt;code&gt;expect_body&lt;/code&gt;: text the response body must contain, for health endpoints that answer 200 either way.&lt;/li&gt;&#xA;&lt;li&gt;On macOS, &lt;code&gt;exec&lt;/code&gt; checks and plugins no longer inherit the daemon&amp;rsquo;s sockets and pipes.&lt;/li&gt;&#xA;&lt;li&gt;FreeBSD: the rc.d script starts &lt;code&gt;daemon(8)&lt;/code&gt; with &lt;code&gt;-f&lt;/code&gt;, so the SSH session that runs &lt;code&gt;service emfor start&lt;/code&gt; no longer hangs. The daemon, the rc.d script, the port and the tests have run on FreeBSD 15.1-RELEASE (arm64) for the first time.&lt;/li&gt;&#xA;&lt;li&gt;Every fuzz target runs for 10 minutes before a release. For 0.4.0 the five targets (plugin answers, peer messages, metrics, the plugin protocol and the config) ran about 18 million inputs between them without a crash.&lt;/li&gt;&#xA;&lt;li&gt;The user manual is now on this site too: &lt;a class=&#34;link&#34; href=&#34;https://embark.systems/en/emfor/&#34; &gt;emfor User Manual&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;6-known-limits&#34;&gt;6. Known limits&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;FreeBSD has been tested on 15.1-RELEASE arm64 only; not with poudriere, on 14.x or on amd64.&lt;/li&gt;&#xA;&lt;li&gt;The &lt;code&gt;ldap&lt;/code&gt; plugin has been checked against Active Directory only without a certificate and without a successful login; the &lt;code&gt;mysql&lt;/code&gt; plugin, at release, only against MariaDB (since checked against MySQL 8.0 and 8.4, which pass).&lt;/li&gt;&#xA;&lt;li&gt;With &lt;code&gt;[store] sync = false&lt;/code&gt; (the default), a host crash can leave the store damaged; the daemon then sets it aside at the next start and the state history is lost. Chapter 8 of the manual shows how to keep the store on a tmpfs.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;7-how-to-upgrade&#34;&gt;7. How to upgrade&#xA;&lt;/h2&gt;&lt;ol&gt;&#xA;&lt;li&gt;On every host, upgrade the clients first: &lt;code&gt;libemfor&lt;/code&gt; and the language packages from 0.4.0.&lt;/li&gt;&#xA;&lt;li&gt;Upgrade and restart the daemon host by host. With a peer group, add &lt;code&gt;encrypt = false&lt;/code&gt; to &lt;code&gt;[peers]&lt;/code&gt; on each host as you upgrade it (0.3.0 does not know the key, so do not add it ahead).&lt;/li&gt;&#xA;&lt;li&gt;Once every member runs 0.4.0, remove &lt;code&gt;encrypt = false&lt;/code&gt; and reload each daemon.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;The full list of changes is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.4.0/CHANGELOG.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;CHANGELOG&lt;/a&gt;, and the source is in the &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/releases/tag/v0.4.0&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;v0.4.0 release on GitHub&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h3 id=&#34;-technical-collaboration-and-architecture-consulting&#34;&gt;💡 Technical collaboration and architecture consulting&#xA;&lt;/h3&gt;&lt;p&gt;Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends?&#xA;We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.&lt;/p&gt;&#xA;&lt;p&gt;📧 &lt;a class=&#34;link&#34; href=&#34;mailto:service@embark.systems&#34; &gt;Book a technical consultation&lt;/a&gt;&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>emfor Peer Groups: Health Checks That Help Each Other Within a Data Centre</title>
            <link>https://embark.systems/en/posts/emfor-peer-groups/</link>
            <pubDate>Fri, 13 Nov 2026 18:00:00 +0800</pubDate>
            <guid>https://embark.systems/en/posts/emfor-peer-groups/</guid>
            <description>&lt;img src=&#34;https://embark.systems/images/emfor-peer-vote.png&#34; alt=&#34;Featured image of post emfor Peer Groups: Health Checks That Help Each Other Within a Data Centre&#34; /&gt;&#xA;    &lt;blockquote&gt;&#xA;        &lt;p&gt;&lt;strong&gt;Updated 2026-11-14:&lt;/strong&gt; revised for emfor 0.4.0. Since 0.4.0, a member that is not an assigned checker makes its own check at least every third &lt;code&gt;interval&lt;/code&gt; (every fifth in 0.3.0) and starts a fast recheck run when its own check fails; peer messages are now encrypted. See &lt;a class=&#34;link&#34; href=&#34;https://embark.systems/en/posts/emfor-0.4.0/&#34; &gt;the emfor 0.4.0 release notes&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;    &lt;/blockquote&gt;&#xA;&lt;h2 id=&#34;1-what-emfor-is&#34;&gt;1. What emfor is&#xA;&lt;/h2&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;emfor&lt;/a&gt; is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use (TCP, HTTP, Nagios-style scripts, and plugins for redis, MySQL, PostgreSQL, LDAP and more) and writes which backends are healthy to an LMDB file on the same host. When an application needs a database server, it asks the client library for an address; the library reads the local file and, following the service&amp;rsquo;s &lt;code&gt;balance&lt;/code&gt; setting, returns an answer such as &lt;code&gt;10.0.3.1:5432&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;              checks over the network&#xA;  backends &amp;lt;------------------------------ emfor daemon ---------&amp;gt; DNS on 127.0.0.1&#xA;                                               |  writes&#xA;                                               v&#xA;                                     local LMDB state store&#xA;                                               |  reads&#xA;                                       client library&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The key point of this design is that &lt;strong&gt;a lookup never touches the network&lt;/strong&gt;. It takes about a microsecond and keeps working when other hosts or the network fail. Each host uses its own config and sees the backends from where it stands; there is no central server.&lt;/p&gt;&#xA;&lt;p&gt;But &amp;ldquo;everyone looks for themselves&amp;rdquo; has a cost.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-three-problems-with-checking-alone&#34;&gt;2. Three problems with checking alone&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;1. Failures are noticed slowly.&lt;/strong&gt; A backend is marked unhealthy only after &lt;code&gt;fall&lt;/code&gt; consecutive failed checks. With the defaults, &lt;code&gt;interval = 2s&lt;/code&gt; and &lt;code&gt;fall = 3&lt;/code&gt;, that is about 6 seconds. With twenty hosts in one data centre, every one of them spends those 6 seconds on its own, although they are all looking at the same broken backend.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;2. What one host sees is not necessarily the truth.&lt;/strong&gt; A failed check may mean the backend is really down, or only that this host&amp;rsquo;s network, firewall rules or load are the problem. The other way round, a host that can reach a backend does not prove that every host can. A single host cannot tell the difference.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;3. The number of checks grows with the number of hosts.&lt;/strong&gt; With N hosts each checking every backend, a backend answers N checks every &lt;code&gt;interval&lt;/code&gt;. With enough hosts, the health checks themselves become a load on the backend.&lt;/p&gt;&#xA;&lt;p&gt;Peer groups are designed for these three problems.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-what-a-peer-group-does&#34;&gt;3. What a peer group does&#xA;&lt;/h2&gt;&lt;p&gt;The emfor daemons of one data centre (&lt;code&gt;site&lt;/code&gt;) that list each other under &lt;code&gt;[peers]&lt;/code&gt; in their config form a peer group. Members talk over signed UDP messages and do three things.&lt;/p&gt;&#xA;&lt;h3 id=&#34;1-failure-reports-someone-noticed-so-i-check-right-away&#34;&gt;1. Failure reports: someone noticed, so I check right away&#xA;&lt;/h3&gt;&lt;p&gt;Every member sends a heartbeat every second, so everyone knows who is live. When a member sees a backend go from &lt;code&gt;healthy&lt;/code&gt; to &lt;code&gt;unhealthy&lt;/code&gt;, it tells every other member. The members that receive the report &lt;strong&gt;do not simply believe it&lt;/strong&gt;: they recheck that backend at once, then every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks, instead of waiting for their next &lt;code&gt;interval&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The result: once one member has noticed a failure, the others confirm it with their own checks and follow within about a second. A report is only a hint that it is time to look; it never changes a backend&amp;rsquo;s state by itself.&lt;/p&gt;&#xA;&lt;h3 id=&#34;2-majority-vote-three-assigned-checkers-vote&#34;&gt;2. Majority vote: three assigned checkers vote&#xA;&lt;/h3&gt;&lt;p&gt;What does change the state is the vote. With at least three live members (this host included), every backend has three &lt;strong&gt;assigned checkers&lt;/strong&gt;. They are chosen by rendezvous hashing: for each candidate member, compute&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;the first 4 bytes of MD5(&amp;#34;&amp;lt;service name&amp;gt;/&amp;lt;backend&amp;gt;/&amp;lt;member address&amp;gt;&amp;#34;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;and take the three largest values. Because every member uses the same formula on the same list of live members, everyone picks the same three without any coordination, and when a member joins or leaves, only a few backends get different assigned checkers.&lt;/p&gt;&#xA;&lt;p&gt;After each of its own checks, an assigned checker sends its result (&lt;code&gt;up&lt;/code&gt; or &lt;code&gt;down&lt;/code&gt;) to the other members. After each of its own checks, every member counts the votes for that backend from its assigned checkers that are recent and come from a member with the same check definition:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Votes&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Result&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Reason reads&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Two or more &lt;code&gt;down&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Down, even if this host&amp;rsquo;s own check passed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;peers 2/3: connect: Connection refused&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fewer than two &lt;code&gt;down&lt;/code&gt;, own check failed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;This host&amp;rsquo;s own result&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;local: connect: Connection refused&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fewer than three live members, or no vote&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;This host&amp;rsquo;s own result, unchanged&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The original reason, no prefix&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The prefix tells an operator at a glance where a verdict came from: &lt;code&gt;peers 2/3:&lt;/code&gt; is the group&amp;rsquo;s consensus, &lt;code&gt;local:&lt;/code&gt; is a problem only this host sees. The latter deserves attention; it usually means something is wrong with this host&amp;rsquo;s own network or firewall.&lt;/p&gt;&#xA;&lt;p&gt;The opposite case is visible too: a backend the group votes down while this host&amp;rsquo;s own check passes is marked &lt;code&gt;local-up&lt;/code&gt; by &lt;code&gt;emfor status&lt;/code&gt; and counted by &lt;code&gt;emfor_peer_disagreements&lt;/code&gt;. &amp;ldquo;I can reach it, the others cannot&amp;rdquo; is often a sign of a network partition or of a firewall rule on the other hosts.&lt;/p&gt;&#xA;&lt;p&gt;&amp;ldquo;The same check definition&amp;rdquo; matters as well. Each service&amp;rsquo;s &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;check_options&lt;/code&gt; and &lt;code&gt;timeout&lt;/code&gt; are hashed into a digest, and only members with the same digest have their votes counted. When two hosts check the same service differently, their results cannot be compared directly; &lt;code&gt;emfor status&lt;/code&gt; then shows &amp;ldquo;check definition differs&amp;rdquo; as a reminder to make the configs match.&lt;/p&gt;&#xA;&lt;h3 id=&#34;3-split-checks-fewer-checks-for-those-who-are-not-assigned&#34;&gt;3. Split checks: fewer checks for those who are not assigned&#xA;&lt;/h3&gt;&lt;p&gt;With a vote in place, members that are not assigned checkers need not check every time. While two assigned checkers agree (two &lt;code&gt;up&lt;/code&gt; or two &lt;code&gt;down&lt;/code&gt; votes), a member that is not one of them takes the vote&amp;rsquo;s result and skips the check. It never lets go completely, though:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it makes its own check at least every third &lt;code&gt;interval&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;it checks every time while there are fewer than two agreeing votes;&lt;/li&gt;&#xA;&lt;li&gt;when its own check fails while the assigned checkers say up, it starts a fast recheck run (every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks), as after a peer&amp;rsquo;s &lt;code&gt;down&lt;/code&gt; report;&lt;/li&gt;&#xA;&lt;li&gt;after its own check fails, it checks every &lt;code&gt;interval&lt;/code&gt; again until a check passes;&lt;/li&gt;&#xA;&lt;li&gt;when the vote says up but this host has not measured a latency yet, it checks once itself, so a backend is never declared healthy without a measurement.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;With three assigned checkers and N members, a backend gets about&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;3 + (N − 3) / 3 checks&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;per &lt;code&gt;interval&lt;/code&gt; instead of N. With 20 hosts, that drops from 20 checks per &lt;code&gt;interval&lt;/code&gt; to about 8.7.&lt;/p&gt;&#xA;&lt;p&gt;The price: a failure that only one host that is not an assigned checker can see (a routing problem that affects only that host, say) is noticed there only at its next own check plus the fast recheck run. Measured over 12 failures with &lt;code&gt;interval = &amp;quot;2s&amp;quot;&lt;/code&gt; and &lt;code&gt;fall = 3&lt;/code&gt;, that took 0.8 to 5.8 seconds, 3.3 seconds median; 0.3.0, which checked itself only every fifth &lt;code&gt;interval&lt;/code&gt; and had no fast run here, took up to about 14 seconds. This is latency traded for fewer checks. &lt;code&gt;emfor status&lt;/code&gt; marks such backends &lt;code&gt;voted&lt;/code&gt;, so operators know which backends this host judges mostly by the vote.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-lines-the-design-does-not-cross&#34;&gt;4. Lines the design does not cross&#xA;&lt;/h2&gt;&lt;p&gt;With peer groups added, emfor&amp;rsquo;s original properties are unchanged:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Lookups still read only the local store.&lt;/strong&gt; What the group exchanges only affects how the daemon decides; the client library does not know the group exists, and a lookup gains no network round trip.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Unreachable members mean a fall back to local judgement.&lt;/strong&gt; When members stop or the network breaks, there are simply no votes, and the daemon goes back to deciding from its own checks, exactly as without a group.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;No master, and no spanning data centres.&lt;/strong&gt; All members are equal, and assigned checkers come from a hash, so there is no election. A group is limited to one &lt;code&gt;site&lt;/code&gt;, because different data centres see different networks and should not vote for each other.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Messages are encrypted and protected against replay.&lt;/strong&gt; Every message is encrypted with ChaCha20-Poly1305 under a key derived from a shared secret (0.3.0 only signed them with HMAC-SHA256 and sent them in the clear). The receiver checks that the sender&amp;rsquo;s address is a member and that the timestamp is within 5 seconds of its own clock, and rejects replayed old messages. The secret can be rotated without downtime: after a host sees its secret file change, it keeps sending with the old secret for 60 seconds and accepts it for 120 seconds.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;5-configuration&#34;&gt;5. Configuration&#xA;&lt;/h2&gt;&lt;p&gt;On every host of the data centre, add:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt;                               &lt;span style=&#34;color:#75715e&#34;&gt;# required with [peers]&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;listen&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;                   &lt;span style=&#34;color:#75715e&#34;&gt;# this host&amp;#39;s address in the group&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;members&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.7:5302&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;secret_file&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/etc/emfor/peers.key&amp;#34;&lt;/span&gt;       &lt;span style=&#34;color:#75715e&#34;&gt;# at least 32 bytes, mode 600 or 640&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;members&lt;/code&gt; lists every member, this host included; the same list can go on every host of the data centre, and only &lt;code&gt;listen&lt;/code&gt; differs. The vote and the split need no settings of their own; they use each service&amp;rsquo;s existing &lt;code&gt;interval&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Some services are not a good fit for a peer group, for example when the check result depends on the host by design. Individual services can stay out:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;local-cache.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;127.0.0.1:11211&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;false&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A service with &lt;code&gt;peers = false&lt;/code&gt; sends no reports or results on this host, ignores the ones it receives, and always decides from its own checks; the other services stay in the group. Changing the setting only takes a reload, and the backends keep their state. The host announces the services it leaves out in its heartbeats (as short hashes of their names), and the other members then leave it out when choosing those services&amp;rsquo; assigned checkers, so with enough live members their backends still have three assigned checkers that vote. Each host can therefore decide on its own which services to leave out. Peer groups and this announcement both came in emfor 0.3.0, so every member of a group reads it.&lt;/p&gt;&#xA;&lt;h2 id=&#34;6-seeing-what-the-group-does&#34;&gt;6. Seeing what the group does&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor status&lt;/code&gt; shows the group under the daemon line:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 4/4 live, checks 12/40 backends here&#xA;peers: check definition of api.svc.emfor differs at 10.0.0.6:5302&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The first line shows which members are live and that, of the 40 backends shown, this host checks 12 itself every time (the rest are marked &lt;code&gt;voted&lt;/code&gt;). &lt;code&gt;emfor status --json&lt;/code&gt; also lists every backend&amp;rsquo;s assigned checkers:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;assigned_checkers&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#960050;background-color:#1e0010&#34;&gt;:&lt;/span&gt; [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  {&lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;service&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;backend&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;   &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;checkers&amp;#34;&lt;/span&gt;: [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.8:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;]}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;In Prometheus, &lt;code&gt;emfor_peer_up&lt;/code&gt; shows member liveness, &lt;code&gt;emfor_peer_messages_rejected_total&lt;/code&gt; the dropped messages by reason (&lt;code&gt;auth&lt;/code&gt; for a different secret, &lt;code&gt;stale&lt;/code&gt; for clock drift, &lt;code&gt;sender&lt;/code&gt; for mismatched member lists), &lt;code&gt;emfor_peer_disagreements&lt;/code&gt; the backends in disagreement, and &lt;code&gt;emfor_peer_checks_skipped_total&lt;/code&gt; how many checks the split has saved.&lt;/p&gt;&#xA;&lt;h2 id=&#34;7-summary&#34;&gt;7. Summary&#xA;&lt;/h2&gt;&lt;p&gt;A peer group does not turn emfor into a distributed system. It is closer to the daemons of one data centre nudging each other:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;when one host notices a failure, the others recheck and follow within about a second;&lt;/li&gt;&#xA;&lt;li&gt;three assigned checkers vote, telling &amp;ldquo;the backend is really down&amp;rdquo; apart from &amp;ldquo;only my host has a problem&amp;rdquo;;&lt;/li&gt;&#xA;&lt;li&gt;members that are not assigned checkers skip most checks thanks to the vote, so the checks a backend receives no longer grow linearly with the number of hosts.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Lookups still read only a local file and still finish within microseconds, and whatever happens to any member of the group, the worst case is a fall back to each host deciding on its own.&lt;/p&gt;&#xA;&lt;p&gt;emfor&amp;rsquo;s source code, full manual and example configs are on GitHub: &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;github.com/embark-systems/emfor&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h3 id=&#34;-technical-collaboration-and-architecture-consulting&#34;&gt;💡 Technical collaboration and architecture consulting&#xA;&lt;/h3&gt;&lt;p&gt;Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends?&#xA;We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.&lt;/p&gt;&#xA;&lt;p&gt;📧 &lt;a class=&#34;link&#34; href=&#34;mailto:service@embark.systems&#34; &gt;Book a technical consultation&lt;/a&gt;&lt;/p&gt;&#xA;</description>
        </item></channel>
</rss>
