<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>Distributed-Systems on Embark Systems</title>
        <link>https://embark.systems/en/tags/distributed-systems/</link>
        <description>Recent content in Distributed-Systems on Embark Systems</description>
        <generator>Hugo -- gohugo.io</generator>
        <language>en</language>
        <lastBuildDate>Fri, 13 Nov 2026 18:00:00 +0800</lastBuildDate><atom:link href="https://embark.systems/en/tags/distributed-systems/index.xml" rel="self" type="application/rss+xml" /><item>
            <title>emfor Peer Groups: Health Checks That Help Each Other Within a Data Centre</title>
            <link>https://embark.systems/en/posts/emfor-peer-groups/</link>
            <pubDate>Fri, 13 Nov 2026 18:00:00 +0800</pubDate>
            <guid>https://embark.systems/en/posts/emfor-peer-groups/</guid>
            <description>&lt;img src=&#34;https://embark.systems/images/emfor-peer-vote.png&#34; alt=&#34;Featured image of post emfor Peer Groups: Health Checks That Help Each Other Within a Data Centre&#34; /&gt;&#xA;    &lt;blockquote&gt;&#xA;        &lt;p&gt;&lt;strong&gt;Updated 2026-11-14:&lt;/strong&gt; revised for emfor 0.4.0. Since 0.4.0, a member that is not an assigned checker makes its own check at least every third &lt;code&gt;interval&lt;/code&gt; (every fifth in 0.3.0) and starts a fast recheck run when its own check fails; peer messages are now encrypted. See &lt;a class=&#34;link&#34; href=&#34;https://embark.systems/en/posts/emfor-0.4.0/&#34; &gt;the emfor 0.4.0 release notes&lt;/a&gt;.&lt;/p&gt;&#xA;&#xA;    &lt;/blockquote&gt;&#xA;&lt;h2 id=&#34;1-what-emfor-is&#34;&gt;1. What emfor is&#xA;&lt;/h2&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;emfor&lt;/a&gt; is an open-source health-check daemon that runs on every application host. It checks the backends of the services your applications use (TCP, HTTP, Nagios-style scripts, and plugins for redis, MySQL, PostgreSQL, LDAP and more) and writes which backends are healthy to an LMDB file on the same host. When an application needs a database server, it asks the client library for an address; the library reads the local file and, following the service&amp;rsquo;s &lt;code&gt;balance&lt;/code&gt; setting, returns an answer such as &lt;code&gt;10.0.3.1:5432&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;              checks over the network&#xA;  backends &amp;lt;------------------------------ emfor daemon ---------&amp;gt; DNS on 127.0.0.1&#xA;                                               |  writes&#xA;                                               v&#xA;                                     local LMDB state store&#xA;                                               |  reads&#xA;                                       client library&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The key point of this design is that &lt;strong&gt;a lookup never touches the network&lt;/strong&gt;. It takes about a microsecond and keeps working when other hosts or the network fail. Each host uses its own config and sees the backends from where it stands; there is no central server.&lt;/p&gt;&#xA;&lt;p&gt;But &amp;ldquo;everyone looks for themselves&amp;rdquo; has a cost.&lt;/p&gt;&#xA;&lt;h2 id=&#34;2-three-problems-with-checking-alone&#34;&gt;2. Three problems with checking alone&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;1. Failures are noticed slowly.&lt;/strong&gt; A backend is marked unhealthy only after &lt;code&gt;fall&lt;/code&gt; consecutive failed checks. With the defaults, &lt;code&gt;interval = 2s&lt;/code&gt; and &lt;code&gt;fall = 3&lt;/code&gt;, that is about 6 seconds. With twenty hosts in one data centre, every one of them spends those 6 seconds on its own, although they are all looking at the same broken backend.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;2. What one host sees is not necessarily the truth.&lt;/strong&gt; A failed check may mean the backend is really down, or only that this host&amp;rsquo;s network, firewall rules or load are the problem. The other way round, a host that can reach a backend does not prove that every host can. A single host cannot tell the difference.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;3. The number of checks grows with the number of hosts.&lt;/strong&gt; With N hosts each checking every backend, a backend answers N checks every &lt;code&gt;interval&lt;/code&gt;. With enough hosts, the health checks themselves become a load on the backend.&lt;/p&gt;&#xA;&lt;p&gt;Peer groups are designed for these three problems.&lt;/p&gt;&#xA;&lt;h2 id=&#34;3-what-a-peer-group-does&#34;&gt;3. What a peer group does&#xA;&lt;/h2&gt;&lt;p&gt;The emfor daemons of one data centre (&lt;code&gt;site&lt;/code&gt;) that list each other under &lt;code&gt;[peers]&lt;/code&gt; in their config form a peer group. Members talk over signed UDP messages and do three things.&lt;/p&gt;&#xA;&lt;h3 id=&#34;1-failure-reports-someone-noticed-so-i-check-right-away&#34;&gt;1. Failure reports: someone noticed, so I check right away&#xA;&lt;/h3&gt;&lt;p&gt;Every member sends a heartbeat every second, so everyone knows who is live. When a member sees a backend go from &lt;code&gt;healthy&lt;/code&gt; to &lt;code&gt;unhealthy&lt;/code&gt;, it tells every other member. The members that receive the report &lt;strong&gt;do not simply believe it&lt;/strong&gt;: they recheck that backend at once, then every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks, instead of waiting for their next &lt;code&gt;interval&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;The result: once one member has noticed a failure, the others confirm it with their own checks and follow within about a second. A report is only a hint that it is time to look; it never changes a backend&amp;rsquo;s state by itself.&lt;/p&gt;&#xA;&lt;h3 id=&#34;2-majority-vote-three-assigned-checkers-vote&#34;&gt;2. Majority vote: three assigned checkers vote&#xA;&lt;/h3&gt;&lt;p&gt;What does change the state is the vote. With at least three live members (this host included), every backend has three &lt;strong&gt;assigned checkers&lt;/strong&gt;. They are chosen by rendezvous hashing: for each candidate member, compute&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;the first 4 bytes of MD5(&amp;#34;&amp;lt;service name&amp;gt;/&amp;lt;backend&amp;gt;/&amp;lt;member address&amp;gt;&amp;#34;)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;and take the three largest values. Because every member uses the same formula on the same list of live members, everyone picks the same three without any coordination, and when a member joins or leaves, only a few backends get different assigned checkers.&lt;/p&gt;&#xA;&lt;p&gt;After each of its own checks, an assigned checker sends its result (&lt;code&gt;up&lt;/code&gt; or &lt;code&gt;down&lt;/code&gt;) to the other members. After each of its own checks, every member counts the votes for that backend from its assigned checkers that are recent and come from a member with the same check definition:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Votes&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Result&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Reason reads&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Two or more &lt;code&gt;down&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Down, even if this host&amp;rsquo;s own check passed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;peers 2/3: connect: Connection refused&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fewer than two &lt;code&gt;down&lt;/code&gt;, own check failed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;This host&amp;rsquo;s own result&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;local: connect: Connection refused&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fewer than three live members, or no vote&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;This host&amp;rsquo;s own result, unchanged&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The original reason, no prefix&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The prefix tells an operator at a glance where a verdict came from: &lt;code&gt;peers 2/3:&lt;/code&gt; is the group&amp;rsquo;s consensus, &lt;code&gt;local:&lt;/code&gt; is a problem only this host sees. The latter deserves attention; it usually means something is wrong with this host&amp;rsquo;s own network or firewall.&lt;/p&gt;&#xA;&lt;p&gt;The opposite case is visible too: a backend the group votes down while this host&amp;rsquo;s own check passes is marked &lt;code&gt;local-up&lt;/code&gt; by &lt;code&gt;emfor status&lt;/code&gt; and counted by &lt;code&gt;emfor_peer_disagreements&lt;/code&gt;. &amp;ldquo;I can reach it, the others cannot&amp;rdquo; is often a sign of a network partition or of a firewall rule on the other hosts.&lt;/p&gt;&#xA;&lt;p&gt;&amp;ldquo;The same check definition&amp;rdquo; matters as well. Each service&amp;rsquo;s &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;check_options&lt;/code&gt; and &lt;code&gt;timeout&lt;/code&gt; are hashed into a digest, and only members with the same digest have their votes counted. When two hosts check the same service differently, their results cannot be compared directly; &lt;code&gt;emfor status&lt;/code&gt; then shows &amp;ldquo;check definition differs&amp;rdquo; as a reminder to make the configs match.&lt;/p&gt;&#xA;&lt;h3 id=&#34;3-split-checks-fewer-checks-for-those-who-are-not-assigned&#34;&gt;3. Split checks: fewer checks for those who are not assigned&#xA;&lt;/h3&gt;&lt;p&gt;With a vote in place, members that are not assigned checkers need not check every time. While two assigned checkers agree (two &lt;code&gt;up&lt;/code&gt; or two &lt;code&gt;down&lt;/code&gt; votes), a member that is not one of them takes the vote&amp;rsquo;s result and skips the check. It never lets go completely, though:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;it makes its own check at least every third &lt;code&gt;interval&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;it checks every time while there are fewer than two agreeing votes;&lt;/li&gt;&#xA;&lt;li&gt;when its own check fails while the assigned checkers say up, it starts a fast recheck run (every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks), as after a peer&amp;rsquo;s &lt;code&gt;down&lt;/code&gt; report;&lt;/li&gt;&#xA;&lt;li&gt;after its own check fails, it checks every &lt;code&gt;interval&lt;/code&gt; again until a check passes;&lt;/li&gt;&#xA;&lt;li&gt;when the vote says up but this host has not measured a latency yet, it checks once itself, so a backend is never declared healthy without a measurement.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;With three assigned checkers and N members, a backend gets about&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;3 + (N − 3) / 3 checks&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;per &lt;code&gt;interval&lt;/code&gt; instead of N. With 20 hosts, that drops from 20 checks per &lt;code&gt;interval&lt;/code&gt; to about 8.7.&lt;/p&gt;&#xA;&lt;p&gt;The price: a failure that only one host that is not an assigned checker can see (a routing problem that affects only that host, say) is noticed there only at its next own check plus the fast recheck run. Measured over 12 failures with &lt;code&gt;interval = &amp;quot;2s&amp;quot;&lt;/code&gt; and &lt;code&gt;fall = 3&lt;/code&gt;, that took 0.8 to 5.8 seconds, 3.3 seconds median; 0.3.0, which checked itself only every fifth &lt;code&gt;interval&lt;/code&gt; and had no fast run here, took up to about 14 seconds. This is latency traded for fewer checks. &lt;code&gt;emfor status&lt;/code&gt; marks such backends &lt;code&gt;voted&lt;/code&gt;, so operators know which backends this host judges mostly by the vote.&lt;/p&gt;&#xA;&lt;h2 id=&#34;4-lines-the-design-does-not-cross&#34;&gt;4. Lines the design does not cross&#xA;&lt;/h2&gt;&lt;p&gt;With peer groups added, emfor&amp;rsquo;s original properties are unchanged:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Lookups still read only the local store.&lt;/strong&gt; What the group exchanges only affects how the daemon decides; the client library does not know the group exists, and a lookup gains no network round trip.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Unreachable members mean a fall back to local judgement.&lt;/strong&gt; When members stop or the network breaks, there are simply no votes, and the daemon goes back to deciding from its own checks, exactly as without a group.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;No master, and no spanning data centres.&lt;/strong&gt; All members are equal, and assigned checkers come from a hash, so there is no election. A group is limited to one &lt;code&gt;site&lt;/code&gt;, because different data centres see different networks and should not vote for each other.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Messages are encrypted and protected against replay.&lt;/strong&gt; Every message is encrypted with ChaCha20-Poly1305 under a key derived from a shared secret (0.3.0 only signed them with HMAC-SHA256 and sent them in the clear). The receiver checks that the sender&amp;rsquo;s address is a member and that the timestamp is within 5 seconds of its own clock, and rejects replayed old messages. The secret can be rotated without downtime: after a host sees its secret file change, it keeps sending with the old secret for 60 seconds and accepts it for 120 seconds.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;5-configuration&#34;&gt;5. Configuration&#xA;&lt;/h2&gt;&lt;p&gt;On every host of the data centre, add:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt;                               &lt;span style=&#34;color:#75715e&#34;&gt;# required with [peers]&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;listen&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;                   &lt;span style=&#34;color:#75715e&#34;&gt;# this host&amp;#39;s address in the group&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;members&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.7:5302&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;secret_file&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/etc/emfor/peers.key&amp;#34;&lt;/span&gt;       &lt;span style=&#34;color:#75715e&#34;&gt;# at least 32 bytes, mode 600 or 640&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;members&lt;/code&gt; lists every member, this host included; the same list can go on every host of the data centre, and only &lt;code&gt;listen&lt;/code&gt; differs. The vote and the split need no settings of their own; they use each service&amp;rsquo;s existing &lt;code&gt;interval&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Some services are not a good fit for a peer group, for example when the check result depends on the host by design. Individual services can stay out:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;local-cache.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;127.0.0.1:11211&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;false&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A service with &lt;code&gt;peers = false&lt;/code&gt; sends no reports or results on this host, ignores the ones it receives, and always decides from its own checks; the other services stay in the group. Changing the setting only takes a reload, and the backends keep their state. The host announces the services it leaves out in its heartbeats (as short hashes of their names), and the other members then leave it out when choosing those services&amp;rsquo; assigned checkers, so with enough live members their backends still have three assigned checkers that vote. Each host can therefore decide on its own which services to leave out. Peer groups and this announcement both came in emfor 0.3.0, so every member of a group reads it.&lt;/p&gt;&#xA;&lt;h2 id=&#34;6-seeing-what-the-group-does&#34;&gt;6. Seeing what the group does&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor status&lt;/code&gt; shows the group under the daemon line:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 4/4 live, checks 12/40 backends here&#xA;peers: check definition of api.svc.emfor differs at 10.0.0.6:5302&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The first line shows which members are live and that, of the 40 backends shown, this host checks 12 itself every time (the rest are marked &lt;code&gt;voted&lt;/code&gt;). &lt;code&gt;emfor status --json&lt;/code&gt; also lists every backend&amp;rsquo;s assigned checkers:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;assigned_checkers&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#960050;background-color:#1e0010&#34;&gt;:&lt;/span&gt; [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  {&lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;service&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;backend&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;   &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;checkers&amp;#34;&lt;/span&gt;: [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.8:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;]}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;In Prometheus, &lt;code&gt;emfor_peer_up&lt;/code&gt; shows member liveness, &lt;code&gt;emfor_peer_messages_rejected_total&lt;/code&gt; the dropped messages by reason (&lt;code&gt;auth&lt;/code&gt; for a different secret, &lt;code&gt;stale&lt;/code&gt; for clock drift, &lt;code&gt;sender&lt;/code&gt; for mismatched member lists), &lt;code&gt;emfor_peer_disagreements&lt;/code&gt; the backends in disagreement, and &lt;code&gt;emfor_peer_checks_skipped_total&lt;/code&gt; how many checks the split has saved.&lt;/p&gt;&#xA;&lt;h2 id=&#34;7-summary&#34;&gt;7. Summary&#xA;&lt;/h2&gt;&lt;p&gt;A peer group does not turn emfor into a distributed system. It is closer to the daemons of one data centre nudging each other:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;when one host notices a failure, the others recheck and follow within about a second;&lt;/li&gt;&#xA;&lt;li&gt;three assigned checkers vote, telling &amp;ldquo;the backend is really down&amp;rdquo; apart from &amp;ldquo;only my host has a problem&amp;rdquo;;&lt;/li&gt;&#xA;&lt;li&gt;members that are not assigned checkers skip most checks thanks to the vote, so the checks a backend receives no longer grow linearly with the number of hosts.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Lookups still read only a local file and still finish within microseconds, and whatever happens to any member of the group, the worst case is a fall back to each host deciding on its own.&lt;/p&gt;&#xA;&lt;p&gt;emfor&amp;rsquo;s source code, full manual and example configs are on GitHub: &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;github.com/embark-systems/emfor&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;hr&gt;&#xA;&lt;h3 id=&#34;-technical-collaboration-and-architecture-consulting&#34;&gt;💡 Technical collaboration and architecture consulting&#xA;&lt;/h3&gt;&lt;p&gt;Are your services struggling with inconsistent health checks, slow failure detection, or check traffic that weighs on your backends?&#xA;We can help you assess your architecture and adopt the health-check and service-discovery approach that fits.&lt;/p&gt;&#xA;&lt;p&gt;📧 &lt;a class=&#34;link&#34; href=&#34;mailto:service@embark.systems&#34; &gt;Book a technical consultation&lt;/a&gt;&lt;/p&gt;&#xA;</description>
        </item></channel>
</rss>
