<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
    <channel>
        <title>emfor User Manual on Embark Systems</title>
        <link>https://embark.systems/en/emfor/</link>
        <description>Recent content in emfor User Manual on Embark Systems</description>
        <generator>Hugo -- gohugo.io</generator>
        <language>en</language><atom:link href="https://embark.systems/en/emfor/index.xml" rel="self" type="application/rss+xml" /><item>
            <title>0. Guide to the Manual</title>
            <link>https://embark.systems/en/emfor/00-guide/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/00-guide/</guid>
            <description>&lt;p&gt;This manual is for the people who install emfor, configure it, and write&#xA;the applications that use it. It explains how the pieces fit together and&#xA;walks through each task; the documents under &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/tree/v0.6.0/docs/deploy/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/&lt;/a&gt; and&#xA;the client READMEs hold the full detail and are linked where they apply.&lt;/p&gt;&#xA;&lt;p&gt;The manual describes emfor 0.5.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Chapter&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What it covers&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../01-concepts/&#34; &gt;1. Concepts&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon, the state store, clients, DNS, health states, and what happens when something fails&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../02-installation/&#34; &gt;2. Installation&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Building, install prefixes, systemd, FreeBSD, and the client libraries&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../03-configuration/&#34; &gt;3. Configuration&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Every table and key of &lt;code&gt;emfor.toml&lt;/code&gt;, validation, and what a reload can change&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/&#34; &gt;4. Health checks&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tcp&lt;/code&gt;, &lt;code&gt;http&lt;/code&gt;, &lt;code&gt;exec&lt;/code&gt;, plugins and &lt;code&gt;emfor-check&lt;/code&gt;; rise, fall, scheduling and reasons&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/&#34; &gt;5. Choosing a backend&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The &lt;code&gt;balance&lt;/code&gt; methods, &lt;code&gt;on_all_down&lt;/code&gt;, and a stale store&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../06-clients/&#34; &gt;6. Client libraries&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Looking up a backend from Rust, C, Go, Python, Perl and PHP&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;7. DNS&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The DNS server, its answers, SRV records, forwarding from the host resolver, and caching&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../07-dns-recipes/&#34; &gt;7b. DNS recipes&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;nginx, HAProxy, Caddy, Prometheus and pf on emfor&amp;rsquo;s DNS answers, and clients that ask for SRV records themselves&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../08-operations/&#34; &gt;8. Operations&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Commands, signals and reload, &lt;code&gt;emfor status&lt;/code&gt;, &lt;code&gt;emfor lookup&lt;/code&gt;, metrics, logs, exit codes, the store on disk, upgrading&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../09-troubleshooting/&#34; &gt;9. Troubleshooting&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Symptoms, their causes, and what to do&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;a class=&#34;link&#34; href=&#34;../10-migrating/&#34; &gt;10. Moving from HAProxy or Consul&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;What the same configuration looks like in emfor, what has no equivalent, and how to try emfor next to what you have&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;New to emfor? Read chapter 1, then follow the quick start in the&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/README.md#quick-start&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;README&lt;/a&gt;, then chapter 3.&lt;/p&gt;&#xA;&lt;h2 id=&#34;conventions&#34;&gt;Conventions&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;Service names in the examples end in &lt;code&gt;.svc.emfor&lt;/code&gt;, the default DNS zone.&#xA;Any host-name-like name works; only names under the zone resolve over DNS.&lt;/li&gt;&#xA;&lt;li&gt;Paths are the defaults: the config at &lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt;, the store&#xA;at &lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;$&lt;/code&gt; starts a command you type; lines without it are output.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</description>
        </item><item>
            <title>1. Concepts</title>
            <link>https://embark.systems/en/emfor/01-concepts/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/01-concepts/</guid>
            <description>&lt;h2 id=&#34;what-emfor-does&#34;&gt;What emfor does&#xA;&lt;/h2&gt;&lt;p&gt;emfor is a health-check daemon that runs on every application host. It&#xA;checks the backends of the services your applications use, and publishes&#xA;which backends are healthy in a file on the same host. An application that&#xA;needs, say, a database server asks the emfor client library for one; the&#xA;library reads the local file and returns an address such as&#xA;&lt;code&gt;10.0.3.1:5432&lt;/code&gt;. The lookup sends nothing over the network, takes about a&#xA;microsecond, and keeps working when other hosts, or the network, fail.&lt;/p&gt;&#xA;&lt;p&gt;&lt;img alt=&#34;How emfor fits on one host: the daemon checks the backends over the network, writes the local LMDB state store and serves DNS on 127.0.0.1; the client library and emfor status read the store.&#34; loading=&#34;lazy&#34; sizes=&#34;(max-width: 767px) calc(100vw - 30px), (max-width: 1023px) 700px, (max-width: 1279px) 950px, 1232px&#34; src=&#34;https://embark.systems/images/emfor-architecture.svg&#34;&gt;&lt;/p&gt;&#xA;&lt;p&gt;Each host runs its own daemon with its own config and sees the backends&#xA;from where it stands. There is no central server. The daemons of one data&#xA;centre can optionally form a &lt;a class=&#34;link&#34; href=&#34;#peer-groups&#34; &gt;peer group&lt;/a&gt; to pass failures on&#xA;and to vote on each backend; lookups still read only the local store.&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-parts&#34;&gt;The parts&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Part&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What it is&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor server&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon. It reads &lt;code&gt;emfor.toml&lt;/code&gt;, checks every backend repeatedly, and writes the results to the state store. One per host (one per store).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The state store&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;An LMDB file, &lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt; by default, with one record per service. The daemon is its only writer; any number of processes read it.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Client libraries&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_client&lt;/code&gt; (Rust) and &lt;code&gt;libemfor&lt;/code&gt; (C), with wrappers for Go, Python, Perl and PHP. A lookup reads the store and picks one backend.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;DNS server&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Optional. Answers A, AAAA and SRV queries for service names in one zone, from the daemon&amp;rsquo;s memory, for programs that cannot use a client library.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor status&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Shows what the daemon has published. Also usable as a monitoring check.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/metrics&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Optional Prometheus endpoint.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor-check&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A plugin program with checks for redis, MySQL/MariaDB, PostgreSQL, LDAP, DNS, RADIUS, SIP, Diameter, SMTP, AMQP, MongoDB, gRPC, QUIC and HTTP/3.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;services-and-backends&#34;&gt;Services and backends&#xA;&lt;/h2&gt;&lt;p&gt;A &lt;strong&gt;service&lt;/strong&gt; is a name, such as &lt;code&gt;api.svc.emfor&lt;/code&gt;, with a list of&#xA;&lt;strong&gt;backends&lt;/strong&gt;: the &lt;code&gt;ip:port&lt;/code&gt; addresses that serve it. Each service has a&#xA;&lt;strong&gt;check&lt;/strong&gt; (how to test one backend), timing settings, a &lt;strong&gt;balance&lt;/strong&gt; method&#xA;(how to choose among healthy backends) and an &lt;strong&gt;on_all_down&lt;/strong&gt; policy (what&#xA;to do when none is healthy).&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:8080&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Backends are always IP addresses, never host names: emfor does not depend&#xA;on DNS to find them.&lt;/p&gt;&#xA;&lt;h2 id=&#34;check-results-and-health-states&#34;&gt;Check results and health states&#xA;&lt;/h2&gt;&lt;p&gt;Every check ends with one &lt;strong&gt;result&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Result&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Counts against the backend&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;up&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The backend passed.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;down&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The backend failed (refused, timed out, wrong answer).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Yes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;error&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor could not perform the check (a local problem such as an unreadable password file or a crashed plugin).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Each backend has a &lt;strong&gt;health state&lt;/strong&gt;:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;State&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;When&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;unknown&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;After the daemon starts, after the backend is added, after its &lt;code&gt;check&lt;/code&gt; changes, and after a long run of &lt;code&gt;error&lt;/code&gt; results.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;healthy&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;After &lt;code&gt;rise&lt;/code&gt; consecutive &lt;code&gt;up&lt;/code&gt; results (default 2).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;unhealthy&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;After &lt;code&gt;fall&lt;/code&gt; consecutive &lt;code&gt;down&lt;/code&gt; results (default 3).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The counters make a single lost packet harmless: one failed check does not&#xA;remove a backend, and one passed check does not bring it back. An &lt;code&gt;error&lt;/code&gt;&#xA;changes nothing, so a problem on the checking host does not mark good&#xA;backends down; only when a backend has had nothing but errors for longer&#xA;than &lt;code&gt;fall × interval&lt;/code&gt; does it go back to &lt;code&gt;unknown&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-a-lookup-returns&#34;&gt;What a lookup returns&#xA;&lt;/h2&gt;&lt;p&gt;The client takes the service&amp;rsquo;s &lt;strong&gt;healthy&lt;/strong&gt; backends as candidates and lets&#xA;&lt;code&gt;balance&lt;/code&gt; pick one. When none is healthy, &lt;code&gt;on_all_down&lt;/code&gt; decides:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;open&lt;/code&gt; (the default): the application gets a backend anyway, which is&#xA;usually better than failing outright when the checks, not the backends,&#xA;are at fault. Backends still &lt;code&gt;unknown&lt;/code&gt; (not yet decided, or whose checks&#xA;only give errors) come before those known to be &lt;code&gt;unhealthy&lt;/code&gt;; see&#xA;&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#when-no-backend-is-healthy&#34; &gt;chapter 5&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;closed&lt;/code&gt;: the lookup fails with &lt;code&gt;ALL_DOWN&lt;/code&gt;, and DNS answers &lt;code&gt;SERVFAIL&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;the-heartbeat-and-a-stale-store&#34;&gt;The heartbeat and a stale store&#xA;&lt;/h2&gt;&lt;p&gt;The daemon writes a &lt;strong&gt;heartbeat&lt;/strong&gt; (the current time) into the store at&#xA;least once a second. A client compares it with its own clock: when the&#xA;heartbeat is more than 3 seconds old (&lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt;), or the&#xA;daemon has stopped and set it to 0, the store is &lt;strong&gt;stale&lt;/strong&gt;. In a stale&#xA;store the client treats every backend as not healthy, and &lt;code&gt;on_all_down&lt;/code&gt;&#xA;decides. An application therefore never trusts health data from a daemon&#xA;that has hung or gone away, and it still gets an address from an &lt;code&gt;open&lt;/code&gt;&#xA;service.&lt;/p&gt;&#xA;&lt;p&gt;When the daemon starts it rewrites every service record with all backends&#xA;&lt;code&gt;unknown&lt;/code&gt;; it never reuses health state from a previous run. While a&#xA;backend is &lt;code&gt;unknown&lt;/code&gt; it is checked about every 200 ms, so with the default&#xA;settings every backend is known within about half a second of a start.&lt;/p&gt;&#xA;&lt;h2 id=&#34;peer-groups&#34;&gt;Peer groups&#xA;&lt;/h2&gt;&lt;p&gt;Hosts in one data centre see the same backends. Without help, each daemon&#xA;needs &lt;code&gt;fall&lt;/code&gt; failed checks of its own to notice that a backend went down.&#xA;In a &lt;strong&gt;peer group&lt;/strong&gt;, the daemons of one &lt;code&gt;site&lt;/code&gt; that list each other under&#xA;&lt;code&gt;[peers]&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;chapter 3&lt;/a&gt;) send each other&#xA;authenticated UDP messages:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;a heartbeat every second, so each member knows which others are live;&lt;/li&gt;&#xA;&lt;li&gt;when a member sees a backend go from &lt;code&gt;healthy&lt;/code&gt; to &lt;code&gt;unhealthy&lt;/code&gt;, a report&#xA;to every other member, which then rechecks that backend at once and&#xA;every 200 ms, up to &lt;code&gt;fall&lt;/code&gt; checks, instead of waiting for its next&#xA;&lt;code&gt;interval&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;A report never changes a backend&amp;rsquo;s state by itself. What does is the&#xA;&lt;strong&gt;majority vote&lt;/strong&gt;: with at least three live members (this one included),&#xA;every backend has three &lt;strong&gt;assigned checkers&lt;/strong&gt;, chosen from the live members&#xA;by hashing the service, the backend and each member&amp;rsquo;s address, so members&#xA;that see the same live members pick the same three. For a few seconds after&#xA;a member starts, stops or becomes unreachable, members may disagree on who&#xA;is live and so on the three. After each of its own checks an assigned&#xA;checker sends its result (&lt;code&gt;up&lt;/code&gt; or &lt;code&gt;down&lt;/code&gt;) to the others. After each of its&#xA;own checks, every member counts the results of that backend&amp;rsquo;s assigned&#xA;checkers that are recent (from the last three of the sender&amp;rsquo;s intervals)&#xA;and come from a member with the same check definition:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;two or more &lt;code&gt;down&lt;/code&gt; votes: the backend counts as down here too, even if&#xA;this host&amp;rsquo;s own check passed; the reason reads &lt;code&gt;peers 2/3: &amp;lt;reason&amp;gt;&lt;/code&gt;&#xA;(down votes / votes counted);&lt;/li&gt;&#xA;&lt;li&gt;otherwise this host&amp;rsquo;s own result counts, and a &lt;code&gt;down&lt;/code&gt; reads&#xA;&lt;code&gt;local: &amp;lt;reason&amp;gt;&lt;/code&gt;: this host sees a problem the others do not;&lt;/li&gt;&#xA;&lt;li&gt;with fewer than three live members, or no vote at all, the own result&#xA;counts unchanged, as without a group.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;The counted result then goes through &lt;code&gt;rise&lt;/code&gt; and &lt;code&gt;fall&lt;/code&gt; as usual. Each&#xA;result a member sends counts at one check time only: a member with a short&#xA;&lt;code&gt;interval&lt;/code&gt; does not count a slower member&amp;rsquo;s single &lt;code&gt;down&lt;/code&gt; again and again,&#xA;and a check that only repeats votes already counted (with the own check&#xA;passing) changes nothing. A backend&#xA;that the group sees down while this host&amp;rsquo;s own check passes is shown by&#xA;&lt;code&gt;emfor status&lt;/code&gt; with the marker &lt;code&gt;local-up&lt;/code&gt;. Lookups still read only the&#xA;local store. Members that are not reachable change nothing: the vote falls&#xA;back to each daemon&amp;rsquo;s own checks. There is no master, and a group never&#xA;spans two sites. A service can be left out of the group with&#xA;&lt;code&gt;peers = false&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;chapter 3&lt;/a&gt;); the host tells&#xA;the other members, which then choose the service&amp;rsquo;s assigned checkers&#xA;among the rest.&lt;/p&gt;&#xA;&lt;p&gt;The checks are split, too. A member that is not an assigned checker of a&#xA;backend does not check it while two of its assigned checkers agree: two&#xA;&lt;code&gt;up&lt;/code&gt; votes count as &lt;code&gt;up&lt;/code&gt;, two &lt;code&gt;down&lt;/code&gt; votes as &lt;code&gt;down&lt;/code&gt; (&lt;code&gt;peers 2/3:&lt;/code&gt;). It&#xA;still makes its own check, with the service&amp;rsquo;s usual &lt;code&gt;check&lt;/code&gt;, at least&#xA;every third &lt;code&gt;interval&lt;/code&gt;, and every time while there are fewer than two&#xA;agreeing votes. When its own check fails while the assigned checkers see&#xA;the backend up, it starts a fast recheck run (every 200 ms, up to &lt;code&gt;fall&lt;/code&gt;&#xA;checks, as after a peer&amp;rsquo;s report) and then checks every &lt;code&gt;interval&lt;/code&gt; until&#xA;its own check passes, so a failure only this host sees still reads&#xA;&lt;code&gt;local: &amp;lt;reason&amp;gt;&lt;/code&gt;; such a failure is noticed within about three&#xA;&lt;code&gt;interval&lt;/code&gt;s plus &lt;code&gt;fall&lt;/code&gt; fast checks instead of &lt;code&gt;fall&lt;/code&gt; checks. With three&#xA;assigned checkers and N members a backend gets about 3 + (N − 3) / 3&#xA;checks per &lt;code&gt;interval&lt;/code&gt; instead of N.&lt;/p&gt;&#xA;&lt;h2 id=&#34;failure-behavior-at-a-glance&#34;&gt;Failure behavior at a glance&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What fails&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What applications see&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;It turns &lt;code&gt;unhealthy&lt;/code&gt; after &lt;code&gt;fall&lt;/code&gt; failed checks (about 6 seconds with the defaults) and is no longer returned. In a peer group, once one member has noticed, the others follow within about a second; when two of its three assigned checkers see it down, every member marks it down after &lt;code&gt;fall&lt;/code&gt; checks, also where it still answers. A failure that only one member that is not an assigned checker sees takes up to about three &lt;code&gt;interval&lt;/code&gt;s longer there, but then needs &lt;code&gt;fall&lt;/code&gt; checks 200 ms apart instead of &lt;code&gt;fall&lt;/code&gt; &lt;code&gt;interval&lt;/code&gt;s (with the defaults it is down there after at most about 6.4 seconds, against 6 without a group).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Every backend of a service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;open&lt;/code&gt;: an &lt;code&gt;unknown&lt;/code&gt; backend if there is one, otherwise any backend; &lt;code&gt;closed&lt;/code&gt;: &lt;code&gt;ALL_DOWN&lt;/code&gt; / &lt;code&gt;SERVFAIL&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The emfor daemon stops or hangs&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;After 3 seconds the store is stale: &lt;code&gt;open&lt;/code&gt; services return any backend, &lt;code&gt;closed&lt;/code&gt; services fail. Lookups never block.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store file is missing or unreadable&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Lookups fail with &lt;code&gt;UNAVAILABLE&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The checking host loses its network&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Checks fail, backends turn &lt;code&gt;unhealthy&lt;/code&gt;; the applications on that host cannot reach them either.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A plugin crashes or stops answering&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Its checks are &lt;code&gt;error&lt;/code&gt; (no state change) until the daemon has restarted it. A plugin that leaves 3 requests in a row unanswered is killed and restarted.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;files-the-daemon-uses&#34;&gt;Files the daemon uses&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;File&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Purpose&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The config. Another path with &lt;code&gt;--config&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;lt;store&amp;gt;&lt;/code&gt; (&lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The state store.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;lt;store&amp;gt;-lock&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;LMDB&amp;rsquo;s lock file. Every reader registers in it, so applications need write access to it.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;lt;store&amp;gt;.writer&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Held locked by the running daemon, so that only one daemon writes a store.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;lt;store&amp;gt;.corrupt&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A damaged store moved aside at start, kept for inspection.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;</description>
        </item><item>
            <title>2. Installation</title>
            <link>https://embark.systems/en/emfor/02-installation/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/02-installation/</guid>
            <description>&lt;p&gt;emfor is built from source with Cargo. Linux and FreeBSD are the&#xA;deployment platforms; macOS works for development.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-you-install-where&#34;&gt;What you install where&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Host&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Install&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Every application host&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon &lt;code&gt;emfor&lt;/code&gt;, its config, and a service definition (systemd or rc.d).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Hosts that use a check of &lt;code&gt;emfor-check&lt;/code&gt; (redis, MySQL/MariaDB, PostgreSQL, LDAP, DNS, RADIUS, SIP, Diameter, SMTP, AMQP, MongoDB, gRPC, QUIC or HTTP/3)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Also &lt;code&gt;emfor-check&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Hosts whose applications use a client library&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;libemfor&lt;/code&gt;, and the wrapper for the application&amp;rsquo;s language.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Hosts whose programs use DNS only&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Nothing more; forward the zone from the host resolver (&lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;building-the-daemon&#34;&gt;Building the daemon&#xA;&lt;/h2&gt;&lt;p&gt;You need a Rust toolchain that supports the 2024 edition (Rust 1.85 or&#xA;later). From a checkout or an unpacked source release:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo build --release --bin emfor --bin emfor-check&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;install -m &lt;span style=&#34;color:#ae81ff&#34;&gt;0755&lt;/span&gt; target/release/emfor /usr/local/bin/emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;install -m &lt;span style=&#34;color:#ae81ff&#34;&gt;0755&lt;/span&gt; target/release/emfor-check /usr/local/libexec/emfor-check&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;emfor-check&lt;/code&gt; is optional; install it only if you use its plugins&#xA;(&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#plugins-and-emfor-check&#34; &gt;chapter 4&lt;/a&gt;). The database&#xA;drivers are linked into &lt;code&gt;emfor-check&lt;/code&gt; only, not into the daemon or the&#xA;client library.&lt;/p&gt;&#xA;&lt;h3 id=&#34;the-default-config-path&#34;&gt;The default config path&#xA;&lt;/h3&gt;&lt;p&gt;&lt;code&gt;emfor&lt;/code&gt; reads &lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt; unless &lt;code&gt;--config&lt;/code&gt; names another&#xA;file. The default is fixed when the binary is built, from the environment&#xA;variable &lt;code&gt;EMFOR_PREFIX&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;code&gt;EMFOR_PREFIX&lt;/code&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default config path&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;not set, empty, or &lt;code&gt;/usr&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/usr/local&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/usr/local/etc/emfor/emfor.toml&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/opt/emfor&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/opt/emfor/etc/emfor/emfor.toml&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;EMFOR_PREFIX&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;/usr/local cargo build --release --bin emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The prefix must be an absolute path, or the build fails. &lt;code&gt;emfor --help&lt;/code&gt;&#xA;shows the path that was built in. Only the config path follows the prefix:&#xA;the store stays at the config&amp;rsquo;s &lt;code&gt;[store] path&lt;/code&gt; (default&#xA;&lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt;), which is also where clients look by default.&lt;/p&gt;&#xA;&lt;h3 id=&#34;with-cargo-install&#34;&gt;With cargo install&#xA;&lt;/h3&gt;&lt;p&gt;The repository root is a workspace, not a package, so name each crate:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;EMFOR_PREFIX&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;/usr/local cargo install --locked --path crates/emfor-daemon --root /usr/local&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cargo install --locked --path crates/emfor-check --root /usr/local&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;--root&lt;/code&gt; only decides where the binaries go; set &lt;code&gt;EMFOR_PREFIX&lt;/code&gt; as well.&lt;/p&gt;&#xA;&lt;h2 id=&#34;installing-a-package&#34;&gt;Installing a package&#xA;&lt;/h2&gt;&lt;p&gt;The repository builds a deb and an rpm (&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/packages/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;packages/&lt;/a&gt;;&#xA;&lt;code&gt;packages/build.sh&lt;/code&gt;). Each installs &lt;code&gt;emfor&lt;/code&gt; and &lt;code&gt;emfor-check&lt;/code&gt; under&#xA;&lt;code&gt;/usr/bin&lt;/code&gt;, the systemd unit, a config that only reads &lt;code&gt;/etc/emfor/conf.d&lt;/code&gt;,&#xA;the commented example config under &lt;code&gt;/usr/share/doc/emfor/&lt;/code&gt;, and creates the&#xA;&lt;code&gt;emfor&lt;/code&gt; user. Put your services in files of &lt;code&gt;/etc/emfor/conf.d/&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#splitting-the-config-over-a-directory&#34; &gt;chapter&#xA;3&lt;/a&gt;), then:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dpkg -i emfor_0.6.0-1_amd64.deb      &lt;span style=&#34;color:#75715e&#34;&gt;# Debian, Ubuntu: also enables and starts it&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;rpm -i emfor-0.6.0-1.x86_64.rpm      &lt;span style=&#34;color:#75715e&#34;&gt;# RHEL and relatives: then systemctl enable --now emfor&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor check-config&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl reload emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The packages are not signed and are built from this repository, not&#xA;published anywhere; &lt;code&gt;packages/build.sh&lt;/code&gt; builds them for your architecture.&lt;/p&gt;&#xA;&lt;h2 id=&#34;running-under-systemd&#34;&gt;Running under systemd&#xA;&lt;/h2&gt;&lt;p&gt;The repository has a hardened unit,&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/systemd/emfor.service&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/systemd/emfor.service&lt;/a&gt;. It&#xA;runs &lt;code&gt;/usr/local/bin/emfor server --config /etc/emfor/emfor.toml&lt;/code&gt; as user&#xA;&lt;code&gt;emfor&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;useradd --system --no-create-home --shell /usr/sbin/nologin emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;install -m &lt;span style=&#34;color:#ae81ff&#34;&gt;0644&lt;/span&gt; examples/systemd/emfor.service /etc/systemd/system/emfor.service&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;install -d /etc/emfor /etc/emfor/conf.d&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;install -m &lt;span style=&#34;color:#ae81ff&#34;&gt;0644&lt;/span&gt; examples/emfor.toml /etc/emfor/emfor.toml   &lt;span style=&#34;color:#75715e&#34;&gt;# then edit it&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor check-config&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl daemon-reload&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;systemctl enable --now emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor status&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;What the unit does for you:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;StateDirectory=emfor&lt;/code&gt; creates &lt;code&gt;/var/lib/emfor&lt;/code&gt; for the store, owned by&#xA;&lt;code&gt;emfor&lt;/code&gt;. Do not switch to &lt;code&gt;DynamicUser=&lt;/code&gt;: its directory is not reachable&#xA;by applications.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;systemctl reload emfor&lt;/code&gt; runs &lt;code&gt;emfor check-config&lt;/code&gt; first and sends SIGHUP&#xA;only when the config is valid.&lt;/li&gt;&#xA;&lt;li&gt;The daemon is restarted after a crash, including exit status 70, but not&#xA;after status 1 or 2, which a retry does not fix&#xA;(&lt;a class=&#34;link&#34; href=&#34;../08-operations/#exit-codes&#34; &gt;exit codes&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Logs go to the journal: &lt;code&gt;journalctl -u emfor&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;LimitNOFILE=65536&lt;/code&gt; raises the open-file limit from systemd&amp;rsquo;s default&#xA;1024; checks, plugins and DNS and metrics connections all use files.&lt;/li&gt;&#xA;&lt;li&gt;The file system is read-only except &lt;code&gt;/var/lib/emfor&lt;/code&gt;. A &lt;code&gt;[store] path&lt;/code&gt;&#xA;elsewhere needs &lt;code&gt;ReadWritePaths=&lt;/code&gt;; DNS on port 53 needs&#xA;&lt;code&gt;CAP_NET_BIND_SERVICE&lt;/code&gt; (see &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/daemon.md#running-under-systemd&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/daemon.md&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Build without &lt;code&gt;EMFOR_PREFIX&lt;/code&gt; (or with &lt;code&gt;/usr&lt;/code&gt;) so that &lt;code&gt;emfor status&lt;/code&gt; and&#xA;&lt;code&gt;emfor check-config&lt;/code&gt; find &lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt; without &lt;code&gt;--config&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;/etc/emfor/conf.d&lt;/code&gt; is for the files of &lt;code&gt;include_dir&lt;/code&gt;&#xA;(&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#splitting-the-config-over-a-directory&#34; &gt;chapter 3&lt;/a&gt;).&#xA;The sample config leaves &lt;code&gt;include_dir&lt;/code&gt; commented out; if you set it, the&#xA;directory must exist and must not be writable by group or others.&lt;/p&gt;&#xA;&lt;h2 id=&#34;running-on-freebsd&#34;&gt;Running on FreeBSD&#xA;&lt;/h2&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/tree/v0.6.0/packages/freebsd/&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;packages/freebsd/&lt;/a&gt; is an example port&#xA;(&lt;code&gt;net-mgmt/emfor&lt;/code&gt;) that builds &lt;code&gt;emfor&lt;/code&gt; and &lt;code&gt;emfor-check&lt;/code&gt;, installs an rc.d&#xA;script and a sample config, and creates the &lt;code&gt;emfor&lt;/code&gt; user, the empty&#xA;&lt;code&gt;/usr/local/etc/emfor/conf.d&lt;/code&gt; for &lt;code&gt;include_dir&lt;/code&gt; and &lt;code&gt;/var/db/emfor&lt;/code&gt;, where its sample config puts the store (applications set&#xA;&lt;code&gt;EMFOR_STORE=/var/db/emfor/state.mdb&lt;/code&gt;). The rc.d script runs the daemon&#xA;under &lt;code&gt;daemon(8)&lt;/code&gt;, which starts it again 2 seconds after it exits. After&#xA;installing the port and editing &lt;code&gt;/usr/local/etc/emfor/emfor.toml&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sysrc emfor_enable&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;YES&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;service emfor start&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor status&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;rc.conf variable&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_enable&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NO&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Start at boot.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_config&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/usr/local/etc/emfor/emfor.toml&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Config file.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_user&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;User the daemon runs as. It needs write access to the store&amp;rsquo;s directory.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;code&gt;service emfor configtest&lt;/code&gt; checks the config (as &lt;code&gt;emfor_user&lt;/code&gt;);&#xA;&lt;code&gt;service emfor reload&lt;/code&gt; checks it and then sends SIGHUP; &lt;code&gt;start&lt;/code&gt; refuses an&#xA;invalid config. The daemon&amp;rsquo;s log goes to syslog with the tag &lt;code&gt;emfor&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;Without the port, build emfor with &lt;code&gt;EMFOR_PREFIX=/usr/local&lt;/code&gt;, install the&#xA;port&amp;rsquo;s rc.d script with its prefix filled in, and create the &lt;code&gt;emfor&lt;/code&gt; user&#xA;(or set &lt;code&gt;emfor_user&lt;/code&gt;), which needs write access to the directory of&#xA;&lt;code&gt;[store] path&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;sed &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;s|%%PREFIX%%|/usr/local|g&amp;#39;&lt;/span&gt; packages/freebsd/files/emfor.in &amp;gt; /usr/local/etc/rc.d/emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;chmod &lt;span style=&#34;color:#ae81ff&#34;&gt;0555&lt;/span&gt; /usr/local/etc/rc.d/emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;pw useradd emfor -d /nonexistent -s /usr/sbin/nologin -c &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;emfor daemon&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Read the comments in the script and the port&amp;rsquo;s README before using them.&#xA;Both have been run on FreeBSD 15.1-RELEASE (arm64, in a VM), the port with&#xA;&lt;code&gt;make stage check-plist install&lt;/code&gt;; not with poudriere, on 14.x or on amd64.&lt;/p&gt;&#xA;&lt;h2 id=&#34;installing-the-client-libraries&#34;&gt;Installing the client libraries&#xA;&lt;/h2&gt;&lt;p&gt;Every client reads the store through &lt;code&gt;libemfor&lt;/code&gt;, built from&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/rust/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;clients/rust&lt;/a&gt;. It contains the daemon&amp;rsquo;s own&#xA;LMDB build: a system liblmdb cannot read the store.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Install the client library from the same emfor release as the daemon,&#xA;and upgrade both together.&lt;/strong&gt; A mismatch makes lookups fail with&#xA;&lt;code&gt;UNAVAILABLE&lt;/code&gt; and a message that says so.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Language&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;How&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rust&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Depend on the &lt;code&gt;emfor-client&lt;/code&gt; crate from the same release (&lt;code&gt;clients/rust&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;C&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;cargo install cargo-c&lt;/code&gt;, then in &lt;code&gt;clients/rust&lt;/code&gt;: &lt;code&gt;cargo cinstall --release --features capi --prefix /usr/local&lt;/code&gt;. This installs &lt;code&gt;libemfor&lt;/code&gt;, &lt;code&gt;emfor.h&lt;/code&gt; and &lt;code&gt;emfor.pc&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Go&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install &lt;code&gt;libemfor&lt;/code&gt;, then &lt;code&gt;go get github.com/embark-systems/emfor/clients/go@v0.6.0&lt;/code&gt; (needs cgo and &lt;code&gt;pkg-config emfor&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Python&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install &lt;code&gt;libemfor&lt;/code&gt;, then &lt;code&gt;pip install clients/python&lt;/code&gt;. Set &lt;code&gt;EMFOR_LIBRARY&lt;/code&gt; if the library is outside the standard directories.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Perl&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install &lt;code&gt;libemfor&lt;/code&gt;, then in &lt;code&gt;clients/perl&lt;/code&gt;: &lt;code&gt;perl Makefile.PL &amp;amp;&amp;amp; make -f Makefile.xs &amp;amp;&amp;amp; make -f Makefile.xs install&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;PHP&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install &lt;code&gt;libemfor&lt;/code&gt;, then build the extension with &lt;code&gt;phpize&lt;/code&gt; (see &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/php/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;clients/php&lt;/a&gt;) and enable &lt;code&gt;extension=emfor&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Prefer the shared library: it exports only &lt;code&gt;emfor_*&lt;/code&gt; symbols, so its LMDB&#xA;cannot clash with another LMDB in the same process. &lt;a class=&#34;link&#34; href=&#34;../06-clients/&#34; &gt;Chapter 6&lt;/a&gt;&#xA;shows how to use each client.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>3. Configuration</title>
            <link>https://embark.systems/en/emfor/03-configuration/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/03-configuration/</guid>
            <description>&lt;p&gt;The daemon reads one TOML file, &lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt; by default&#xA;(&lt;a class=&#34;link&#34; href=&#34;../02-installation/#the-default-config-path&#34; &gt;chapter 2&lt;/a&gt;). Every key is&#xA;optional except the services&amp;rsquo; &lt;code&gt;check&lt;/code&gt; and &lt;code&gt;backends&lt;/code&gt;.&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/emfor.toml&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/emfor.toml&lt;/a&gt; shows every option with&#xA;comments.&lt;/p&gt;&#xA;&lt;h2 id=&#34;value-formats&#34;&gt;Value formats&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Kind&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Format&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Examples&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Duration&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A string with a unit&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;500ms&amp;quot;&lt;/code&gt;, &lt;code&gt;&amp;quot;2s&amp;quot;&lt;/code&gt;, &lt;code&gt;&amp;quot;1m&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Size&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A string with a unit&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;64MiB&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;File mode&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;An octal string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;0644&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Percentage&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A string ending in &lt;code&gt;%&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;10%&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Address&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;IPv4:port&lt;/code&gt; or &lt;code&gt;[IPv6]:port&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;10.0.0.11:8080&amp;quot;&lt;/code&gt;, &lt;code&gt;&amp;quot;[2001:db8::11]:8080&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Unknown keys anywhere in the file are errors, so a misspelled key never&#xA;goes unnoticed.&lt;/p&gt;&#xA;&lt;h2 id=&#34;top-level&#34;&gt;Top level&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;site&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The data centre this host is in. Used by &lt;code&gt;balance = &amp;quot;locality&amp;quot;&lt;/code&gt;, and required with &lt;code&gt;[peers]&lt;/code&gt;. 1–63 characters of &lt;code&gt;a-z&lt;/code&gt;, &lt;code&gt;0-9&lt;/code&gt; and &lt;code&gt;-&lt;/code&gt; (lowercased when read). Must come before the first table.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;include_dir&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A directory of further &lt;code&gt;.toml&lt;/code&gt; files that are merged into this config (&lt;a class=&#34;link&#34; href=&#34;#splitting-the-config-over-a-directory&#34; &gt;below&lt;/a&gt;). Only allowed in the main file. Must come before the first table.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;store&#34;&gt;[store]&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;store&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/var/lib/emfor/state.mdb&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;map_size&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;64MiB&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;file_mode&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;0644&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;lock_file_mode&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;0666&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;sync&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;false&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;path&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store file. Clients look here unless &lt;code&gt;EMFOR_STORE&lt;/code&gt; says otherwise.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;map_size&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;64MiB&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Largest size the store can grow to; at least &lt;code&gt;1MiB&lt;/code&gt;, rounded up to a multiple of &lt;code&gt;64KiB&lt;/code&gt; (LMDB needs a multiple of the page size). Commits fail with &lt;code&gt;MDB_MAP_FULL&lt;/code&gt; when it is reached.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;file_mode&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;0644&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Mode the store file is created with. Applications need to read it.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;lock_file_mode&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;0666&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Mode of &lt;code&gt;&amp;lt;path&amp;gt;-lock&lt;/code&gt;. Applications need to write it. See &lt;a class=&#34;link&#34; href=&#34;../08-operations/#who-can-use-the-store&#34; &gt;who can use the store&lt;/a&gt; before keeping the default.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;sync&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt; flushes every commit to disk. See &lt;a class=&#34;link&#34; href=&#34;../08-operations/#the-store-on-disk&#34; &gt;the store on disk&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;log&#34;&gt;[log]&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;level&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;info&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;trace&lt;/code&gt;, &lt;code&gt;debug&lt;/code&gt;, &lt;code&gt;info&lt;/code&gt;, &lt;code&gt;warn&lt;/code&gt; or &lt;code&gt;error&lt;/code&gt;. When the environment variable &lt;code&gt;RUST_LOG&lt;/code&gt; is set, it overrides this for the whole run.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;dns&#34;&gt;[dns]&#xA;&lt;/h2&gt;&lt;p&gt;The DNS server runs only when the table is present; an empty &lt;code&gt;[dns]&lt;/code&gt; table&#xA;turns it on with the defaults. See &lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;listen&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[&amp;quot;127.0.0.1:5300&amp;quot;]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Addresses to answer on, UDP and TCP.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;zone&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;svc.emfor&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Only names under the zone are answered. At most 242 characters. A service in the zone must leave room for the SRV target name of each backend, &lt;code&gt;&amp;lt;ip label&amp;gt;.&amp;lt;service&amp;gt;&lt;/code&gt; in at most 253 characters (an IPv6 label is 39).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;ttl&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;1s&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;TTL of answers and of the zone SOA; whole seconds from &lt;code&gt;0s&lt;/code&gt; to &lt;code&gt;3600s&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;rate_limit&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;100&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;UDP responses a second for each client network (IPv4 /24, IPv6 /56); &lt;code&gt;0&lt;/code&gt; is no limit. Loopback clients and TCP are never limited. See &lt;a class=&#34;link&#34; href=&#34;../07-dns/#rate-limit&#34; &gt;Rate limit&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;metrics&#34;&gt;[metrics]&#xA;&lt;/h2&gt;&lt;p&gt;The Prometheus endpoint runs only when the table is present.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;listen&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[&amp;quot;127.0.0.1:5301&amp;quot;]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Addresses to serve &lt;code&gt;/metrics&lt;/code&gt; on.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;peers&#34;&gt;[peers]&#xA;&lt;/h2&gt;&lt;p&gt;Optional. Makes this daemon a member of a &lt;strong&gt;peer group&lt;/strong&gt;: the emfor daemons&#xA;of one &lt;code&gt;site&lt;/code&gt; that list each other here. Members send each other a&#xA;heartbeat every second, and when one of them sees a backend go from&#xA;&lt;code&gt;healthy&lt;/code&gt; to &lt;code&gt;unhealthy&lt;/code&gt; it tells the others, which recheck that backend at&#xA;once and every 200ms (up to &lt;code&gt;fall&lt;/code&gt; checks) instead of waiting for their&#xA;next &lt;code&gt;interval&lt;/code&gt;. With three or more live members, three of them are&#xA;assigned to each backend: they check it every &lt;code&gt;interval&lt;/code&gt; and send their&#xA;results to the others; two &lt;code&gt;down&lt;/code&gt; results among those three mark the&#xA;backend down on every member. The other members let two agreeing results&#xA;decide and check the backend themselves about every third &lt;code&gt;interval&lt;/code&gt;, and&#xA;recheck every 200ms when their own check fails while the others see the&#xA;backend up. The&#xA;vote and the split need no setting of their own; they use the services&#39;&#xA;&lt;code&gt;interval&lt;/code&gt;, and only members whose &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;check_options&lt;/code&gt; and &lt;code&gt;timeout&lt;/code&gt;&#xA;match count. Clients still read only&#xA;the local store. Without &lt;code&gt;[peers]&lt;/code&gt; nothing of this runs. See&#xA;&lt;a class=&#34;link&#34; href=&#34;../01-concepts/#peer-groups&#34; &gt;chapter 1&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;A service with &lt;code&gt;peers = false&lt;/code&gt; (in its table or in &lt;code&gt;[defaults]&lt;/code&gt;) is left&#xA;out of the group on this host: it sends no reports or results, ignores&#xA;those it receives, and always decides from its own checks. The other&#xA;services stay in the group. The host announces the services it leaves out&#xA;in its heartbeats (as short hashes of their names, at most 80 of them), and&#xA;the other members then do not count it as one of those services&amp;rsquo; assigned&#xA;checkers, so their backends keep three where enough members are live. Each&#xA;host can therefore leave out its own services. Peer groups and this&#xA;announcement both came in 0.3.0, and 0.2.0 refuses a config with&#xA;&lt;code&gt;[peers]&lt;/code&gt;, so every member reads it. With more than 80 services left out&#xA;the host logs a warning and the others still count it for the rest.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt;                               &lt;span style=&#34;color:#75715e&#34;&gt;# required with [peers]&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;listen&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;members&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.7:5302&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;secret_file&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/etc/emfor/peers.key&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;listen&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;required&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;This daemon&amp;rsquo;s UDP address in the group, one of &lt;code&gt;members&lt;/code&gt;. A specific address, not &lt;code&gt;0.0.0.0&lt;/code&gt; or &lt;code&gt;::&lt;/code&gt;; other members check that messages come from it. 5302 is the usual port.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;members&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;required&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Every member&amp;rsquo;s &lt;code&gt;listen&lt;/code&gt;, this daemon included: 2 to 64 addresses, no repeats, all IPv4 or all IPv6 like &lt;code&gt;listen&lt;/code&gt;. The same list can go on every host of the site.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;secret_file&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;required&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Absolute path to the shared secret that authenticates and encrypts every message: at least 32 bytes after one trailing line feed is removed, not readable by other users and not writable by group or others (&lt;code&gt;chmod 600&lt;/code&gt; or &lt;code&gt;640&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;previous_secret_file&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Optional. Absolute path to the secret &lt;code&gt;secret_file&lt;/code&gt; held before its last change, with the same conditions. Read only at start, so that a daemon restarted during a secret change keeps signing with and accepting the previous secret as if it had not restarted (see below).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;encrypt&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt; sends every message encrypted, with keys derived from the secret: &lt;code&gt;EMP3&lt;/code&gt; (XChaCha20-Poly1305, 24-byte random nonce) to members that run this release, &lt;code&gt;EMP2&lt;/code&gt; (ChaCha20-Poly1305, 12-byte random nonce) to the others, picked per member as described below. &lt;code&gt;false&lt;/code&gt; sends them only signed (&lt;code&gt;EMP1&lt;/code&gt;: HMAC-SHA256), so backend names, addresses and reasons can be read on the network. All three are accepted either way; with &lt;code&gt;true&lt;/code&gt; the daemon logs a warning, once a minute per member, for each member that still sends &lt;code&gt;EMP1&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;accept_plaintext&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt; drops every &lt;code&gt;EMP1&lt;/code&gt; message (counted as &lt;code&gt;plaintext&lt;/code&gt; in &lt;code&gt;emfor_peer_messages_rejected_total&lt;/code&gt;), so nobody who learns the secret can make this daemon read a signed but unencrypted message. Needs &lt;code&gt;encrypt = true&lt;/code&gt;. Set it once every member sends &lt;code&gt;EMP2&lt;/code&gt; or &lt;code&gt;EMP3&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Members only share reports for a service when its check is defined the&#xA;same way on both: the same &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;check_options&lt;/code&gt; and &lt;code&gt;timeout&lt;/code&gt;. Other&#xA;settings (&lt;code&gt;interval&lt;/code&gt;, &lt;code&gt;rise&lt;/code&gt;, &lt;code&gt;fall&lt;/code&gt;, the backend list) may differ.&lt;/p&gt;&#xA;&lt;p&gt;The secret file is read again every second. To change the secret, write the&#xA;new one on every host within a minute of each other: after its own file&#xA;changed, a host still signs with the previous secret for 60 seconds and&#xA;accepts it for 120 seconds, so no message is dropped meanwhile. If the file&#xA;changes again within those 60 seconds (say, to fix a wrong secret), the&#xA;host keeps signing with the secret it signed with before, for 60 seconds&#xA;after the latest change.&lt;/p&gt;&#xA;&lt;p&gt;The previous secret is kept only in memory. A daemon restarted during a&#xA;change knows only the new secret: members whose file has not changed yet&#xA;drop its messages (&lt;code&gt;auth&lt;/code&gt;), and it theirs, until their files change. To&#xA;restart safely during a change, keep the old secret in a second file and&#xA;name it in &lt;code&gt;previous_secret_file&lt;/code&gt;, for example:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cp -p /etc/emfor/peers.key /etc/emfor/peers.key.old  &lt;span style=&#34;color:#75715e&#34;&gt;# keeps the mode&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;cat new.key &amp;gt; /etc/emfor/peers.key  &lt;span style=&#34;color:#75715e&#34;&gt;# the same new secret on every host; keeps the mode&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A daemon that starts within 120 seconds of the change to &lt;code&gt;secret_file&lt;/code&gt;&#xA;(by the file&amp;rsquo;s modification time) then signs with the old secret until 60&#xA;seconds after that change and accepts it until 120 seconds after it. Later&#xA;starts ignore &lt;code&gt;previous_secret_file&lt;/code&gt;, so it can stay in the config.&lt;/p&gt;&#xA;&lt;p&gt;Messages more than 5 seconds off the receiver&amp;rsquo;s clock are dropped, so keep&#xA;the hosts&amp;rsquo; clocks in sync (NTP).&lt;/p&gt;&#xA;&lt;p&gt;With &lt;code&gt;encrypt = true&lt;/code&gt; the format is picked per member. Every &lt;code&gt;EMP2&lt;/code&gt;&#xA;message this release sends carries &lt;code&gt;&amp;quot;emp3&amp;quot;: true&lt;/code&gt; in its encrypted&#xA;payload; a member is sent &lt;code&gt;EMP3&lt;/code&gt; once the latest encrypted message accepted&#xA;from it is &lt;code&gt;EMP3&lt;/code&gt; or such an &lt;code&gt;EMP2&lt;/code&gt;, and &lt;code&gt;EMP2&lt;/code&gt; otherwise: before any&#xA;message from it was accepted, after a start or a reload that added it, and&#xA;when it sends &lt;code&gt;EMP2&lt;/code&gt; without the field (0.4.0) or &lt;code&gt;EMP1&lt;/code&gt;. Each change is logged at&#xA;&lt;code&gt;info&lt;/code&gt; (&amp;ldquo;sending EMP3 to peer &amp;hellip; from now on&amp;rdquo;). &lt;code&gt;EMP3&lt;/code&gt; exists because a&#xA;12-byte random nonce is safe for about 2^32 messages per secret, which a&#xA;large group can send within days; a 24-byte one has no such limit. A&#xA;group running 0.4.0 needs no configuration change to upgrade: upgrade the&#xA;hosts one at a time, and members of this release use &lt;code&gt;EMP3&lt;/code&gt; between each&#xA;other after their first message, &lt;code&gt;EMP2&lt;/code&gt; with 0.4.0 members. Going back to&#xA;0.4.0 needs no change either, with &lt;code&gt;encrypt&lt;/code&gt; on or off: the other members&#xA;send the host &lt;code&gt;EMP2&lt;/code&gt; again once they accept its first message. Until then,&#xA;about one second after it starts, it drops their &lt;code&gt;EMP3&lt;/code&gt; messages (counted&#xA;as &lt;code&gt;malformed&lt;/code&gt;).&lt;/p&gt;&#xA;&lt;p&gt;Encryption was added in 0.4.0. Members running 0.3.0 send and accept&#xA;only &lt;code&gt;EMP1&lt;/code&gt; and drop encrypted messages (counted as &lt;code&gt;malformed&lt;/code&gt;), so to&#xA;upgrade a group from 0.3.0, set &lt;code&gt;encrypt = false&lt;/code&gt; on each host as it is&#xA;upgraded; once every member runs this release, remove the line (or set&#xA;&lt;code&gt;encrypt = true&lt;/code&gt;) and reload each daemon. Once no member logs the&#xA;plaintext warning, you can set &lt;code&gt;accept_plaintext = false&lt;/code&gt; everywhere and&#xA;reload. The encryption has no forward&#xA;secrecy: whoever later gets the secret can read messages recorded before.&#xA;Rotate the secret regularly, as above, and keep the file readable only by&#xA;the daemon.&lt;/p&gt;&#xA;&lt;h2 id=&#34;defaults&#34;&gt;[defaults]&#xA;&lt;/h2&gt;&lt;p&gt;Values every service uses unless it sets its own. The built-in values:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;defaults&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;2s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;timeout&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;1s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;rise&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;2&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;fall&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;jitter&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10%&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;on_all_down&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;open&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;balance&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;round-robin&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;peers&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;slow_start&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;0s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Rule&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;100ms&lt;/code&gt; to &lt;code&gt;24h&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Time from the end of one check of a backend to the start of the next.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;timeout&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;at least &lt;code&gt;10ms&lt;/code&gt;, at most &lt;code&gt;interval&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;How long one check may take. A plugin gets it in whole milliseconds, less a small margin.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;rise&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;at least 1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consecutive passes before a backend is healthy.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fall&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;at least 1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consecutive failures before a backend is unhealthy.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;jitter&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;0%&lt;/code&gt; to &lt;code&gt;50%&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Random spread of &lt;code&gt;interval&lt;/code&gt;, so checks do not line up.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;on_all_down&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;open&lt;/code&gt; or &lt;code&gt;closed&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;What lookups do when no backend is healthy (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#when-no-backend-is-healthy&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;see &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/&#34; &gt;chapter 5&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;How a backend is chosen.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;peers&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Whether the service takes part in the &lt;a class=&#34;link&#34; href=&#34;#peers&#34; &gt;peer group&lt;/a&gt;. No effect without &lt;code&gt;[peers]&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;slow_start&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;0s&lt;/code&gt; (off), or &lt;code&gt;1s&lt;/code&gt; to &lt;code&gt;1h&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;How long a backend that recovers gets a growing share of traffic (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#slow-start&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;servicesname&#34;&gt;[services.&amp;ldquo;name&amp;rdquo;]&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:8080&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;1s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;dns_ttl&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;5s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Required&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;yes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tcp&lt;/code&gt;, &lt;code&gt;http&lt;/code&gt;, &lt;code&gt;exec&lt;/code&gt;, or the name of a declared plugin.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;backends&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;yes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A non-empty list of backends (below).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check_options&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;no&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Options of the checker (&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/&#34; &gt;chapter 4&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval&lt;/code&gt;, &lt;code&gt;timeout&lt;/code&gt;, &lt;code&gt;rise&lt;/code&gt;, &lt;code&gt;fall&lt;/code&gt;, &lt;code&gt;jitter&lt;/code&gt;, &lt;code&gt;on_all_down&lt;/code&gt;, &lt;code&gt;balance&lt;/code&gt;, &lt;code&gt;peers&lt;/code&gt;, &lt;code&gt;slow_start&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;no&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Override &lt;code&gt;[defaults]&lt;/code&gt; for this service.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;dns_ttl&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;no&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Overrides &lt;code&gt;[dns] ttl&lt;/code&gt; for this service; whole seconds, &lt;code&gt;0s&lt;/code&gt; to &lt;code&gt;3600s&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;strong&gt;Service names&lt;/strong&gt; are host-name-like: lowercased, one trailing dot&#xA;removed, at most 253 characters of dot-separated labels, each 1–63&#xA;characters of &lt;code&gt;a-z&lt;/code&gt;, &lt;code&gt;0-9&lt;/code&gt; and &lt;code&gt;-&lt;/code&gt; not starting or ending with &lt;code&gt;-&lt;/code&gt;. Two&#xA;names that differ only in case are duplicates. Clients normalize the name&#xA;they look up the same way.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Backends&lt;/strong&gt; are strings, or tables when they need a weight, a site,&#xA;&lt;code&gt;backup&lt;/code&gt; or &lt;code&gt;drain&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.11:9200&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.12:9200&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;weight&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.1.4.11:9200&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc2&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.2.4.11:9200&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;backup&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.13:9200&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;drain&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;addr&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;required&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;IPv4:port&lt;/code&gt; or &lt;code&gt;[IPv6]:port&lt;/code&gt;, port 1–65535. Unique within the service.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;weight&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 to 100. Used by &lt;code&gt;balance = &amp;quot;weighted&amp;quot;&lt;/code&gt;, and by &lt;code&gt;&amp;quot;sticky&amp;quot;&lt;/code&gt; with an affinity key.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;site&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The backend&amp;rsquo;s data centre. Used by &lt;code&gt;balance = &amp;quot;locality&amp;quot;&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;backup&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;. A backup is used only while no other backend passes its checks (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#backup-backends&#34; &gt;chapter 5&lt;/a&gt;). At least one backend of a service must not be a backup.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;drain&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt; or &lt;code&gt;false&lt;/code&gt;. A drained backend is never chosen, by any &lt;code&gt;balance&lt;/code&gt; or in DNS, while its checks go on (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;code&gt;weight&lt;/code&gt;, &lt;code&gt;site&lt;/code&gt;, &lt;code&gt;backup&lt;/code&gt; and &lt;code&gt;drain&lt;/code&gt; are accepted under every &lt;code&gt;balance&lt;/code&gt;, so you&#xA;can switch &lt;code&gt;balance&lt;/code&gt; without editing the backends.&lt;/p&gt;&#xA;&lt;h2 id=&#34;pluginsname&#34;&gt;[plugins.&amp;ldquo;name&amp;rdquo;]&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;plugins&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;redis&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;command&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/usr/local/libexec/emfor-check&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;redis&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Key&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;command&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Required. The program and its arguments; the first element is an absolute path to an executable. No element may contain NUL. Run without a shell.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The name is 1–63 characters of &lt;code&gt;a-z&lt;/code&gt;, &lt;code&gt;0-9&lt;/code&gt;, &lt;code&gt;_&lt;/code&gt; and &lt;code&gt;-&lt;/code&gt;, and not &lt;code&gt;tcp&lt;/code&gt;,&#xA;&lt;code&gt;http&lt;/code&gt; or &lt;code&gt;exec&lt;/code&gt;. A service uses the plugin with &lt;code&gt;check = &amp;quot;&amp;lt;name&amp;gt;&amp;quot;&lt;/code&gt;. Every&#xA;declared plugin is started, used or not. See&#xA;&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#plugins-and-emfor-check&#34; &gt;chapter 4&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;splitting-the-config-over-a-directory&#34;&gt;Splitting the config over a directory&#xA;&lt;/h2&gt;&lt;p&gt;With &lt;code&gt;include_dir = &amp;quot;conf.d&amp;quot;&lt;/code&gt; in the main file, the daemon reads the main&#xA;file and then every &lt;code&gt;.toml&lt;/code&gt; file in that directory, and merges them into&#xA;one config before checking it. A relative path is relative to the directory&#xA;of the main file (&lt;code&gt;/etc/emfor/conf.d&lt;/code&gt; for &lt;code&gt;/etc/emfor/emfor.toml&lt;/code&gt;). This&#xA;lets a package own &lt;code&gt;emfor.toml&lt;/code&gt; while each team or configuration tool owns&#xA;the files for its services:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;/etc/emfor/emfor.toml            include_dir = &amp;#34;conf.d&amp;#34;, [store], [dns]&#xA;/etc/emfor/conf.d/01-plugins.toml&#xA;/etc/emfor/conf.d/10-web.toml&#xA;/etc/emfor/conf.d/99-local.toml&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;Which files, in what order.&lt;/strong&gt; Regular files, and links to regular&#xA;files, whose name ends in &lt;code&gt;.toml&lt;/code&gt; and does not start with &lt;code&gt;.&lt;/code&gt;. They are&#xA;read in the byte order of their names, not the order of your locale, and a&#xA;later file overrides an earlier one; the main file is the lowest layer.&#xA;Write the numbers with leading zeros: &lt;code&gt;9-a.toml&lt;/code&gt; sorts after &lt;code&gt;10-a.toml&lt;/code&gt;.&#xA;Everything else is ignored (&lt;code&gt;web.toml~&lt;/code&gt;, &lt;code&gt;.web.toml.swp&lt;/code&gt;,&#xA;&lt;code&gt;web.toml.disabled&lt;/code&gt;, subdirectories), so renaming a file takes it out of&#xA;the config. A name that is not valid UTF-8 is skipped with a warning in&#xA;the log. An empty directory is fine.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;How files merge.&lt;/strong&gt; Two tables merge key by key, as deep as they go.&#xA;Anything else is replaced as a whole by the later value: a number, a string,&#xA;a list (including &lt;code&gt;backends&lt;/code&gt; and &lt;code&gt;command&lt;/code&gt;), or a value of another type.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# emfor.toml&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.1:80&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.2:80&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;2s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# conf.d/99-local.toml&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;5s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The service keeps its &lt;code&gt;check&lt;/code&gt; and both &lt;code&gt;backends&lt;/code&gt; and gets &lt;code&gt;interval = &amp;quot;5s&amp;quot;&lt;/code&gt;. To change the backends, give the whole list again. A later file&#xA;cannot remove a service, a plugin or a backend; take it out of the file&#xA;that defines it, or rename that file so it is ignored. A service whose&#xA;merged table has no &lt;code&gt;check&lt;/code&gt; or no &lt;code&gt;backends&lt;/code&gt; is an error, as in a single&#xA;file. The merged config is checked, started, reloaded and shared with peers&#xA;exactly like the same text in one file; how it is split does not change&#xA;the service definitions peers compare.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Only in the main file.&lt;/strong&gt; &lt;code&gt;[store]&lt;/code&gt; and &lt;code&gt;include_dir&lt;/code&gt; itself. &lt;code&gt;emfor status&lt;/code&gt; and &lt;code&gt;emfor lookup&lt;/code&gt; read only the main file, so a fragment cannot&#xA;move the store, and a fragment cannot include another directory. Anyone&#xA;who only runs &lt;code&gt;status&lt;/code&gt; and &lt;code&gt;lookup&lt;/code&gt; needs to read the main file, not the&#xA;directory. Everything else, including &lt;code&gt;site&lt;/code&gt;, &lt;code&gt;[dns]&lt;/code&gt;, &lt;code&gt;[peers]&lt;/code&gt; and&#xA;&lt;code&gt;[plugins]&lt;/code&gt;, may be in any file.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Permissions.&lt;/strong&gt; The directory and every file read from it must not be&#xA;writable by group or others (mode &lt;code&gt;0755&lt;/code&gt; and &lt;code&gt;0644&lt;/code&gt; are fine, &lt;code&gt;0775&lt;/code&gt; and&#xA;&lt;code&gt;0664&lt;/code&gt; are not); otherwise the config is refused and the error names the&#xA;path and mode. The owner must be root or the user running emfor (for a&#xA;link, the owner of the file it points to); otherwise the config is refused&#xA;too, because a file owned by anyone else can be changed by that user. A&#xA;fragment can declare a plugin &lt;code&gt;command&lt;/code&gt; or an &lt;code&gt;exec&lt;/code&gt; check, so whoever can&#xA;write there can run programs as the daemon&amp;rsquo;s user. Run&#xA;&lt;code&gt;emfor check-config&lt;/code&gt; as the user the daemon runs as (the systemd unit and&#xA;the FreeBSD &lt;code&gt;configtest&lt;/code&gt; do); run by hand as root, it accepts only files&#xA;owned by root.&#xA;The main file is not checked this way, so protect it yourself (&lt;code&gt;0644&lt;/code&gt;,&#xA;owned by root).&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;The directory must exist.&lt;/strong&gt; A package creates it. A config whose&#xA;&lt;code&gt;include_dir&lt;/code&gt; does not exist is an error: the daemon logs it and exits 2 at&#xA;start, and a reload is rejected and the old config keeps running.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Errors name their file.&lt;/strong&gt; A TOML syntax error is reported as&#xA;&lt;code&gt;file:line:column&lt;/code&gt;, and any other error as its TOML path followed by the&#xA;file that set that key (&lt;code&gt;services.&amp;quot;api.svc.emfor&amp;quot;.interval: ... (in /etc/emfor/conf.d/99-local.toml)&lt;/code&gt;). When an error comes from values in two&#xA;files, such as a &lt;code&gt;timeout&lt;/code&gt; in one and an &lt;code&gt;interval&lt;/code&gt; in another, the file&#xA;named is the later one.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Writing the files.&lt;/strong&gt; Write a file to a temporary name outside the&#xA;directory or ending in something other than &lt;code&gt;.toml&lt;/code&gt;, then rename it into&#xA;place. A reload that reads a half-written file is rejected and the old&#xA;config stays, but an atomic rename avoids the failed reload.&lt;/p&gt;&#xA;&lt;h2 id=&#34;checking-a-config&#34;&gt;Checking a config&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor check-config&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;config ok: &lt;span style=&#34;color:#ae81ff&#34;&gt;8&lt;/span&gt; services, &lt;span style=&#34;color:#ae81ff&#34;&gt;19&lt;/span&gt; backends&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;emfor check-config&lt;/code&gt; checks the whole file and prints every error, each&#xA;with its TOML path, for example:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;services.&amp;#34;api.svc.emfor&amp;#34;.timeout: timeout must not exceed interval&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;With &lt;code&gt;include_dir&lt;/code&gt; set, it then lists the files it read, the main file&#xA;first and the others in the order they were merged:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;config ok: 8 services, 19 backends&#xA;read /etc/emfor/emfor.toml&#xA;read /etc/emfor/conf.d/10-web.toml&#xA;read /etc/emfor/conf.d/99-local.toml&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;The daemon logs the same list at start and after each reload. Without&#xA;&lt;code&gt;include_dir&lt;/code&gt; the output is the single line.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Seeing the merged config.&lt;/strong&gt; &lt;code&gt;emfor check-config --show&lt;/code&gt; checks the config&#xA;the same way, and then prints the merged config as TOML on standard output,&#xA;each value followed by the file that last set it:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;# include_dir = &amp;#34;conf.d&amp;#34; (its files are merged below)&#xA;&#xA;[services.&amp;#34;api.svc.emfor&amp;#34;]&#xA;backends = [&amp;#34;10.0.0.1:80&amp;#34;, &amp;#34;10.0.0.2:80&amp;#34;] # /etc/emfor/emfor.toml&#xA;check = &amp;#34;tcp&amp;#34; # /etc/emfor/emfor.toml&#xA;interval = &amp;#34;5s&amp;#34; # /etc/emfor/conf.d/99-local.toml&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;A list such as &lt;code&gt;backends&lt;/code&gt; is set as a whole, so its comment names the file&#xA;that set the list. The &lt;code&gt;config ok&lt;/code&gt; and &lt;code&gt;read&lt;/code&gt; lines go to standard error, so&#xA;&lt;code&gt;emfor check-config --show &amp;gt; merged.toml&lt;/code&gt; saves a file that checks again&#xA;anywhere, because the &lt;code&gt;include_dir&lt;/code&gt; line is written as a comment (its files&#xA;are merged already); comments aside the file has the same content. Relative&#xA;paths in the config other than &lt;code&gt;include_dir&lt;/code&gt; do not depend on the file&amp;rsquo;s&#xA;directory. An invalid config prints nothing&#xA;on standard output and exits 2. Without &lt;code&gt;include_dir&lt;/code&gt; the output starts with&#xA;one comment naming the file and has no other comments. The built-in defaults&#xA;are not added: a key you did not write does not appear. Nothing is&#xA;redacted; the config holds password file paths, not passwords, but a plugin&#xA;&lt;code&gt;command&lt;/code&gt; is printed as written, so do not put a secret there. The comment&#xA;format is for people; do not parse it.&lt;/p&gt;&#xA;&lt;p&gt;It exits 0 when the config is valid and 2 otherwise (1 when it cannot write its output, for example to a full disk; a closed pipe is not an error). It starts the declared&#xA;plugins so that they can check their services&amp;rsquo; &lt;code&gt;check_options&lt;/code&gt;, and stops&#xA;them before it exits. It does not open the store. A daemon started with an&#xA;invalid config logs the same errors and exits 2.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-a-reload-changes&#34;&gt;What a reload changes&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;SIGHUP&lt;/code&gt; (&lt;code&gt;systemctl reload emfor&lt;/code&gt;, &lt;code&gt;service emfor reload&lt;/code&gt;) re-reads the&#xA;file, and the files of &lt;code&gt;include_dir&lt;/code&gt; with it. A config with any error, in&#xA;any file, or with an &lt;code&gt;include_dir&lt;/code&gt; that has gone missing, is rejected as a&#xA;whole: the daemon logs every&#xA;error and keeps running the old config.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Change&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;On reload&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Services and backends added or removed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied; new backends start &lt;code&gt;unknown&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check&lt;/code&gt; or &lt;code&gt;check_options&lt;/code&gt; of a service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied; its backends start over from &lt;code&gt;unknown&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval&lt;/code&gt;, &lt;code&gt;timeout&lt;/code&gt;, &lt;code&gt;rise&lt;/code&gt;, &lt;code&gt;fall&lt;/code&gt;, &lt;code&gt;jitter&lt;/code&gt;, &lt;code&gt;on_all_down&lt;/code&gt;, &lt;code&gt;peers&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied; backends keep their state. A new &lt;code&gt;interval&lt;/code&gt; or &lt;code&gt;jitter&lt;/code&gt; also applies to the wait already under way: the next check is due the new wait after the last one ended, at once when that is past.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance&lt;/code&gt;, &lt;code&gt;weight&lt;/code&gt;, &lt;code&gt;site&lt;/code&gt; (top level and backends), &lt;code&gt;backup&lt;/code&gt;, &lt;code&gt;drain&lt;/code&gt;, &lt;code&gt;dns_ttl&lt;/code&gt;, &lt;code&gt;slow_start&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied; backends keep their state, and a backend in slow start keeps its recovery time.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[dns] zone&lt;/code&gt;, &lt;code&gt;[dns] ttl&lt;/code&gt;, &lt;code&gt;[dns] rate_limit&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[log] level&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied (unless &lt;code&gt;RUST_LOG&lt;/code&gt; is set).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A plugin&amp;rsquo;s &lt;code&gt;command&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The plugin is restarted, and the services that use it start over from &lt;code&gt;unknown&lt;/code&gt;. Unchanged plugins keep running.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Anything in &lt;code&gt;[store]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rejected; needs a restart.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[dns] listen&lt;/code&gt;, adding or removing &lt;code&gt;[dns]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rejected (&lt;code&gt;dns listen settings require restart&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[metrics] listen&lt;/code&gt;, adding or removing &lt;code&gt;[metrics]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rejected (&lt;code&gt;metrics listen settings require restart&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[peers] members&lt;/code&gt;, &lt;code&gt;[peers] secret_file&lt;/code&gt;, &lt;code&gt;[peers] encrypt&lt;/code&gt;, &lt;code&gt;[peers] accept_plaintext&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Applied; added members start as not live. &lt;code&gt;previous_secret_file&lt;/code&gt; is checked, but read only at start.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[peers] listen&lt;/code&gt;, adding or removing &lt;code&gt;[peers]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Rejected (&lt;code&gt;peers listen settings require restart&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;</description>
        </item><item>
            <title>4. Health checks</title>
            <link>https://embark.systems/en/emfor/04-health-checks/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/04-health-checks/</guid>
            <description>&lt;p&gt;A service&amp;rsquo;s &lt;code&gt;check&lt;/code&gt; decides how each backend is tested, and its&#xA;&lt;code&gt;check_options&lt;/code&gt; table configures that checker. Every check ends in &lt;code&gt;up&lt;/code&gt;,&#xA;&lt;code&gt;down&lt;/code&gt; or &lt;code&gt;error&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../01-concepts/#check-results-and-health-states&#34; &gt;chapter 1&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;code&gt;check&lt;/code&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Tests&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Good for&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tcp&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A TCP connection, optionally TLS and a send/expect exchange&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Any TCP service; greeting-based protocols such as SMTP&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;http&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One &lt;code&gt;GET&lt;/code&gt; request and its status code&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Web services and health endpoints&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;exec&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A program run once per check&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Existing Nagios-style scripts, infrequent checks&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;a plugin name&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A long-running plugin program&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;redis, MySQL, PostgreSQL, LDAP, DNS, RADIUS, SIP, Diameter, SMTP, AMQP, MongoDB, gRPC, QUIC and HTTP/3 (&lt;code&gt;emfor-check&lt;/code&gt;), or your own&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;A check that has not finished within the service&amp;rsquo;s &lt;code&gt;timeout&lt;/code&gt; is &lt;code&gt;down&lt;/code&gt;&#xA;with reason &lt;code&gt;timeout&lt;/code&gt;. The one exception is a plugin: a plugin that does not&#xA;answer in time gives &lt;code&gt;error&lt;/code&gt;, because the plugin, not the backend, is late.&lt;/p&gt;&#xA;&lt;h2 id=&#34;tcp&#34;&gt;tcp&#xA;&lt;/h2&gt;&lt;p&gt;Without options, a backend is &lt;code&gt;up&lt;/code&gt; when it accepts a connection; emfor&#xA;closes the connection at once. A refused or unreachable connection is&#xA;&lt;code&gt;down&lt;/code&gt;, also when this host has no route to the backend or its own&#xA;firewall blocks the connection: the applications on this host could not&#xA;reach the backend either. Only a shortage on this host (open files,&#xA;buffers, memory, local ports) makes it &lt;code&gt;error&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Option&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Type&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;send&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Bytes written once after connecting. TOML escapes such as &lt;code&gt;\r\n&lt;/code&gt; are sent as those bytes. Needs &lt;code&gt;expect&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;expect&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Bytes that must appear, case-sensitively, in what the backend sends (read up to 16384 bytes). At most 16384 bytes.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tls&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;boolean&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Do a TLS handshake right after connecting; never falls back to plaintext.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tls_verify&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;boolean&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;false&lt;/code&gt; accepts any certificate (still encrypted). Needs &lt;code&gt;tls = true&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;server_name&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;the backend IP&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Name (or IP) the certificate must match; sent as SNI when it is a name. Needs &lt;code&gt;tls = true&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;ca_file&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;absolute path&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;system CAs&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;PEM file of CA certificates to trust instead. When the config is loaded it must exist and hold at least one certificate usable as a CA. Needs &lt;code&gt;tls = true&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;smtp.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.6.11:25&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.6.12:25&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;expect&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;220&amp;#34;&lt;/span&gt; }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;memcache.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.7.11:11211&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;send&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;stats\r\n&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;expect&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;STAT pid&amp;#34;&lt;/span&gt; }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;imap.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.8.11:993&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;tls&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;server_name&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;imap.internal&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;expect&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;* OK&amp;#34;&lt;/span&gt; }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Typical reasons: &lt;code&gt;expect: got &amp;quot;421 too busy\r\n&amp;quot;&lt;/code&gt; (the first 64 bytes&#xA;received, escaped), &lt;code&gt;send: ...&lt;/code&gt;, &lt;code&gt;tls: ...&lt;/code&gt;,&#xA;&lt;code&gt;tls: no answer to the TLS handshake (does the port speak TLS?)&lt;/code&gt;, and&#xA;&lt;code&gt;tls: connection closed during the TLS handshake (does the server have a certificate?)&lt;/code&gt; when the server closes the connection instead of&#xA;answering, as Active Directory does on port 636 without a certificate.&lt;/p&gt;&#xA;&lt;h2 id=&#34;http&#34;&gt;http&#xA;&lt;/h2&gt;&lt;p&gt;One &lt;code&gt;GET&lt;/code&gt; per check to the backend&amp;rsquo;s IP and port. Redirects are not&#xA;followed.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Option&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Type&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;scheme&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;http&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;https&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;http&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Protocol.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;path&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string starting with &lt;code&gt;/&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;&amp;quot;/&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Request path. &lt;code&gt;#&lt;/code&gt;, &lt;code&gt;\&lt;/code&gt; and control characters are refused, because the request URL would cut or change the path there; a space or other special character is sent percent-encoded.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;host_header&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;Host&lt;/code&gt; header to send: a host name or IP address with an optional &lt;code&gt;:port&lt;/code&gt;, IPv6 in brackets (&lt;code&gt;api.internal&lt;/code&gt;, &lt;code&gt;api.internal:8443&lt;/code&gt;, &lt;code&gt;[2001:db8::1]&lt;/code&gt;). The connection still goes to the backend; with &lt;code&gt;https&lt;/code&gt; a host name is also the TLS server name.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;expect_status&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;non-empty list&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[&amp;quot;200-399&amp;quot;]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Status codes that count as &lt;code&gt;up&lt;/code&gt;: codes (&lt;code&gt;200&lt;/code&gt; or &lt;code&gt;&amp;quot;200&amp;quot;&lt;/code&gt;) or inclusive ranges (&lt;code&gt;&amp;quot;200-299&amp;quot;&lt;/code&gt;), all within 100–599.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tls_verify&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;boolean&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;For &lt;code&gt;https&lt;/code&gt;: &lt;code&gt;false&lt;/code&gt; accepts any certificate.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;expect_body&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;string&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Bytes that must appear, case-sensitively, in the body (read up to 65536 bytes). At most 16384 bytes.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;host_header&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.internal&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;expect_status&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;200-299&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;expect_body&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;&amp;#34;status&amp;#34;:&amp;#34;UP&amp;#34;&amp;#39;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;A response with another status is &lt;code&gt;down&lt;/code&gt; with the status code in the&#xA;reason, and its body is not read. With &lt;code&gt;expect_body&lt;/code&gt;, a response whose&#xA;status matches is read until the text is found, the body ends, or 65536&#xA;bytes have been read; without the text it is &lt;code&gt;down&lt;/code&gt; with&#xA;&lt;code&gt;expect_body: got &amp;quot;&amp;lt;the first 64 bytes, escaped&amp;gt;&amp;quot;&lt;/code&gt;, and a body that cannot&#xA;be read is &lt;code&gt;down&lt;/code&gt; with a reason starting &lt;code&gt;body: &lt;/code&gt;. Use it for health&#xA;endpoints that answer 200 either way and put the state in the body. The&#xA;latency &lt;code&gt;fastest&lt;/code&gt; uses then includes reading the body.&lt;/p&gt;&#xA;&lt;h2 id=&#34;exec&#34;&gt;exec&#xA;&lt;/h2&gt;&lt;p&gt;Runs a program once per check, without a shell. The backend is passed in&#xA;the environment variables &lt;code&gt;EMFOR_IP&lt;/code&gt;, &lt;code&gt;EMFOR_PORT&lt;/code&gt; and &lt;code&gt;EMFOR_SERVICE&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;report.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;exec&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.3.1:5432&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;timeout&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;5s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;command&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/usr/local/libexec/check_pg_replication&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;--max-lag&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;60&amp;#34;&lt;/span&gt;] }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Exit&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Result&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;up&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 or 2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;down&lt;/code&gt;; the reason is the first line of stdout, or &lt;code&gt;exit code N&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;any other code, killed by a signal, or failed to start&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;error&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;still running at &lt;code&gt;timeout&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;down&lt;/code&gt; (&lt;code&gt;timeout&lt;/code&gt;); the program and all its children are killed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;code&gt;command&lt;/code&gt; is the only option: a non-empty list of strings without NUL whose first element is an&#xA;absolute path to an executable. To run an existing Nagios plugin, which wants&#xA;&lt;code&gt;-H&lt;/code&gt; and &lt;code&gt;-p&lt;/code&gt; on its command line, see&#xA;&lt;a class=&#34;link&#34; href=&#34;../08-operations/#watching-emfor-from-nagios-icinga-or-zabbix&#34; &gt;Watching emfor from Nagios&lt;/a&gt;. Starting a process per check is costly; for&#xA;frequent checks of many backends, use a plugin.&lt;/p&gt;&#xA;&lt;h2 id=&#34;plugins-and-emfor-check&#34;&gt;Plugins and emfor-check&#xA;&lt;/h2&gt;&lt;p&gt;A plugin is a long-running program that the daemon starts once and sends&#xA;check requests to, as JSON lines on its stdin and stdout. One process&#xA;serves every service that uses it, and it can keep connections and state&#xA;between checks.&lt;/p&gt;&#xA;&lt;p&gt;emfor ships one plugin program, &lt;code&gt;emfor-check&lt;/code&gt;. Its argument selects the&#xA;check:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;plugins&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;pgsql&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;command&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/usr/local/libexec/emfor-check&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;pgsql&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;db.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;pgsql&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;balance&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;failover&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.3.1:5432&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.3.2:5432&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;user&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;database&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;postgres&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;role&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;primary&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;password_file&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/etc/emfor/pg.pass&amp;#34;&lt;/span&gt; }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Argument&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Up when&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;redis&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;PING&lt;/code&gt; answers &lt;code&gt;PONG&lt;/code&gt; (after &lt;code&gt;AUTH&lt;/code&gt; when a password is set)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;mysql&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The login succeeds and &lt;code&gt;SELECT 1&lt;/code&gt; returns 1 (tested with MariaDB, and MySQL 8.0 and 8.4)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;pgsql&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The login succeeds and &lt;code&gt;SELECT 1&lt;/code&gt; returns 1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;ldap&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A simple bind succeeds&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;dns&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One query gets an expected response code (and records, with &lt;code&gt;expect_answer&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;radius&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A Status-Server (or Access-Request) gets an accept signed with the shared secret&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;sip&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;An OPTIONS request gets a final status code in &lt;code&gt;expect_status&lt;/code&gt; (default 200 to 499)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;diameter&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A capabilities exchange (CER / CEA) and a device watchdog (DWR / DWA) succeed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;smtp&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The greeting is 220 and &lt;code&gt;EHLO&lt;/code&gt; answers 250, over STARTTLS or TLS when asked for&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;amqp&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;An AMQP 0-9-1 login succeeds and the vhost opens&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;mongodb&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;hello&lt;/code&gt; answers, the SCRAM-SHA-256 login succeeds when &lt;code&gt;user&lt;/code&gt; is set, and the role matches&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;grpc&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The standard health service (&lt;code&gt;grpc.health.v1.Health/Check&lt;/code&gt;) answers &lt;code&gt;SERVING&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;quic&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A QUIC handshake over UDP completes with a verified certificate and an ALPN in &lt;code&gt;alpn&lt;/code&gt;; with &lt;code&gt;http3 = true&lt;/code&gt;, an HTTP/3 &lt;code&gt;GET&lt;/code&gt; also gets a status in &lt;code&gt;expect_status&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Main options: &lt;code&gt;user&lt;/code&gt;, &lt;code&gt;password_file&lt;/code&gt; (never the password itself; read on&#xA;every check), &lt;code&gt;database&lt;/code&gt; (&lt;code&gt;pgsql&lt;/code&gt;), &lt;code&gt;role&lt;/code&gt; (&lt;code&gt;master&lt;/code&gt;/&lt;code&gt;replica&lt;/code&gt; for redis,&#xA;&lt;code&gt;primary&lt;/code&gt;/&lt;code&gt;replica&lt;/code&gt; for MySQL and PostgreSQL; &lt;code&gt;primary&lt;/code&gt; lets &lt;code&gt;failover&lt;/code&gt;&#xA;follow the primary, and &lt;code&gt;replica&lt;/code&gt; also needs replication to be running;&#xA;&lt;code&gt;primary&lt;/code&gt;/&lt;code&gt;secondary&lt;/code&gt; for MongoDB),&#xA;&lt;code&gt;max_lag&lt;/code&gt; (with &lt;code&gt;role = &amp;quot;replica&amp;quot;&lt;/code&gt;, the seconds a replica may be behind),&#xA;and &lt;code&gt;tls&lt;/code&gt;, &lt;code&gt;starttls&lt;/code&gt; (&lt;code&gt;ldap&lt;/code&gt;), &lt;code&gt;tls_verify&lt;/code&gt;, &lt;code&gt;server_name&lt;/code&gt;, &lt;code&gt;ca_file&lt;/code&gt;.&#xA;The &lt;code&gt;dns&lt;/code&gt; plugin takes &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;type&lt;/code&gt;, &lt;code&gt;transport&lt;/code&gt;, &lt;code&gt;recursion_desired&lt;/code&gt;,&#xA;&lt;code&gt;expect_rcode&lt;/code&gt; and &lt;code&gt;expect_answer&lt;/code&gt; instead, and the &lt;code&gt;radius&lt;/code&gt; plugin&#xA;&lt;code&gt;secret_file&lt;/code&gt;, &lt;code&gt;request&lt;/code&gt;, &lt;code&gt;application&lt;/code&gt;, &lt;code&gt;user&lt;/code&gt;, &lt;code&gt;password_file&lt;/code&gt;,&#xA;&lt;code&gt;reject_is_up&lt;/code&gt;, &lt;code&gt;nas_identifier&lt;/code&gt; and &lt;code&gt;require_message_authenticator&lt;/code&gt;, and&#xA;the &lt;code&gt;sip&lt;/code&gt; plugin &lt;code&gt;transport&lt;/code&gt;, &lt;code&gt;tls&lt;/code&gt; (with the TLS keys), &lt;code&gt;uri&lt;/code&gt;, &lt;code&gt;from&lt;/code&gt; and&#xA;&lt;code&gt;expect_status&lt;/code&gt;, and the &lt;code&gt;diameter&lt;/code&gt; plugin &lt;code&gt;origin_host&lt;/code&gt;, &lt;code&gt;origin_realm&lt;/code&gt;,&#xA;the application ids, &lt;code&gt;supported_vendor_ids&lt;/code&gt;, &lt;code&gt;host_ip_address&lt;/code&gt; and &lt;code&gt;tls&lt;/code&gt;&#xA;(with the TLS keys and the client certificate &lt;code&gt;tls_cert_file&lt;/code&gt; and&#xA;&lt;code&gt;tls_key_file&lt;/code&gt;), and the &lt;code&gt;smtp&lt;/code&gt; plugin &lt;code&gt;helo&lt;/code&gt;, &lt;code&gt;starttls&lt;/code&gt; and &lt;code&gt;tls&lt;/code&gt; (with&#xA;the TLS keys), and the &lt;code&gt;amqp&lt;/code&gt; plugin &lt;code&gt;user&lt;/code&gt;, &lt;code&gt;password_file&lt;/code&gt;, &lt;code&gt;vhost&lt;/code&gt; and&#xA;&lt;code&gt;tls&lt;/code&gt; (with the TLS keys), and the &lt;code&gt;mongodb&lt;/code&gt; plugin &lt;code&gt;user&lt;/code&gt;,&#xA;&lt;code&gt;password_file&lt;/code&gt;, &lt;code&gt;auth_source&lt;/code&gt;, &lt;code&gt;role&lt;/code&gt; and &lt;code&gt;tls&lt;/code&gt; (with the TLS keys), and the&#xA;&lt;code&gt;grpc&lt;/code&gt; plugin &lt;code&gt;service&lt;/code&gt; and &lt;code&gt;tls&lt;/code&gt; (with the TLS keys), and the &lt;code&gt;quic&lt;/code&gt; plugin&#xA;&lt;code&gt;alpn&lt;/code&gt;, &lt;code&gt;tls_verify&lt;/code&gt;, &lt;code&gt;server_name&lt;/code&gt;, &lt;code&gt;ca_file&lt;/code&gt; (always TLS, so no &lt;code&gt;tls&lt;/code&gt;&#xA;key), &lt;code&gt;http3&lt;/code&gt;, &lt;code&gt;path&lt;/code&gt;, &lt;code&gt;host_header&lt;/code&gt;, &lt;code&gt;expect_status&lt;/code&gt; and &lt;code&gt;expect_body&lt;/code&gt;.&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/plugins.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/plugins.md&lt;/a&gt; lists every option and&#xA;reason.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;emfor check-config&lt;/code&gt; and every reload start the plugins and have them&#xA;validate each service&amp;rsquo;s &lt;code&gt;check_options&lt;/code&gt;, so mistakes show up before the&#xA;config is applied. Validation builds what a check builds: a &lt;code&gt;ca_file&lt;/code&gt; must&#xA;hold a certificate usable as a CA, a &lt;code&gt;tls_key_file&lt;/code&gt; must be the key of&#xA;&lt;code&gt;tls_cert_file&lt;/code&gt;, and a value the protocol sends as a NUL-terminated or&#xA;NUL-separated string (the &lt;code&gt;mysql&lt;/code&gt; and &lt;code&gt;pgsql&lt;/code&gt; &lt;code&gt;user&lt;/code&gt;, the &lt;code&gt;pgsql&lt;/code&gt;&#xA;&lt;code&gt;database&lt;/code&gt; and password, the &lt;code&gt;amqp&lt;/code&gt; &lt;code&gt;user&lt;/code&gt; and password) must not contain&#xA;NUL. A message that must fit a size the protocol sets is built once: the&#xA;&lt;code&gt;amqp&lt;/code&gt; login must fit in the 4096-byte first frame, and the &lt;code&gt;diameter&lt;/code&gt;&#xA;Capabilities-Exchange-Request in 16777215 bytes.&lt;/p&gt;&#xA;&lt;p&gt;When a plugin exits, its pending checks become &lt;code&gt;error&lt;/code&gt; (&lt;code&gt;plugin exited&lt;/code&gt;),&#xA;and the daemon restarts it after 1 s, then 2 s, 4 s … up to 30 s; until&#xA;then its checks are &lt;code&gt;error&lt;/code&gt; (&lt;code&gt;plugin &amp;lt;name&amp;gt; unavailable&lt;/code&gt;). A crashing&#xA;plugin never marks backends down: an &lt;code&gt;error&lt;/code&gt; does not count as a failed&#xA;check. But a backend whose checks have given nothing but errors for longer&#xA;than &lt;code&gt;fall × interval&lt;/code&gt; goes back to &lt;code&gt;unknown&lt;/code&gt; (see&#xA;&lt;a class=&#34;link&#34; href=&#34;../01-concepts/&#34; &gt;chapter 1&lt;/a&gt;), so a plugin that keeps crashing leaves its&#xA;backends &lt;code&gt;unknown&lt;/code&gt; rather than &lt;code&gt;healthy&lt;/code&gt;. A check that panics in a plugin&#xA;built on the &lt;code&gt;emfor-plugin&lt;/code&gt; crate is answered &lt;code&gt;error&lt;/code&gt; with reason&#xA;&lt;code&gt;check panicked: &amp;lt;message&amp;gt;&lt;/code&gt;, and the plugin keeps running.&#xA;Lines a plugin writes to stderr are logged at &lt;code&gt;info&lt;/code&gt;, at most 100 at once&#xA;and 10 a second after that; the log says how many were dropped. Each&#xA;plugin runs in a process group of its own: when it exits or is stopped,&#xA;the processes it started are killed too.&lt;/p&gt;&#xA;&lt;p&gt;To write your own plugin, see&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/plugins/protocol-v1.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/plugins/protocol-v1.md&lt;/a&gt;; the &lt;code&gt;emfor-plugin&lt;/code&gt;&#xA;crate implements the protocol for Rust.&lt;/p&gt;&#xA;&lt;h2 id=&#34;scheduling&#34;&gt;Scheduling&#xA;&lt;/h2&gt;&lt;p&gt;Each backend is checked on its own schedule, and never by two checks at&#xA;once:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The first check after a start or reload begins at a random moment within&#xA;&lt;code&gt;min(interval, 200ms)&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;While the backend is &lt;code&gt;unknown&lt;/code&gt; and checks return &lt;code&gt;up&lt;/code&gt; or &lt;code&gt;down&lt;/code&gt;, the next&#xA;check follows &lt;code&gt;min(interval, 200ms)&lt;/code&gt; after the previous one ends. With&#xA;the defaults (&lt;code&gt;rise = 2&lt;/code&gt;), backends are known within about half a second.&#xA;This fast pace lasts &lt;code&gt;rise + fall&lt;/code&gt; checks at most: a backend whose&#xA;results keep changing so that neither count is reached stays &lt;code&gt;unknown&lt;/code&gt;&#xA;and is then checked every &lt;code&gt;interval&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Otherwise the next check starts &lt;code&gt;interval&lt;/code&gt; after the previous one ends,&#xA;shifted at random by up to &lt;code&gt;jitter&lt;/code&gt; (10% by default) either way.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Because the interval is counted from the end of a check, a slow backend is&#xA;not checked more often than every &lt;code&gt;interval&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;rise-fall-and-errors&#34;&gt;Rise, fall and errors&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;up&lt;/code&gt;: the success counter goes up and the failure counter resets; at&#xA;&lt;code&gt;rise&lt;/code&gt; the backend turns &lt;code&gt;healthy&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;down&lt;/code&gt;: the failure counter goes up and the success counter resets; at&#xA;&lt;code&gt;fall&lt;/code&gt; the backend turns &lt;code&gt;unhealthy&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;error&lt;/code&gt;: no counter and no state changes. When a backend has had only&#xA;errors for longer than &lt;code&gt;fall × interval&lt;/code&gt;, it goes back to &lt;code&gt;unknown&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Time to detect a failure is about &lt;code&gt;fall × interval&lt;/code&gt; (6 s with the&#xA;defaults); time to recover is about &lt;code&gt;rise × interval&lt;/code&gt; (4 s). Every state&#xA;change is logged at &lt;code&gt;info&lt;/code&gt; with the service, backend, old and new state,&#xA;and the reason.&lt;/p&gt;&#xA;&lt;h2 id=&#34;reasons&#34;&gt;Reasons&#xA;&lt;/h2&gt;&lt;p&gt;A &lt;code&gt;down&lt;/code&gt; or &lt;code&gt;error&lt;/code&gt; result carries a reason, shown by &lt;code&gt;emfor status&lt;/code&gt;, in&#xA;logs and in the state history. Before it is stored, the reason is made&#xA;safe: tab, CR and LF become &lt;code&gt;\t&lt;/code&gt;, &lt;code&gt;\r&lt;/code&gt; and &lt;code&gt;\n&lt;/code&gt;; other control characters&#xA;become &lt;code&gt;\u{1b}&lt;/code&gt; and the like; and a reason longer than 512 characters is&#xA;cut and ends in &lt;code&gt;…&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;latency&#34;&gt;Latency&#xA;&lt;/h2&gt;&lt;p&gt;For every backend the daemon measures how long each &lt;code&gt;up&lt;/code&gt; check takes and&#xA;keeps a smoothed value (each new sample weighs 5%). &lt;code&gt;emfor status&lt;/code&gt; and&#xA;&lt;code&gt;/metrics&lt;/code&gt; show it, and &lt;code&gt;balance = &amp;quot;fastest&amp;quot;&lt;/code&gt; uses it. Choose a check that&#xA;resembles real traffic when you rely on it.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>5. Choosing a backend</title>
            <link>https://embark.systems/en/emfor/05-backend-selection/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/05-backend-selection/</guid>
            <description>&lt;p&gt;When an application looks up a service, the client library first finds the&#xA;&lt;strong&gt;candidates&lt;/strong&gt; (normally the healthy backends whose last check passed, in&#xA;config order) and then&#xA;lets the service&amp;rsquo;s &lt;code&gt;balance&lt;/code&gt; choose one. The DNS server does the same for&#xA;A and AAAA queries, with a few differences noted below.&lt;/p&gt;&#xA;&lt;h2 id=&#34;balance-methods&#34;&gt;Balance methods&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;code&gt;balance&lt;/code&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Client lookup&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Use it for&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;round-robin&lt;/code&gt; (default)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Candidates in turn&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Stateless services&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;sticky&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The same affinity key gets the same backend while it is healthy&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Caches and sessions, where a user should keep hitting one backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;failover&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Always the first candidate in config order&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A primary with standbys&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;weighted&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;In proportion to each backend&amp;rsquo;s &lt;code&gt;weight&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Backends of different sizes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;locality&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Candidates in the host&amp;rsquo;s &lt;code&gt;site&lt;/code&gt; first, in turn&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Several data centres&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fastest&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;In turn among the backends whose checks answer fastest&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Backends at different network distances&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h3 id=&#34;round-robin&#34;&gt;round-robin&#xA;&lt;/h3&gt;&lt;p&gt;Each process keeps one counter per service, starting at a random value, so&#xA;the processes on a host spread out without coordinating. Each lookup takes&#xA;the next candidate.&lt;/p&gt;&#xA;&lt;h3 id=&#34;sticky&#34;&gt;sticky&#xA;&lt;/h3&gt;&lt;p&gt;The application passes an &lt;strong&gt;affinity key&lt;/strong&gt;, such as a user or session ID.&#xA;For each candidate, the client computes &lt;code&gt;MD5(key + backend id)&lt;/code&gt; and takes&#xA;the candidate with the highest value (rendezvous hashing). The same key&#xA;therefore picks the same backend in every process, on every host and in&#xA;every language; when that backend goes down, only its keys move, and they&#xA;come back when it recovers. Without a key (or with an empty one), &lt;code&gt;sticky&lt;/code&gt;&#xA;works like &lt;code&gt;round-robin&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;A backend with &lt;code&gt;weight&lt;/code&gt; w gets w times the keys of a backend with weight 1:&#xA;its value is the highest of &lt;code&gt;MD5(key + id)&lt;/code&gt; and &lt;code&gt;MD5(key + id + &amp;quot;#&amp;quot; + i)&lt;/code&gt;&#xA;for i from 1 to w − 1, so a backend of weight 3 gets 3/5 of the keys&#xA;next to two of weight 1. With every weight 1 the picks are the same as&#xA;in releases before this one. Raising a backend&amp;rsquo;s weight only moves keys&#xA;to that backend, and lowering it only moves keys away from it. Slow start&#xA;does not change the weights &lt;code&gt;sticky&lt;/code&gt; uses with a key. A weight above 100 in&#xA;the store, which the daemon never writes, counts as 100. A lookup computes&#xA;one MD5 per unit of weight among the candidates: about 0.8 µs for 10&#xA;backends of weight 1 and 72 µs for 10 of weight 100 (measured on an&#xA;Apple M4 Max), so keep weights small for sticky services that see many&#xA;lookups.&lt;/p&gt;&#xA;&lt;h3 id=&#34;failover&#34;&gt;failover&#xA;&lt;/h3&gt;&lt;p&gt;The first healthy backend in config order. List the primary first. With&#xA;the database plugins&amp;rsquo; &lt;code&gt;role&lt;/code&gt; option, only the current primary passes its&#xA;check, so &lt;code&gt;failover&lt;/code&gt; follows a switchover once the new primary reaches&#xA;&lt;code&gt;rise&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3 id=&#34;weighted&#34;&gt;weighted&#xA;&lt;/h3&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.11:9200&amp;#34;&lt;/span&gt;, { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.12:9200&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;weight&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt; }, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.4.13:9200&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Over every 5 lookups in one process, &lt;code&gt;10.0.4.12&lt;/code&gt; is returned 3 times and&#xA;the others once each. Only healthy backends count; their weights are&#xA;summed again when one goes down.&lt;/p&gt;&#xA;&lt;h3 id=&#34;locality&#34;&gt;locality&#xA;&lt;/h3&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;cache.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;balance&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;locality&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.5.11:6379&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.5.12:6379&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc1&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.1.5.11:6379&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;site&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;dc2&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;While any backend in the host&amp;rsquo;s site is a candidate, lookups rotate over&#xA;those; otherwise over all candidates. Without a top-level &lt;code&gt;site&lt;/code&gt;,&#xA;&lt;code&gt;locality&lt;/code&gt; works like &lt;code&gt;round-robin&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3 id=&#34;fastest&#34;&gt;fastest&#xA;&lt;/h3&gt;&lt;p&gt;The daemon decides which healthy backends are fast, from their smoothed&#xA;check latency, and writes this &lt;strong&gt;pool&lt;/strong&gt; to the store; clients rotate over&#xA;the pool. A backend joins the pool when its latency is within 1.5 times&#xA;the lowest (or 1 ms above it), and leaves when it is over 2 times the&#xA;lowest (and more than 2 ms above it), so that small variations do not move&#xA;backends in and out. When the pool is empty, or the store is stale,&#xA;&lt;code&gt;fastest&lt;/code&gt; works like &lt;code&gt;round-robin&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h3 id=&#34;unknown-values&#34;&gt;Unknown values&#xA;&lt;/h3&gt;&lt;p&gt;A client that reads a &lt;code&gt;balance&lt;/code&gt; it does not know (written by a newer&#xA;daemon) uses &lt;code&gt;round-robin&lt;/code&gt;. A backend &lt;code&gt;state&lt;/code&gt; it does not know counts as&#xA;&lt;code&gt;unhealthy&lt;/code&gt;, so such a backend is used only when an &lt;code&gt;open&lt;/code&gt; service has&#xA;nothing better, and an &lt;code&gt;on_all_down&lt;/code&gt; it does not know counts as &lt;code&gt;open&lt;/code&gt;,&#xA;the default. &lt;code&gt;emfor status&lt;/code&gt; shows them as &lt;code&gt;unrecognized&lt;/code&gt; and&#xA;&lt;code&gt;on_all_down=unknown&lt;/code&gt;. Unknown fields in the store are ignored.&lt;/p&gt;&#xA;&lt;h2 id=&#34;a-backend-whose-checks-start-failing&#34;&gt;A backend whose checks start failing&#xA;&lt;/h2&gt;&lt;p&gt;A backend turns unhealthy only after &lt;code&gt;fall&lt;/code&gt; failed checks in a row; until&#xA;then it is still healthy, but its last check failed and its &lt;code&gt;reason&lt;/code&gt; is&#xA;set (as &lt;code&gt;emfor status&lt;/code&gt; shows). While at least one healthy backend passed&#xA;its last check, such a failing backend is left out of the candidates, so&#xA;lookups and DNS answers stop choosing it at its first failed check. When&#xA;every healthy backend is failing, all of them are candidates, as before.&#xA;A &lt;code&gt;down&lt;/code&gt; and an &lt;code&gt;error&lt;/code&gt; result count alike: if one checker problem hits&#xA;every backend, nothing changes.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The client sees a new &lt;code&gt;reason&lt;/code&gt; up to about one second late (the daemon&#xA;writes &lt;code&gt;reason&lt;/code&gt; changes to the store with its heartbeat); the DNS server&#xA;answers from the daemon&amp;rsquo;s memory and sees it at once.&lt;/li&gt;&#xA;&lt;li&gt;With &lt;code&gt;sticky&lt;/code&gt;, the keys of a failing backend move to the other&#xA;candidates and come back after its next passed check.&lt;/li&gt;&#xA;&lt;li&gt;In a peer group, this host&amp;rsquo;s own failed check counts even while its&#xA;peers still find the backend up (the reason starts with &lt;code&gt;local: &lt;/code&gt;), so&#xA;this host stops choosing it at once.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;backup-backends&#34;&gt;Backup backends&#xA;&lt;/h2&gt;&lt;p&gt;A backend with &lt;code&gt;backup = true&lt;/code&gt; is held back for when the others cannot&#xA;serve, for example a replica in another data centre or a smaller machine.&#xA;While the store is fresh, the candidates are the first of these that is&#xA;not empty:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the primary backends (not backups) whose last check passed;&lt;/li&gt;&#xA;&lt;li&gt;the backups whose last check passed;&lt;/li&gt;&#xA;&lt;li&gt;the healthy primary backends, failing ones included;&lt;/li&gt;&#xA;&lt;li&gt;the healthy backups, failing ones included.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;So backups take over as soon as no primary passes its checks, not only&#xA;after the primaries turn unhealthy, and they give way again at a&#xA;primary&amp;rsquo;s next passed check. Within the chosen set, &lt;code&gt;balance&lt;/code&gt; works as&#xA;usual: &lt;code&gt;failover&lt;/code&gt; takes the first backup in config order, &lt;code&gt;weighted&lt;/code&gt;&#xA;uses the backups&amp;rsquo; weights, and so on. &lt;code&gt;fastest&lt;/code&gt; decides which backends&#xA;are fast among the primaries and among the backups separately, so a fast&#xA;backup does not push the primaries out of the pool. When no backend is&#xA;healthy at all,&#xA;&lt;code&gt;on_all_down&lt;/code&gt; decides and primaries and backups count alike (see below).&#xA;A stale store also uses every backend.&lt;/p&gt;&#xA;&lt;p&gt;DNS answers follow the same order; SRV records list only the chosen set.&#xA;Backups need clients that read store format version 2 (&lt;a class=&#34;link&#34; href=&#34;../06-clients/#store-format-version&#34; &gt;chapter&#xA;6&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;h2 id=&#34;draining-a-backend&#34;&gt;Draining a backend&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;drain = true&lt;/code&gt; on a backend (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#servicesname&#34; &gt;chapter 3&lt;/a&gt;)&#xA;takes it out of every choice, for an upgrade, a reboot or an investigation,&#xA;without removing it from the config. Edit the file and reload&#xA;(&lt;code&gt;systemctl reload emfor&lt;/code&gt;); there is no command that drains at run time.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A drained backend is never chosen: not by any &lt;code&gt;balance&lt;/code&gt; (&lt;code&gt;sticky&lt;/code&gt; and&#xA;&lt;code&gt;failover&lt;/code&gt; included, so a sticky key moves to another backend until the&#xA;drain is removed), not in the last tier of an open service, not in DNS&#xA;A, AAAA and SRV answers, and not in the pool of a &lt;code&gt;fastest&lt;/code&gt; service.&#xA;emfor is not in the data path, so connections that are already open are&#xA;not affected; only new lookups skip it.&lt;/li&gt;&#xA;&lt;li&gt;Its checks keep running. &lt;code&gt;emfor status&lt;/code&gt; shows &lt;code&gt;drained&lt;/code&gt; in place of the&#xA;state, and &lt;code&gt;(checks: healthy)&lt;/code&gt; as the last marker on the line,&#xA;with the state the checks give, so you see at once whether it is ready;&#xA;when you remove &lt;code&gt;drain&lt;/code&gt; and reload, it is back in the choice at once,&#xA;with that state, without counting &lt;code&gt;rise&lt;/code&gt; again. Draining and undraining never start slow start by themselves; a backend that recovered while it was drained ramps from that recovery as usual, so if you undrain it within &lt;code&gt;slow_start&lt;/code&gt; of coming back, it ramps.&lt;/li&gt;&#xA;&lt;li&gt;When every backend of a service is drained, nothing can be chosen: the&#xA;lookup fails with &lt;code&gt;ALL_DOWN&lt;/code&gt; and DNS answers &lt;code&gt;SERVFAIL&lt;/code&gt;, whatever&#xA;&lt;code&gt;on_all_down&lt;/code&gt; is, because &lt;code&gt;open&lt;/code&gt; means &amp;ldquo;choose among the failing&#xA;ones&amp;rdquo; and a drained backend is not one of them.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;drain&lt;/code&gt; is a setting of this host. It is not sent to the peers of a peer&#xA;group and does not change the service definition they compare, so hosts&#xA;can drain different backends.&lt;/li&gt;&#xA;&lt;li&gt;Older clients read a drained backend as an unhealthy one (the store&#xA;holds &lt;code&gt;unhealthy&lt;/code&gt; with the reason &lt;code&gt;drained&lt;/code&gt;). They skip it too, except in&#xA;the last tier of an open service, when every backend is down: then an&#xA;older client can still choose it. Upgrade &lt;code&gt;libemfor&lt;/code&gt; on the hosts before&#xA;you rely on &lt;code&gt;drain&lt;/code&gt; for an open service.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;This is not HAProxy&amp;rsquo;s &lt;code&gt;DRAIN&lt;/code&gt;, which lets existing sessions continue:&#xA;emfor has no sessions, and a lookup that skips the backend is the whole&#xA;effect.&lt;/p&gt;&#xA;&lt;p&gt;For a deploy that waits for the running requests to end and then puts the&#xA;backend back, see &lt;a class=&#34;link&#34; href=&#34;../08-operations/#rolling-deploy-with-drain&#34; &gt;Rolling deploy with drain&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;slow-start&#34;&gt;Slow start&#xA;&lt;/h2&gt;&lt;p&gt;A backend that comes back may have cold caches or connection pools. With&#xA;&lt;code&gt;slow_start&lt;/code&gt; set (for example &lt;code&gt;slow_start = &amp;quot;30s&amp;quot;&lt;/code&gt;), a backend that turns&#xA;from &lt;code&gt;unhealthy&lt;/code&gt; to &lt;code&gt;healthy&lt;/code&gt; gets a growing share of traffic over that&#xA;time instead of its full share at once.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;The time is split into 10 equal steps. In the first tenth the backend&#xA;counts as 1/10 of its weight, in the second as 2/10, and so on; after&#xA;&lt;code&gt;slow_start&lt;/code&gt; it counts in full. Its weight is its &lt;code&gt;weight&lt;/code&gt; for&#xA;&lt;code&gt;weighted&lt;/code&gt; and 1 for every other method.&lt;/li&gt;&#xA;&lt;li&gt;While at least one candidate is in slow start, &lt;code&gt;round-robin&lt;/code&gt;,&#xA;&lt;code&gt;weighted&lt;/code&gt;, &lt;code&gt;locality&lt;/code&gt;, &lt;code&gt;fastest&lt;/code&gt; and &lt;code&gt;sticky&lt;/code&gt; without a key choose by&#xA;these reduced weights. &lt;code&gt;locality&lt;/code&gt; and &lt;code&gt;fastest&lt;/code&gt; first narrow to the&#xA;site or the pool, as usual. &lt;code&gt;failover&lt;/code&gt; and &lt;code&gt;sticky&lt;/code&gt; with a key are not&#xA;affected.&lt;/li&gt;&#xA;&lt;li&gt;Only a recovery counts: a backend that is &lt;code&gt;unknown&lt;/code&gt; when the daemon&#xA;starts, or that a reload adds or restarts, gets its full share at once.&#xA;A backend that turns unhealthy again leaves slow start.&lt;/li&gt;&#xA;&lt;li&gt;It has an effect only when some backends recover and others are already&#xA;healthy. A service with one backend still sends it everything, and&#xA;backends that recover together keep equal shares.&lt;/li&gt;&#xA;&lt;li&gt;A stale store has no slow start, and a clock that went back ends it.&lt;/li&gt;&#xA;&lt;li&gt;Clients older than this feature ignore it: a recovered backend gets its&#xA;full share at once.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;DNS A/AAAA answers follow the same reduced weights, and SRV records carry&#xA;them as their weight (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;chapter 7&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;h2 id=&#34;when-no-backend-is-healthy&#34;&gt;When no backend is healthy&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;code&gt;on_all_down&lt;/code&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Lookup&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;DNS A/AAAA/SRV&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;open&lt;/code&gt; (default)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Chooses among the open candidates below&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Answers with the open candidates below&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;closed&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fails with &lt;code&gt;ALL_DOWN&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;With &lt;code&gt;open&lt;/code&gt;, the candidates are, in this order, the first set that is not&#xA;empty:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;the &lt;code&gt;unknown&lt;/code&gt; backends whose last check passed (no &lt;code&gt;reason&lt;/code&gt;);&lt;/li&gt;&#xA;&lt;li&gt;every &lt;code&gt;unknown&lt;/code&gt; backend;&lt;/li&gt;&#xA;&lt;li&gt;every backend.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;An &lt;code&gt;unhealthy&lt;/code&gt; backend failed &lt;code&gt;fall&lt;/code&gt; checks in a row, while an &lt;code&gt;unknown&lt;/code&gt;&#xA;one is not decided yet (just started, added or reset, or its checks only&#xA;gave errors for &lt;code&gt;fall × interval&lt;/code&gt;), so it is the better bet. When the&#xA;daemon starts, every backend is &lt;code&gt;unknown&lt;/code&gt; and all of them are candidates.&#xA;A stale store uses every backend. Clients older than this rule choose&#xA;among all backends.&lt;/p&gt;&#xA;&lt;p&gt;The lookup does not tell the application that it came from an all-down&#xA;service. Use &lt;code&gt;open&lt;/code&gt; when trying a backend is better than failing (the&#xA;checks themselves may be at fault), and &lt;code&gt;closed&lt;/code&gt; when sending traffic to a&#xA;broken backend is worse than failing fast.&lt;/p&gt;&#xA;&lt;h2 id=&#34;a-stale-store&#34;&gt;A stale store&#xA;&lt;/h2&gt;&lt;p&gt;When the daemon&amp;rsquo;s heartbeat is older than &lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt; (3 s by&#xA;default), or the daemon stopped cleanly, the client treats no backend as&#xA;healthy and &lt;code&gt;on_all_down&lt;/code&gt; decides, exactly as above. A &lt;code&gt;fastest&lt;/code&gt; service&#xA;ignores its pool. Lookups never block or wait for the daemon.&lt;/p&gt;&#xA;&lt;h2 id=&#34;dns-differences&#34;&gt;DNS differences&#xA;&lt;/h2&gt;&lt;p&gt;The DNS server answers from the daemon&amp;rsquo;s memory with the same health&#xA;states, but:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;An A query considers only IPv4 backends and an AAAA query only IPv6&#xA;backends. A service with no backend of that family gets an empty answer.&#xA;Client lookups consider both families together.&lt;/li&gt;&#xA;&lt;li&gt;Round-robin and weighted choices are made by the daemon, once per query,&#xA;not per application process.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;sticky&lt;/code&gt; has no affinity key over DNS and works like &lt;code&gt;round-robin&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;SRV answers list every candidate, with priority and weight set from&#xA;&lt;code&gt;balance&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;chapter 7&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Resolver caching can return the same answer for up to the TTL.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</description>
        </item><item>
            <title>6. Client libraries</title>
            <link>https://embark.systems/en/emfor/06-clients/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/06-clients/</guid>
            <description>&lt;p&gt;A client library has one call: look up a service, get back one backend&#xA;address. It reads the local store, so it needs the emfor daemon running on&#xA;the same host, and nothing else.&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-lookup&#34;&gt;The lookup&#xA;&lt;/h2&gt;&lt;p&gt;Every client takes a service name and an optional affinity key, and&#xA;returns a string:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;ip:port&lt;/code&gt; for IPv4, such as &lt;code&gt;10.0.0.11:8080&lt;/code&gt;;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;[ip]:port&lt;/code&gt; for IPv6, such as &lt;code&gt;[2001:db8::11]:8080&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Both forms can be passed to the usual &amp;ldquo;connect to host:port&amp;rdquo; functions of&#xA;most languages. The name is matched without regard to case and with one&#xA;trailing dot ignored. The affinity key matters only for&#xA;&lt;code&gt;balance = &amp;quot;sticky&amp;quot;&lt;/code&gt;; an empty key is no key.&lt;/p&gt;&#xA;&lt;p&gt;Call &lt;code&gt;lookup&lt;/code&gt; for every new connection, not once at startup: the answer&#xA;changes as backends fail and recover, and a lookup costs about a&#xA;microsecond.&lt;/p&gt;&#xA;&lt;h2 id=&#34;errors&#34;&gt;Errors&#xA;&lt;/h2&gt;&lt;p&gt;Every client fails with one of four codes, named the same in every&#xA;language:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Code&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What to do&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store cannot be used: it does not exist, cannot be opened, was written by another emfor release&amp;rsquo;s LMDB, is not initialized, or &lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt; is invalid.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Check that the daemon runs, &lt;code&gt;EMFOR_STORE&lt;/code&gt;, file permissions, and that &lt;code&gt;libemfor&lt;/code&gt; comes from the daemon&amp;rsquo;s release. The message says which.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;INCOMPATIBLE&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store has another format version, or a record cannot be read.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install the client from the daemon&amp;rsquo;s release.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NOT_FOUND&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No service of that name.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Check the name against the config.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;ALL_DOWN&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No healthy backend, and the service has &lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Fail the request, or retry later.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;A stale store is not an error of its own: it makes every backend count as&#xA;unhealthy, so an &lt;code&gt;open&lt;/code&gt; service still returns an address and a &lt;code&gt;closed&lt;/code&gt;&#xA;one fails with &lt;code&gt;ALL_DOWN&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;configuration&#34;&gt;Configuration&#xA;&lt;/h2&gt;&lt;p&gt;Clients have no configuration call. Two environment variables are read on&#xA;the first lookup in a process (and again in a child after &lt;code&gt;fork&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Variable&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Default&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;EMFOR_STORE&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;/var/lib/emfor/state.mdb&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Path of the store; must match the daemon&amp;rsquo;s &lt;code&gt;[store] path&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;3s&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;How old the heartbeat may be: a positive whole number with &lt;code&gt;ms&lt;/code&gt; or &lt;code&gt;s&lt;/code&gt;, such as &lt;code&gt;1500ms&lt;/code&gt;. An invalid value makes every lookup fail with &lt;code&gt;UNAVAILABLE&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;examples&#34;&gt;Examples&#xA;&lt;/h2&gt;&lt;p&gt;&lt;strong&gt;Rust&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-rust&#34; data-lang=&#34;rust&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;match&lt;/span&gt; emfor_client::lookup(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, Some(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;user-42&amp;#34;&lt;/span&gt;)) {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    Ok(addr) &lt;span style=&#34;color:#f92672&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;println!&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;connect to &lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;{addr}&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;),&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    Err(e) &lt;span style=&#34;color:#f92672&#34;&gt;=&amp;gt;&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;eprintln!&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;{e}&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;), &lt;span style=&#34;color:#75715e&#34;&gt;// &amp;#34;ALL_DOWN: ...&amp;#34;; e.code is an ErrorCode&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;C&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-c&#34; data-lang=&#34;c&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;#include&lt;/span&gt; &lt;span style=&#34;color:#75715e&#34;&gt;&amp;lt;emfor.h&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;char&lt;/span&gt; addr[EMFOR_ADDR_MAX];&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor_status st &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;emfor_lookup&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;user-42&amp;#34;&lt;/span&gt;, addr, &lt;span style=&#34;color:#66d9ef&#34;&gt;sizeof&lt;/span&gt; addr);&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;if&lt;/span&gt; (st &lt;span style=&#34;color:#f92672&#34;&gt;==&lt;/span&gt; EMFOR_OK)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#a6e22e&#34;&gt;printf&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;connect to %s&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;, addr);&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;else&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#a6e22e&#34;&gt;fprintf&lt;/span&gt;(stderr, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;%s: %s&lt;/span&gt;&lt;span style=&#34;color:#ae81ff&#34;&gt;\n&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;emfor_status_name&lt;/span&gt;(st), &lt;span style=&#34;color:#a6e22e&#34;&gt;emfor_last_error&lt;/span&gt;());&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Build with &lt;code&gt;cc app.c $(pkg-config --cflags --libs emfor)&lt;/code&gt;. C also has&#xA;&lt;code&gt;EMFOR_INVALID_ARGUMENT&lt;/code&gt; for a NULL or non-UTF-8 argument or a buffer that&#xA;is too small; &lt;code&gt;EMFOR_ADDR_MAX&lt;/code&gt; bytes always fit.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Go&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-go&#34; data-lang=&#34;go&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;github.com/embark-systems/emfor/clients/go&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;err&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;:=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;emfor&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Lookup&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;user-42&amp;#34;&lt;/span&gt;)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;var&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;e&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;*&lt;/span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;emfor&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Error&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;if&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;errors&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;As&lt;/span&gt;(&lt;span style=&#34;color:#a6e22e&#34;&gt;err&lt;/span&gt;, &lt;span style=&#34;color:#f92672&#34;&gt;&amp;amp;&lt;/span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;e&lt;/span&gt;) &lt;span style=&#34;color:#f92672&#34;&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;e&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;Code&lt;/span&gt; &lt;span style=&#34;color:#f92672&#34;&gt;==&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;emfor&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;AllDown&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#x9;&lt;span style=&#34;color:#75715e&#34;&gt;// no healthy backend&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;Python&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-python&#34; data-lang=&#34;python&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;import&lt;/span&gt; emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;try&lt;/span&gt;:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    addr &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; emfor&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;lookup(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, affinity_key&lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;user-42&amp;#34;&lt;/span&gt;)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;except&lt;/span&gt; emfor&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;AllDownError:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;...&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;except&lt;/span&gt; emfor&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;Error &lt;span style=&#34;color:#66d9ef&#34;&gt;as&lt;/span&gt; e:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    log&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;warning(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;: &lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;%s&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;&lt;/span&gt;, e&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;code, e&lt;span style=&#34;color:#f92672&#34;&gt;.&lt;/span&gt;message)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;Perl&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-perl&#34; data-lang=&#34;perl&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;use&lt;/span&gt; Emfor;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;my&lt;/span&gt; $addr &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; eval { Emfor::lookup(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;api.svc.emfor&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;user-42&amp;#39;&lt;/span&gt;) };&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;warn &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;$@\n&amp;#34;&lt;/span&gt; &lt;span style=&#34;color:#66d9ef&#34;&gt;if&lt;/span&gt; $@;   &lt;span style=&#34;color:#75715e&#34;&gt;# an Emfor::Error with -&amp;gt;code and -&amp;gt;message&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;strong&gt;PHP&lt;/strong&gt;&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-php&#34; data-lang=&#34;php&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;try&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    $addr &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#a6e22e&#34;&gt;Emfor\lookup&lt;/span&gt;(&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;api.svc.emfor&amp;#39;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#39;user-42&amp;#39;&lt;/span&gt;);&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;} &lt;span style=&#34;color:#66d9ef&#34;&gt;catch&lt;/span&gt; (&lt;span style=&#34;color:#a6e22e&#34;&gt;Emfor\LookupException&lt;/span&gt; $e) {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#a6e22e&#34;&gt;error_log&lt;/span&gt;($e&lt;span style=&#34;color:#f92672&#34;&gt;-&amp;gt;&lt;/span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;getMessage&lt;/span&gt;());   &lt;span style=&#34;color:#75715e&#34;&gt;// $e-&amp;gt;errorCode is &amp;#34;ALL_DOWN&amp;#34; etc.&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Each client&amp;rsquo;s README covers installing it and its exact API:&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/rust/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Rust&lt;/a&gt;, &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/c/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;C&lt;/a&gt;,&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/go/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Go&lt;/a&gt;, &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/python/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Python&lt;/a&gt;,&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/perl/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Perl&lt;/a&gt;, &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/clients/php/README.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;PHP&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;from-the-shell-emfor-lookup&#34;&gt;From the shell: emfor lookup&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor lookup&lt;/code&gt; answers from the shell what a client library would answer,&#xA;for scripts and cron jobs that have no client, and with &lt;code&gt;--explain&lt;/code&gt; why.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-console&#34; data-lang=&#34;console&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor lookup api.svc.emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor lookup api.svc.emfor --key user-42&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It reads the store at the &lt;code&gt;[store] path&lt;/code&gt; of the config (&lt;code&gt;--config&lt;/code&gt;, as&#xA;&lt;code&gt;emfor status&lt;/code&gt; does) and applies &lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt; like a client. It&#xA;never writes the store. &lt;code&gt;--key&lt;/code&gt; is the affinity key of a &lt;code&gt;sticky&lt;/code&gt; service.&lt;/p&gt;&#xA;&lt;p&gt;Every run is a new process, so a rotating &lt;code&gt;balance&lt;/code&gt; (&lt;code&gt;round-robin&lt;/code&gt;,&#xA;&lt;code&gt;weighted&lt;/code&gt;, &lt;code&gt;locality&lt;/code&gt;, &lt;code&gt;fastest&lt;/code&gt;, and &lt;code&gt;sticky&lt;/code&gt; without a key) gives one of&#xA;the candidates, starting at a random point: two runs can print different&#xA;backends. A script that needs the same answer every time uses a &lt;code&gt;failover&lt;/code&gt;&#xA;service, or a &lt;code&gt;sticky&lt;/code&gt; one with &lt;code&gt;--key&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;--explain&lt;/code&gt; says why: whether the store is stale, which tier the candidates&#xA;come from (the tiers of &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/&#34; &gt;chapter 5&lt;/a&gt;), every backend&#xA;that is out and the reason, and how &lt;code&gt;balance&lt;/code&gt; chooses.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-console&#34; data-lang=&#34;console&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor lookup api.svc.emfor --explain&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;service: api.svc.emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;answering: as a client library answers&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;store: /var/lib/emfor/state.mdb (heartbeat 0.4s ago, fresh)&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;balance: round-robin  on_all_down: open&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;tier: healthy primary backends whose last check passed&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;candidates:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;excluded:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.12:8080  last check failed&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.13:8080  backup held back&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.14:8080  unhealthy&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;selection: round-robin: the candidates in turn&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  the actual choice depends on the rotation start of the process, which is random in a new process&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;answer: 10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The reasons are &lt;code&gt;stale&lt;/code&gt; (a stale store makes no backend healthy or&#xA;&lt;code&gt;unknown&lt;/code&gt;), &lt;code&gt;unhealthy&lt;/code&gt;, &lt;code&gt;unknown&lt;/code&gt; (a better tier has candidates), &lt;code&gt;last check failed&lt;/code&gt;, &lt;code&gt;backup held back&lt;/code&gt;, &lt;code&gt;not in the pool&lt;/code&gt; (&lt;code&gt;fastest&lt;/code&gt;) and &lt;code&gt;not at this site&lt;/code&gt; (&lt;code&gt;locality&lt;/code&gt;). A candidate in slow start shows its step and effective&#xA;weight. &lt;code&gt;failover&lt;/code&gt; and &lt;code&gt;sticky&lt;/code&gt; with a key name the backend they choose; the&#xA;other methods list the candidates and the rule, because the choice depends&#xA;on the rotation of the process. &lt;code&gt;--explain&lt;/code&gt; does not advance any rotation.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;--json&lt;/code&gt; prints one JSON object (&lt;code&gt;service&lt;/code&gt;, &lt;code&gt;via&lt;/code&gt;, &lt;code&gt;family&lt;/code&gt;, &lt;code&gt;store&lt;/code&gt;,&#xA;&lt;code&gt;balance&lt;/code&gt;, &lt;code&gt;on_all_down&lt;/code&gt;, &lt;code&gt;result&lt;/code&gt;, and with &lt;code&gt;--explain&lt;/code&gt; also &lt;code&gt;explain&lt;/code&gt;);&#xA;later releases only add fields. The text is not a stable interface.&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;--dns A&lt;/code&gt; and &lt;code&gt;--dns AAAA&lt;/code&gt; (these imply &lt;code&gt;--explain&lt;/code&gt;) explain what the DNS&#xA;server of the daemon answers to an A or an AAAA query: only the backends of&#xA;that family are looked at, the store counts as fresh, &lt;code&gt;sticky&lt;/code&gt; has no key,&#xA;and &lt;code&gt;weighted&lt;/code&gt; is the daemon&amp;rsquo;s smooth weighted round-robin. It reads the&#xA;store, which gets a new &lt;code&gt;reason&lt;/code&gt; up to about one second after the daemon&#xA;decides it, so for that second it can differ from the live DNS answer; the&#xA;rotation counters are in the daemon&amp;rsquo;s memory and are not shown. SRV answers&#xA;are not explained.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Exit status&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;An address was printed (with &lt;code&gt;--dns&lt;/code&gt;: the service has candidates for the family)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No backend can be chosen: &lt;code&gt;ALL_DOWN&lt;/code&gt;, or no backend of the family with &lt;code&gt;--dns&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No such service, an invalid command line, or a config that cannot be read&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;3&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store cannot be used (&lt;code&gt;UNAVAILABLE&lt;/code&gt;) or has a format this command does not read (&lt;code&gt;INCOMPATIBLE&lt;/code&gt;), or the output cannot be written (a full disk; a closed pipe is not an error)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;code&gt;--explain&lt;/code&gt; does not change the exit status, and a failed lookup still prints&#xA;what is known.&lt;/p&gt;&#xA;&lt;h2 id=&#34;threads-processes-and-fork&#34;&gt;Threads, processes and fork&#xA;&lt;/h2&gt;&lt;ul&gt;&#xA;&lt;li&gt;Lookups are safe from many threads at once.&lt;/li&gt;&#xA;&lt;li&gt;The store is opened once per process and reused. A lookup holds an LMDB&#xA;reader slot only while it reads. The store has 1024 slots shared by all&#xA;processes on the host; a process killed in the middle of a lookup leaves&#xA;its slot taken until the daemon frees it, about once a minute.&lt;/li&gt;&#xA;&lt;li&gt;After &lt;code&gt;fork&lt;/code&gt;, the child opens the store again on its first lookup; this&#xA;works also when the parent looked up before forking (php-fpm, preforking&#xA;servers), and when other threads of the parent were looking up at the&#xA;moment of the fork.&lt;/li&gt;&#xA;&lt;li&gt;On FreeBSD, &lt;code&gt;fork&lt;/code&gt; waits until no lookup of the process is inside the&#xA;store, and keeps new lookups out until it has returned in the parent and&#xA;in the child; without that, a child forked while another thread looked&#xA;up could wait for ever in its first lookup, on a lock that FreeBSD&amp;rsquo;s&#xA;threading library leaves held in the child. The wait is as long as one&#xA;read, and lookups wait the same way while a fork runs. A thread stopped&#xA;(&lt;code&gt;SIGSTOP&lt;/code&gt;) inside a lookup holds the fork up until it continues.&#xA;&lt;code&gt;libemfor&lt;/code&gt; cannot be unloaded on FreeBSD (&lt;code&gt;dlclose&lt;/code&gt; leaves it mapped),&#xA;because its fork handlers stay registered until the process ends; link&#xA;&lt;code&gt;libemfor.a&lt;/code&gt; statically into a module that is unloaded, or depend on the&#xA;Rust crate from your own cdylib, only with &lt;code&gt;-Wl,-z,nodelete&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;What the guard does not cover on FreeBSD: a &lt;code&gt;fork&lt;/code&gt; from a signal handler&#xA;that interrupted a lookup of the same thread (it returns, but the child&#xA;can hang in its first lookup, as without the guard), and other users of&#xA;process-shared mutexes in the same process (another LMDB environment,&#xA;say), which can still leave a child waiting in its own lookup. Do not&#xA;fork from signal handlers; the library cannot fix the rest.&lt;/li&gt;&#xA;&lt;li&gt;Round-robin counters are per process.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;when-the-store-file-is-replaced-or-grows&#34;&gt;When the store file is replaced or grows&#xA;&lt;/h2&gt;&lt;p&gt;If the store file is replaced while applications run (the daemon moved a&#xA;damaged store aside, or someone deleted it and the daemon created a new&#xA;one), clients notice within about 100 ms and switch to the new file; no&#xA;application restart is needed. The same holds when the daemon is restarted&#xA;with a larger &lt;code&gt;[store] map_size&lt;/code&gt; and the store grows past the old size.&lt;/p&gt;&#xA;&lt;p&gt;A store file that exists but cannot be opened (another LMDB format, no&#xA;permission, not a store) is tried again at most once a second; lookups in&#xA;between fail with the same &lt;code&gt;UNAVAILABLE&lt;/code&gt; error. A missing store file is&#xA;tried on every lookup, so applications started before the daemon find the&#xA;store as soon as it exists.&lt;/p&gt;&#xA;&lt;p&gt;Do not delete the lock file (&lt;code&gt;&amp;lt;store&amp;gt;-lock&lt;/code&gt;) alone. Start the daemon&#xA;before applications that run as root or as the daemon&amp;rsquo;s user: if the lock&#xA;file is missing, the first process to open the store creates it, owned by&#xA;that process&amp;rsquo;s user, and the daemon may then fail to start.&lt;/p&gt;&#xA;&lt;h2 id=&#34;same-release-as-the-daemon&#34;&gt;Same release as the daemon&#xA;&lt;/h2&gt;&lt;p&gt;The client library contains the daemon&amp;rsquo;s own LMDB build, and the store&amp;rsquo;s&#xA;lock file format depends on it. Install &lt;code&gt;libemfor&lt;/code&gt; (and the wrappers) from&#xA;the same emfor release as the daemon, and upgrade them together. A&#xA;mismatch shows up as &lt;code&gt;UNAVAILABLE&lt;/code&gt; with a message that says so.&lt;/p&gt;&#xA;&lt;h3 id=&#34;store-format-version&#34;&gt;Store format version&#xA;&lt;/h3&gt;&lt;p&gt;The daemon writes store format version 2, which adds the backend field&#xA;&lt;code&gt;backup&lt;/code&gt;. Clients from this release read versions 1 and 2, so they work&#xA;with an older daemon too; older clients fail every lookup on a version 2&#xA;store with &lt;code&gt;INCOMPATIBLE&lt;/code&gt;. When upgrading, upgrade the clients (and&#xA;&lt;code&gt;emfor status&lt;/code&gt;) on a host before the daemon.&lt;/p&gt;&#xA;&lt;h2 id=&#34;performance&#34;&gt;Performance&#xA;&lt;/h2&gt;&lt;p&gt;A lookup takes about 1–1.5 µs on one thread. Throughput grows with threads&#xA;up to about four, then the shared reader lock of the store sets the pace&#xA;(about two million lookups a second in total on Linux). See&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/performance.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/performance.md&lt;/a&gt; for the measurements.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>7. DNS</title>
            <link>https://embark.systems/en/emfor/07-dns/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/07-dns/</guid>
            <description>&lt;p&gt;Programs that cannot use a client library (a third-party binary, a&#xA;language without a client, a &lt;code&gt;CGO_ENABLED=0&lt;/code&gt; Go build) can resolve service&#xA;names over DNS instead. emfor&amp;rsquo;s DNS server answers only its own zone, on a&#xA;loopback port; the host&amp;rsquo;s normal resolver forwards that zone to it.&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;program --&amp;gt; host resolver (dnsmasq, unbound, systemd-resolved) --&amp;gt; other names: upstream DNS&#xA;                     |&#xA;                     +-- *.svc.emfor --&amp;gt; emfor on 127.0.0.1:5300&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;emfor is never the host&amp;rsquo;s general DNS server, so if the daemon stops, only&#xA;names in its zone stop resolving.&lt;/p&gt;&#xA;&lt;h2 id=&#34;turning-it-on&#34;&gt;Turning it on&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;dns&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;listen&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;127.0.0.1:5300&amp;#34;&lt;/span&gt;]   &lt;span style=&#34;color:#75715e&#34;&gt;# default; not 5353, which is mDNS&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;zone&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;svc.emfor&amp;#34;&lt;/span&gt;            &lt;span style=&#34;color:#75715e&#34;&gt;# default&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;ttl&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;1s&amp;#34;&lt;/span&gt;                    &lt;span style=&#34;color:#75715e&#34;&gt;# default&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;rate_limit&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;100&lt;/span&gt;              &lt;span style=&#34;color:#75715e&#34;&gt;# default; UDP responses a second per client network&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;An empty &lt;code&gt;[dns]&lt;/code&gt; table uses the defaults. The server listens on UDP and&#xA;TCP. If an address cannot be bound, the daemon exits with status 1. At&#xA;most 128 TCP connections are served at once (more are closed right away),&#xA;and a TCP connection that sends no query for 5 seconds is closed. If a&#xA;listener stops while the daemon runs, the daemon exits with status 70 and&#xA;the service manager restarts it.&lt;/p&gt;&#xA;&lt;p&gt;Only services named under the zone are answered. At start and on reload&#xA;the daemon logs a warning for each service outside the zone, and for each&#xA;service whose backends listen on different ports (A and AAAA answers carry&#xA;no port; use SRV or a client library for such services).&lt;/p&gt;&#xA;&lt;h3 id=&#34;rate-limit&#34;&gt;Rate limit&#xA;&lt;/h3&gt;&lt;p&gt;An SRV answer can be many times the size of its query, so a DNS&#xA;server reachable from other hosts can be used to flood a third party&#xA;with forged-source UDP queries. The server therefore counts UDP&#xA;responses per client network (an IPv4 /24 or an IPv6 /56) and sends at&#xA;most &lt;code&gt;[dns] rate_limit&lt;/code&gt; (100 by default) of them a second to each. Past&#xA;the limit, every other response is an empty one with the TC flag, which&#xA;is no larger than the query (the same size when the query has no EDNS) and makes a real client retry over TCP; the&#xA;rest are not sent. Clients on a loopback address and TCP are never&#xA;limited, so the default setup (&lt;code&gt;listen&lt;/code&gt; on 127.0.0.1, queried by the&#xA;host&amp;rsquo;s own resolver) is not affected. &lt;code&gt;emfor_dns_rate_limited_total&lt;/code&gt;&#xA;counts the limited responses; if a resolver of your site that forwards&#xA;to emfor shows up there, raise &lt;code&gt;rate_limit&lt;/code&gt;, or set &lt;code&gt;0&lt;/code&gt; to turn the&#xA;limit off.&lt;/p&gt;&#xA;&lt;p&gt;At start the daemon logs a warning for each &lt;code&gt;listen&lt;/code&gt; address that is&#xA;not a loopback address: any host that can reach it can look up every&#xA;service. Limit who can reach that port with a firewall.&lt;/p&gt;&#xA;&lt;p&gt;Check it directly before you configure a resolver:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;dig @127.0.0.1 -p &lt;span style=&#34;color:#ae81ff&#34;&gt;5300&lt;/span&gt; api.svc.emfor A          &lt;span style=&#34;color:#75715e&#34;&gt;# Linux, macOS&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;drill -p &lt;span style=&#34;color:#ae81ff&#34;&gt;5300&lt;/span&gt; api.svc.emfor @127.0.0.1           &lt;span style=&#34;color:#75715e&#34;&gt;# FreeBSD&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;h2 id=&#34;answers&#34;&gt;Answers&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Query&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Answer&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A / AAAA for a service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One record: a backend of that family chosen by &lt;code&gt;balance&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#dns-differences&#34; &gt;chapter 5&lt;/a&gt;), with the &lt;code&gt;aa&lt;/code&gt; flag.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A / AAAA, no backend of that family&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NOERROR&lt;/code&gt; with no records (NODATA).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A / AAAA / SRV, no healthy backend and &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#when-no-backend-is-healthy&#34; &gt;&lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt;&lt;/a&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;. For A and AAAA &amp;ldquo;no healthy backend&amp;rdquo; is counted per address family.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A / AAAA / SRV, every backend of the service drained&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;, whatever &lt;code&gt;on_all_down&lt;/code&gt; is (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Any name in the zone, before the daemon has its first state (the first moments after a start)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;: which names exist is not known yet.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;SRV, a backend whose target name would be longer than 255 bytes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;SRV for a service, or &lt;code&gt;_&amp;lt;anything&amp;gt;._tcp.&amp;lt;service&amp;gt;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One SRV record per candidate, addresses in the additional section.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Other types for a service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NOERROR&lt;/code&gt; with no records.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A name under the zone that is no service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NXDOMAIN&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A name outside the zone, or a class other than &lt;code&gt;IN&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;REFUSED&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;EDNS version above 0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;BADVERS&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The TTL is the service&amp;rsquo;s &lt;code&gt;dns_ttl&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#servicesname&#34; &gt;chapter 3&lt;/a&gt;), or &lt;code&gt;[dns] ttl&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#dns&#34; &gt;chapter 3&lt;/a&gt;; 1 second by default).&#xA;The zone has a synthesized SOA whose negative-caching TTL is also&#xA;&lt;code&gt;[dns] ttl&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;srv-records&#34;&gt;SRV records&#xA;&lt;/h2&gt;&lt;p&gt;SRV answers carry each backend&amp;rsquo;s port and list every candidate, chosen&#xA;as for A and AAAA answers: the primaries whose last check passed, else the&#xA;&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#backup-backends&#34; &gt;backups&lt;/a&gt; whose last check&#xA;passed, else every healthy primary when all of them are failing (see &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#a-backend-whose-checks-start-failing&#34; &gt;A&#xA;backend whose checks start&#xA;failing&lt;/a&gt;),&#xA;else every healthy backup. An answer never lists primaries and backups&#xA;together while any backend is healthy. Or,&#xA;when none is healthy and &lt;code&gt;on_all_down&lt;/code&gt; is &lt;code&gt;open&lt;/code&gt;, the &lt;code&gt;unknown&lt;/code&gt; backends&#xA;(those whose last check passed first) before all of them (see &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#when-no-backend-is-healthy&#34; &gt;When no&#xA;backend is healthy&lt;/a&gt;).&#xA;IPv4 and IPv6 backends are listed together. Go&amp;rsquo;s&#xA;&lt;code&gt;net.LookupSRV(&amp;quot;http&amp;quot;, &amp;quot;tcp&amp;quot;, &amp;quot;api.svc.emfor&amp;quot;)&lt;/code&gt; asks for&#xA;&lt;code&gt;_http._tcp.api.svc.emfor&lt;/code&gt;, which emfor answers (other clients that build&#xA;such names were not tried).&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;code&gt;balance&lt;/code&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Priority&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Weight&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Order&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;failover&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0, 1, 2 … in config order&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;config order&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;weighted&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;the backend&amp;rsquo;s &lt;code&gt;weight&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;config order&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;locality&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0 in the host&amp;rsquo;s &lt;code&gt;site&lt;/code&gt;, 1 elsewhere&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;rotates per query&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fastest&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0 in the pool, 1 outside it&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;rotates per query&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;round-robin&lt;/code&gt;, &lt;code&gt;sticky&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;rotates per query&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;While a candidate is in &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#slow-start&#34; &gt;slow&#xA;start&lt;/a&gt;, every method but &lt;code&gt;failover&lt;/code&gt;&#xA;sets each weight to the backend&amp;rsquo;s reduced weight: its &lt;code&gt;weight&lt;/code&gt; (for&#xA;&lt;code&gt;weighted&lt;/code&gt;) or 1, times its step of 1 to 10.&lt;/p&gt;&#xA;&lt;p&gt;Each record&amp;rsquo;s target is a name derived from the backend&amp;rsquo;s address under&#xA;the service name, which emfor resolves whatever the backend&amp;rsquo;s health:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;10.0.0.11&lt;/code&gt; → &lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt;&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;2001:db8::11&lt;/code&gt; → &lt;code&gt;2001-0db8-0000-0000-0000-0000-0000-0011.api.svc.emfor&lt;/code&gt;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Over UDP a reply stays within the size the client advertises with EDNS, but&#xA;emfor caps that at 1232 bytes (a client that advertises 4096 gets 1232), or&#xA;512 bytes without EDNS. When the addresses do not fit,&#xA;they are left out; when even the SRV records do not fit, the reply is&#xA;truncated (TC) and the resolver retries over TCP. In the tests (IPv4&#xA;backends, a short service name) 19 backends fit in 1232 bytes with their&#xA;addresses and 27 without them; at 512 bytes it is 8 and 11. With 40&#xA;backends and a client that advertises 4096, the UDP reply held 27 records&#xA;with TC set, and over TCP all 40 arrived. What a given client does after TC&#xA;was tried for nginx and HAProxy only as far as the sections below say. A name close to the&#xA;255-byte limit can make even a single A record or the SOA too big for 512&#xA;bytes; that reply is truncated too.&lt;/p&gt;&#xA;&lt;h2 id=&#34;forwarding-from-the-host-resolver&#34;&gt;Forwarding from the host resolver&#xA;&lt;/h2&gt;&lt;p&gt;Pick the resolver your hosts use. These snippets use the defaults; adjust&#xA;the zone and port if you changed them.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;dnsmasq&lt;/strong&gt; (&lt;code&gt;/etc/dnsmasq.d/emfor.conf&lt;/code&gt;, then &lt;code&gt;systemctl restart dnsmasq&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;server=/svc.emfor/127.0.0.1#5300&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;unbound&lt;/strong&gt;, also FreeBSD&amp;rsquo;s &lt;code&gt;local_unbound&lt;/code&gt; (&lt;code&gt;/etc/unbound/unbound.conf.d/emfor.conf&lt;/code&gt;&#xA;or &lt;code&gt;/var/unbound/conf.d/emfor.conf&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;server:&#xA;    do-not-query-localhost: no&#xA;    domain-insecure: &amp;#34;svc.emfor&amp;#34;&#xA;&#xA;forward-zone:&#xA;    name: &amp;#34;svc.emfor&amp;#34;&#xA;    forward-addr: 127.0.0.1@5300&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Both &lt;code&gt;server:&lt;/code&gt; lines are required: unbound does not query localhost by&#xA;default, and a validating unbound would reject the unsigned zone.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;systemd-resolved&lt;/strong&gt; (&lt;code&gt;/etc/systemd/resolved.conf.d/emfor.conf&lt;/code&gt;, systemd 246&#xA;or later, then &lt;code&gt;systemctl restart systemd-resolved&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;[Resolve]&#xA;DNS=127.0.0.1:5300&#xA;Domains=~svc.emfor&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;strong&gt;macOS&lt;/strong&gt; (development only; file &lt;code&gt;/etc/resolver/svc.emfor&lt;/code&gt;):&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;nameserver 127.0.0.1&#xA;port 5300&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;Then test through the system resolver: &lt;code&gt;getent hosts api.svc.emfor&lt;/code&gt; on&#xA;Linux and FreeBSD, &lt;code&gt;dscacheutil -q host -a name api.svc.emfor&lt;/code&gt; on macOS.&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/dns-resolver.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/dns-resolver.md&lt;/a&gt; has the verified&#xA;recipes and more checks.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Recipes.&lt;/strong&gt; nginx, HAProxy, Caddy, Prometheus, pf and clients that ask for SRV records themselves are in &lt;a class=&#34;link&#34; href=&#34;../07-dns-recipes/&#34; &gt;chapter 7b&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;caching&#34;&gt;Caching&#xA;&lt;/h2&gt;&lt;p&gt;emfor&amp;rsquo;s short TTL lets a failed backend drop out of answers within about a&#xA;second. Caches can stretch that:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Leave the resolver&amp;rsquo;s minimum cache TTL at 0 (dnsmasq &lt;code&gt;min-cache-ttl&lt;/code&gt;,&#xA;unbound &lt;code&gt;cache-min-ttl&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;A resolver serves repeated queries from its cache for the TTL, so all&#xA;programs on the host get the same round-robin answer for that long. Lower&#xA;&lt;code&gt;dns_ttl&lt;/code&gt;, down to &lt;code&gt;0s&lt;/code&gt;, to rotate more often.&lt;/li&gt;&#xA;&lt;li&gt;Negative answers (a name not yet configured) are cached for &lt;code&gt;[dns] ttl&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Application runtimes may cache on their own (the JVM&amp;rsquo;s&#xA;&lt;code&gt;networkaddress.cache.ttl&lt;/code&gt;); emfor cannot shorten those.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;</description>
        </item><item>
            <title>7b. DNS recipes: proxies and other programs</title>
            <link>https://embark.systems/en/emfor/07-dns-recipes/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/07-dns-recipes/</guid>
            <description>&lt;p&gt;This chapter continues &lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;: what to configure in nginx, HAProxy, Caddy, Prometheus and pf so that they take their servers from emfor&amp;rsquo;s DNS answers, and what clients that ask for SRV records themselves do. Each section says what was tried and what was not.&lt;/p&gt;&#xA;&lt;h2 id=&#34;using-it-with-nginx&#34;&gt;Using it with nginx&#xA;&lt;/h2&gt;&lt;p&gt;Open-source nginx has no active health checks (only &lt;code&gt;max_fails&lt;/code&gt;), and its&#xA;upstream list is fixed until a reload. With nginx 1.27.3 or later it can&#xA;take the list from emfor&amp;rsquo;s SRV answers instead (&lt;code&gt;resolve&lt;/code&gt; and &lt;code&gt;service=&lt;/code&gt; were&#xA;commercial-only before that release), so a backend that fails its checks, a&#xA;backend you drain and a weight you change all reach nginx within a few&#xA;seconds (emfor&amp;rsquo;s own check interval first, then the one second of &lt;code&gt;valid&lt;/code&gt;),&#xA;without a reload:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-nginx&#34; data-lang=&#34;nginx&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;resolver&lt;/span&gt; 127.0.0.1:&lt;span style=&#34;color:#ae81ff&#34;&gt;5300&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;valid=1s&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;ipv6=off&lt;/span&gt;;    &lt;span style=&#34;color:#75715e&#34;&gt;# emfor&amp;#39;s DNS server, asked every second&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;resolver_timeout&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;1s&lt;/span&gt;;                          &lt;span style=&#34;color:#75715e&#34;&gt;# how soon to ask again after a failed lookup&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;upstream&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;api&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;zone&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;api&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;64k&lt;/span&gt;;                              &lt;span style=&#34;color:#75715e&#34;&gt;# required with `resolve`&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;server&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;api.svc.emfor&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;service=http&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;resolve&lt;/span&gt;; &lt;span style=&#34;color:#75715e&#34;&gt;# asks _http._tcp.api.svc.emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;server&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;listen&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;80&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;location&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;/&lt;/span&gt; { &lt;span style=&#34;color:#f92672&#34;&gt;proxy_pass&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;http://api&lt;/span&gt;; }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The &lt;code&gt;resolver&lt;/code&gt; points at emfor directly (&lt;code&gt;[dns] listen&lt;/code&gt;, port 5300 by&#xA;default), not at the host resolver: nginx has its own resolver cache, and&#xA;&lt;code&gt;valid=1s&lt;/code&gt; makes it ask again every second. &lt;code&gt;resolver_timeout 1s&lt;/code&gt; matters&#xA;after a lookup has failed, for example when emfor answers &lt;code&gt;SERVFAIL&lt;/code&gt;: nginx&#xA;asks again only after that time, and its default is 30 seconds (below). &lt;code&gt;service=http&lt;/code&gt; is only the name&#xA;prefix (&lt;code&gt;_http._tcp&lt;/code&gt;); emfor answers &lt;code&gt;_&amp;lt;anything&amp;gt;._tcp.&amp;lt;service&amp;gt;&lt;/code&gt;&#xA;(&lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;SRV records&lt;/a&gt;), so the port comes from each record.&lt;/p&gt;&#xA;&lt;p&gt;What nginx does with the answer, as checked with nginx 1.30.6 (stable) and&#xA;1.31.8 (mainline):&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Health&lt;/strong&gt;: a backend whose checks fail is not in the answer, and nginx&#xA;stops sending to it; when it recovers, it returns. The test waits up to&#xA;10 seconds for each change; with a 200 ms check interval it took about 3.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;drain = true&lt;/code&gt;&lt;/strong&gt; (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;):&#xA;edit emfor&amp;rsquo;s config and reload emfor; nginx is not touched, and the&#xA;backend stops getting new requests. A request that is running on the&#xA;backend finishes, and a connection nginx holds open to it stays open:&#xA;see &lt;a class=&#34;link&#34; href=&#34;../08-operations/#rolling-deploy-with-drain&#34; &gt;Rolling deploy with drain&lt;/a&gt;&#xA;(&lt;code&gt;tests/deploy&lt;/code&gt;) and the stream test below.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Priorities&lt;/strong&gt;: nginx takes the records with the lowest priority number as&#xA;its servers and the others as &lt;code&gt;backup&lt;/code&gt;. With &lt;code&gt;balance = &amp;quot;failover&amp;quot;&lt;/code&gt; the&#xA;first healthy backend is the only server and the second its backup;&#xA;&lt;code&gt;locality&lt;/code&gt; and &lt;code&gt;fastest&lt;/code&gt; also give priority 0 and 1 (SRV records table above), but only &lt;code&gt;failover&lt;/code&gt; was run against nginx.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weights&lt;/strong&gt; of &lt;code&gt;balance = &amp;quot;weighted&amp;quot;&lt;/code&gt; are followed: with weights 3 and 1,&#xA;60 of 80 requests went to the first backend. (nginx&amp;rsquo;s documentation does&#xA;not say it uses the SRV weight.)&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When emfor cannot answer&lt;/strong&gt;: a service with no backend to offer&#xA;(&lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt; and everything down, or every backend drained)&#xA;gets &lt;code&gt;SERVFAIL&lt;/code&gt;; nginx logs &lt;code&gt;could not be resolved&lt;/code&gt; and &lt;strong&gt;keeps the last&#xA;list it had&lt;/strong&gt;, so it goes on sending to those backends. A stopped emfor&#xA;sends no answer at all, so nginx&amp;rsquo;s lookup times out instead; that case&#xA;was not tried (the tests only drain every backend), and what nginx does&#xA;with a timeout may differ from what it does with &lt;code&gt;SERVFAIL&lt;/code&gt;. The lookup fails for nginx&amp;rsquo;s clients only through&#xA;nginx&amp;rsquo;s own &lt;code&gt;proxy_next_upstream&lt;/code&gt; and &lt;code&gt;max_fails&lt;/code&gt;. &lt;strong&gt;When emfor answers&#xA;again, nginx follows only after &lt;code&gt;resolver_timeout&lt;/code&gt;&lt;/strong&gt;, which is 30 seconds&#xA;by default: in the test, with the default, nginx still had its old list 5&#xA;seconds after emfor answered again and followed after about 26 seconds; with&#xA;&lt;code&gt;resolver_timeout 1s&lt;/code&gt; it followed after about a second (with 5 seconds&#xA;about 2, with 10 seconds about 7). A change that you make in this state,&#xA;a drain for example, reaches nginx just as late. That is why the recipe&#xA;sets &lt;code&gt;resolver_timeout 1s&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;HTTPS upstreams&lt;/strong&gt;: a record&amp;rsquo;s target is a name like&#xA;&lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt;. Set &lt;code&gt;proxy_ssl_name&lt;/code&gt; (and&#xA;&lt;code&gt;proxy_ssl_server_name on&lt;/code&gt;) to the name the backend&amp;rsquo;s certificate has. This&#xA;was not tried.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;TCP, with the &lt;code&gt;stream&lt;/code&gt; module.&lt;/strong&gt; The same &lt;code&gt;upstream&lt;/code&gt; works in a &lt;code&gt;stream&lt;/code&gt;&#xA;block, for a database or any other TCP service:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-nginx&#34; data-lang=&#34;nginx&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;stream&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;resolver&lt;/span&gt; 127.0.0.1:&lt;span style=&#34;color:#ae81ff&#34;&gt;5300&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;valid=1s&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;ipv6=off&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;resolver_timeout&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;1s&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;upstream&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;db&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;zone&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;db&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;64k&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;server&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;db.svc.emfor&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;service=pgsql&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;resolve&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;server&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;listen&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;5432&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;proxy_pass&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;db&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;tests/nginx/stream.sh&lt;/code&gt; checked it (with &lt;code&gt;api.svc.emfor&lt;/code&gt; and &lt;code&gt;service=http&lt;/code&gt;, not&#xA;the names above) with two backends answering HTTP, which a TCP proxy passes&#xA;on, with nginx 1.30.6 and 1.31.8. Health, &lt;code&gt;drain&lt;/code&gt;,&#xA;&lt;code&gt;failover&lt;/code&gt; (the second server is a backup) and the weights behave as for&#xA;&lt;code&gt;http&lt;/code&gt;, and so does &lt;code&gt;SERVFAIL&lt;/code&gt;, including the &lt;code&gt;resolver_timeout&lt;/code&gt; after it. What&#xA;is particular to TCP:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;A connection that is already open is not touched.&lt;/strong&gt; When its backend was&#xA;drained, a 6 second answer on a connection to it still arrived whole (1200&#xA;of 1200 bytes) while new connections went to the other backend. When the&#xA;backend was stopped instead, the same connection was cut (160 of 1200&#xA;bytes). So after a drain, long connections to the backend end when the client&#xA;or the backend closes them; nothing moves them.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A new connection to a stopped backend that emfor still lists goes to the&#xA;next server&lt;/strong&gt;: &lt;code&gt;proxy_next_upstream&lt;/code&gt; is on by default in &lt;code&gt;stream&lt;/code&gt;, and 0 of&#xA;25 connections made while a backend was stopped failed.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Not tried: UDP (&lt;code&gt;listen ... udp&lt;/code&gt;), &lt;code&gt;ssl_preread&lt;/code&gt;, &lt;code&gt;proxy_protocol&lt;/code&gt;,&#xA;&lt;code&gt;proxy_timeout&lt;/code&gt; (an idle connection is closed after its default of ten&#xA;minutes, as nginx documents it), nginx Plus.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Older nginx.&lt;/strong&gt; Before 1.27.3 the &lt;code&gt;service=&lt;/code&gt; parameter is rejected at&#xA;&lt;code&gt;nginx -t&lt;/code&gt; (&lt;code&gt;invalid parameter &amp;quot;service=http&amp;quot;&lt;/code&gt;, seen with 1.26.3). Without&#xA;it, use a variable in &lt;code&gt;proxy_pass&lt;/code&gt;, which makes nginx resolve the name at run&#xA;time, and give every backend of the service the same port:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-nginx&#34; data-lang=&#34;nginx&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;resolver&lt;/span&gt; 127.0.0.1:&lt;span style=&#34;color:#ae81ff&#34;&gt;5300&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;valid=1s&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;ipv6=off&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;server&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;listen&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;80&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;location&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;/&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;set&lt;/span&gt; $backend &lt;span style=&#34;color:#e6db74&#34;&gt;api.svc.emfor&lt;/span&gt;;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;proxy_pass&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;http://&lt;/span&gt;$backend:8080;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;An A answer holds one backend, chosen by &lt;code&gt;balance&lt;/code&gt;, so nginx sends all&#xA;requests of a second to that one (it keeps the answer for &lt;code&gt;valid&lt;/code&gt;); the&#xA;rotation is across seconds, not across requests. Both backends were reached&#xA;in the test; how evenly was not measured.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/tests/nginx/run.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;tests/nginx/&lt;/a&gt; runs this on a given nginx image&#xA;(&lt;code&gt;tests/nginx/run.sh nginx:stable&lt;/code&gt;): the checks above, and the older nginx&#xA;case. The Integrations workflow runs it every week and when the DNS or&#xA;candidate code of emfor changes.&lt;/p&gt;&#xA;&lt;p&gt;On FreeBSD 15.1 (arm64, nginx 1.30.6 from the packages) the same checks&#xA;passed, including the older nginx case. The recipes need no change; the&#xA;nginx package reads &lt;code&gt;/usr/local/etc/nginx/nginx.conf&lt;/code&gt;, not &lt;code&gt;/etc/nginx&lt;/code&gt;.&#xA;The Docker test itself does not run there; the script&#xA;(&lt;code&gt;tests/nginx/inside.sh&lt;/code&gt;) was run by hand, and it needed three things the&#xA;Linux container has: &lt;code&gt;dig&lt;/code&gt; (a wrapper around &lt;code&gt;drill&lt;/code&gt; did), the address&#xA;&lt;code&gt;127.0.0.2&lt;/code&gt; on the loopback (&lt;code&gt;ifconfig lo0 alias 127.0.0.2/32&lt;/code&gt;; FreeBSD&amp;rsquo;s&#xA;loopback has only &lt;code&gt;127.0.0.1&lt;/code&gt;, and the older-nginx check binds a backend&#xA;to &lt;code&gt;127.0.0.2&lt;/code&gt;), and &lt;code&gt;nginx&lt;/code&gt; reading &lt;code&gt;/etc/nginx/nginx.conf&lt;/code&gt;. The&#xA;&lt;code&gt;stream&lt;/code&gt; checks (&lt;code&gt;tests/nginx/stream.sh&lt;/code&gt;) passed there too, by hand, with&#xA;one more line: the FreeBSD package builds &lt;code&gt;stream&lt;/code&gt; as a dynamic module, so&#xA;&lt;code&gt;nginx.conf&lt;/code&gt; needs &lt;code&gt;load_module /usr/local/libexec/nginx/ngx_stream_module.so;&lt;/code&gt;&#xA;at its top (without it &lt;code&gt;nginx -t&lt;/code&gt; says &lt;code&gt;unknown directive &amp;quot;stream&amp;quot;&lt;/code&gt;). The&#xA;delay after a &lt;code&gt;SERVFAIL&lt;/code&gt; was the same there: about 25 seconds with the&#xA;default &lt;code&gt;resolver_timeout&lt;/code&gt;, and about a second with &lt;code&gt;resolver_timeout 1s&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;using-it-with-haproxy&#34;&gt;Using it with HAProxy&#xA;&lt;/h2&gt;&lt;p&gt;HAProxy can fill the servers of a backend from SRV answers: a&#xA;&lt;code&gt;server-template&lt;/code&gt; names the SRV query and a number of slots, and HAProxy&#xA;puts the records into the slots, asking again when the answer changes. With&#xA;emfor&amp;rsquo;s DNS server as its resolver, a backend that fails emfor&amp;rsquo;s checks, one&#xA;you drain and one you add all reach HAProxy within a few seconds, without a&#xA;reload:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;resolvers emfor&#xA;    nameserver emfor 127.0.0.1:5300&#xA;    accepted_payload_size 4096&#xA;    hold valid 1s&#xA;    resolve_retries 3&#xA;    timeout resolve 1s&#xA;    timeout retry 1s&#xA;&#xA;backend api&#xA;    balance roundrobin&#xA;    server-template s 1-4 _http._tcp.api.svc.emfor resolvers emfor resolve-opts allow-dup-ip resolve-prefer ipv4 init-addr none&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;nameserver&lt;/code&gt; is emfor&amp;rsquo;s DNS server (&lt;code&gt;[dns] listen&lt;/code&gt;, port 5300 by default),&#xA;not the host resolver.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;accepted_payload_size 4096&lt;/code&gt; tells emfor that HAProxy takes a larger&#xA;answer over UDP than the 512 bytes it asks for by default; emfor caps it&#xA;at 1232 bytes (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;SRV records&lt;/a&gt;: 19 backends with their&#xA;addresses, 27 without). More than 27 backends with HAProxy was not tried,&#xA;nor whether HAProxy then retries over TCP.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;hold valid 1s&lt;/code&gt; makes HAProxy use a new answer after a second; the other&#xA;&lt;code&gt;hold&lt;/code&gt; values stay at their defaults, which matter below.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;1-4&lt;/code&gt; is the number of &lt;strong&gt;slots&lt;/strong&gt;: the most backends HAProxy can use. A&#xA;service with more backends than slots is cut off (the test: three backends&#xA;in four slots use three). Make it larger than the service.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;_http._tcp.api.svc.emfor&lt;/code&gt; is the SRV name; emfor answers&#xA;&lt;code&gt;_&amp;lt;anything&amp;gt;._tcp.&amp;lt;service&amp;gt;&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;SRV records&lt;/a&gt;), and the port&#xA;comes from each record. &lt;code&gt;allow-dup-ip&lt;/code&gt; is needed when several backends&#xA;share one address with different ports, and &lt;code&gt;init-addr none&lt;/code&gt; lets HAProxy&#xA;start before the first answer.&lt;/li&gt;&#xA;&lt;li&gt;No &lt;code&gt;check&lt;/code&gt; is needed: a backend that fails emfor&amp;rsquo;s checks leaves the&#xA;answer. You can add HAProxy&amp;rsquo;s own &lt;code&gt;check&lt;/code&gt; as a second opinion.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;What HAProxy does with the answer, as checked with HAProxy 3.4.7 and 2.8.31:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Health and &lt;code&gt;drain = true&lt;/code&gt;&lt;/strong&gt; (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter&#xA;5&lt;/a&gt;) reach HAProxy within a few&#xA;seconds, as for nginx: the backend gets nothing, and gets requests again&#xA;when it recovers or the drain is removed. HAProxy is not reloaded.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Priorities are not used.&lt;/strong&gt; With &lt;code&gt;balance = &amp;quot;failover&amp;quot;&lt;/code&gt;, HAProxy sent&#xA;requests to both backends (20 and 20 of 40), not only to the first one.&#xA;&lt;code&gt;locality&lt;/code&gt; and &lt;code&gt;fastest&lt;/code&gt; rank backends by priority 0 and 1 too, so they&#xA;should behave the same; that was not run. Do not rely on a backup in the SRV answer; use&#xA;HAProxy&amp;rsquo;s own &lt;code&gt;backup&lt;/code&gt; for that, or the nginx recipe above, which follows&#xA;priorities (tried with &lt;code&gt;failover&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weights are not used.&lt;/strong&gt; With &lt;code&gt;balance = &amp;quot;weighted&amp;quot;&lt;/code&gt; and weights 3 and 1,&#xA;the requests went 40 and 40 of 80, also with &lt;code&gt;weight 1&lt;/code&gt; or&#xA;&lt;code&gt;ignore-weight&lt;/code&gt; on the template.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When emfor does not answer&lt;/strong&gt; (stopped, or restarting): HAProxy keeps its&#xA;servers for the &lt;code&gt;hold&lt;/code&gt; times. With the settings above it still served all&#xA;requests 4 seconds after emfor stopped.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When emfor answers &lt;code&gt;SERVFAIL&lt;/code&gt;&lt;/strong&gt; (no backend to offer, or every backend&#xA;drained): HAProxy kept its servers for a while and then dropped them, and&#xA;every request got &lt;code&gt;503&lt;/code&gt;; in the test it had lost them between 10 and 30&#xA;seconds later, which is the default &lt;code&gt;hold other&lt;/code&gt; (30 s). nginx, in&#xA;the same case, kept its last list for the few seconds the test&#xA;looked; how long it keeps it was not tried.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Which is right depends on what you want to happen when emfor says there is&#xA;no backend. The &lt;code&gt;hold&lt;/code&gt; values are HAProxy&amp;rsquo;s, and were not tuned here.&lt;/p&gt;&#xA;&lt;p&gt;Three variations, as &lt;code&gt;tests/haproxy/more.sh&lt;/code&gt; saw them with HAProxy 3.4.7 and&#xA;2.8.31:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;mode tcp&lt;/code&gt;&lt;/strong&gt; (in &lt;code&gt;defaults&lt;/code&gt; or in the frontend and the backend) changes&#xA;nothing: the servers come from the same answers, a backend that fails its&#xA;checks or is drained gets no connections, and it gets them again when it&#xA;recovers.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;TLS to the backends.&lt;/strong&gt; Add &lt;code&gt;ssl&lt;/code&gt; to the template. With &lt;code&gt;ssl verify none&lt;/code&gt;&#xA;it works. To check the certificate, &lt;code&gt;ssl verify required ca-file &amp;lt;ca&amp;gt; sni str(&amp;lt;name&amp;gt;)&lt;/code&gt; checks it against &lt;code&gt;&amp;lt;name&amp;gt;&lt;/code&gt;: with the right name both&#xA;backends were served, with a name that is not in the certificate nothing&#xA;was. Always set &lt;code&gt;sni&lt;/code&gt;: the names of the SRV targets&#xA;(&lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt;) are not in a certificate, and without &lt;code&gt;sni&lt;/code&gt;&#xA;3.4.7 refused the certificate while 2.8.31 accepted any certificate that&#xA;the CA had signed, whatever its name. Do not rely on &lt;code&gt;verifyhost&lt;/code&gt;: a wrong&#xA;one next to a right &lt;code&gt;sni&lt;/code&gt; was served by both, and a right one without&#xA;&lt;code&gt;sni&lt;/code&gt; was served by 2.8.31 and refused by 3.4.7.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;One &lt;code&gt;nameserver&lt;/code&gt;.&lt;/strong&gt; Use one &lt;code&gt;nameserver&lt;/code&gt; line, the emfor server on the&#xA;same host as HAProxy. HAProxy sends every query to every nameserver and&#xA;uses the first reply that arrives; it does not merge them (in the test, with&#xA;two emfor servers that gave the same answer, HAProxy kept serving with&#xA;either one stopped, and a backend that went down still left the list). When&#xA;the two answers differ (a backend drained on one only), the list changes&#xA;from query to query: HAProxy removes a backend and adds it again, often in&#xA;another slot, and what it serves differs from run to run with the same&#xA;configuration (many runs on Linux with 3.4.7 and 2.8.31, and 6 on FreeBSD&#xA;with 3.4.4). Requests can also fail: after such a change a slot was ready&#xA;with the right address and port 0, and a request sent to it got a &lt;code&gt;503&lt;/code&gt;&#xA;after about 3 seconds (a backend stopped and started every 1.5 seconds:&#xA;with two nameservers 5 of 5 runs on Linux 3.4.7, and with one nameserver&#xA;at most one such request per run; but on FreeBSD the same test gave them&#xA;with one nameserver too: 29 to 45 of 80 requests in each of 5 runs with&#xA;3.4.4 (the package), and 21 to 28 with 3.4.7 (built from the port), so it is&#xA;not a change between those releases, and the second nameserver is not the&#xA;only cause; why FreeBSD shows it and the Linux container does not was not&#xA;found). These were seen by hand, not in the test, and&#xA;the exact cause inside HAProxy was not found. For availability run an emfor server next to each HAProxy&#xA;rather than giving one HAProxy several nameservers. If you give it two&#xA;anyway, they must agree, &lt;code&gt;drain&lt;/code&gt; included; &lt;code&gt;option redispatch&lt;/code&gt; hides the&#xA;failed requests but does not repair the slot. &lt;code&gt;hold obsolete 30s&lt;/code&gt; makes&#xA;the result stable by keeping the union of the answers, so a drain or a&#xA;removal takes that long to count: not recommended.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/tests/haproxy/run.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;tests/haproxy/&lt;/a&gt; runs these checks on a given&#xA;HAProxy image (&lt;code&gt;tests/haproxy/run.sh haproxy:2.8&lt;/code&gt;), including the two things&#xA;HAProxy does not do, so that the test fails if a later HAProxy starts to&#xA;follow priorities or weights and this section must change. The Integrations&#xA;workflow runs it every week and when the DNS or candidate code of emfor&#xA;changes. Not tried: the TLS checks of emfor itself next to HAProxy&amp;rsquo;s, client&#xA;certificates to the backends, &lt;code&gt;resolvers&lt;/code&gt; with more than two name servers,&#xA;other versions than the two above.&lt;/p&gt;&#xA;&lt;p&gt;On FreeBSD 15.1 (arm64, HAProxy 3.4.4 from the packages) the same checks&#xA;passed, priorities and weights not followed included, so the section holds&#xA;there as it is. As with nginx, &lt;code&gt;tests/haproxy/inside.sh&lt;/code&gt; was run by hand;&#xA;it needed nothing else than the HAProxy package. &lt;code&gt;tests/haproxy/more.sh&lt;/code&gt;&#xA;(&lt;code&gt;mode tcp&lt;/code&gt;, TLS to the backends, two name servers) passed there too, by&#xA;hand, with the same results as with 3.4.7, the &lt;code&gt;sni&lt;/code&gt; ones included. Not tried&#xA;on FreeBSD: other HAProxy versions.&lt;/p&gt;&#xA;&lt;h2 id=&#34;using-it-with-caddy&#34;&gt;Using it with Caddy&#xA;&lt;/h2&gt;&lt;p&gt;Caddy&amp;rsquo;s &lt;code&gt;reverse_proxy&lt;/code&gt; can take its upstreams from SRV answers with&#xA;&lt;code&gt;dynamic srv&lt;/code&gt;, so health and drain reach Caddy within a few seconds without a&#xA;reload:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;:80 {&#xA;    reverse_proxy {&#xA;        dynamic srv {&#xA;            name _http._tcp.api.svc.emfor&#xA;            refresh 1s&#xA;            resolvers 127.0.0.1:5300&#xA;            grace_period 30s&#xA;        }&#xA;        transport http {&#xA;            resolvers 127.0.0.1:5300&#xA;        }&#xA;        lb_try_duration 2s&#xA;        lb_try_interval 100ms&#xA;    }&#xA;}&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Two &lt;code&gt;resolvers&lt;/code&gt;.&lt;/strong&gt; The one in &lt;code&gt;dynamic srv&lt;/code&gt; is for the SRV query. The&#xA;targets of the answer are names such as &lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt;, which&#xA;Caddy dials, and the one in &lt;code&gt;transport http&lt;/code&gt; is for that. Without it Caddy&#xA;asks the host resolver, and when that does not know the zone every request&#xA;fails (10 of 10 in the test, &lt;code&gt;no such host&lt;/code&gt; in the log); with a host&#xA;resolver that &lt;a class=&#34;link&#34; href=&#34;../07-dns/#forwarding-from-the-host-resolver&#34; &gt;forwards the&#xA;zone&lt;/a&gt; it is not needed.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;refresh&lt;/code&gt;&lt;/strong&gt; is how often Caddy asks again; the test used 1 second. The&#xA;default is longer (a minute, as I read the Caddy documentation; not&#xA;tried), and a longer &lt;code&gt;refresh&lt;/code&gt; delays a drain or a failure that long.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;grace_period&lt;/code&gt;&lt;/strong&gt; keeps the last list when the lookup fails. emfor&#xA;answers &lt;code&gt;SERVFAIL&lt;/code&gt; when every backend of the service is drained, or is down&#xA;with &lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt; (with the default &lt;code&gt;open&lt;/code&gt; it still answers):&#xA;with &lt;code&gt;grace_period&lt;/code&gt; Caddy kept answering (10 of 10), without it Caddy had no&#xA;upstream and every request failed (10 of 10). nginx kept its last list&#xA;without such a setting in its test.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;&lt;code&gt;lb_try_duration&lt;/code&gt; and &lt;code&gt;lb_try_interval&lt;/code&gt;&lt;/strong&gt; make Caddy try again, so a&#xA;backend that has just stopped, and is still in the list for the second or&#xA;so that emfor and Caddy need to notice it, loses nothing (0 of 25 requests&#xA;in the test). Without them Caddy does not retry, and 3 of 25 were lost.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;What Caddy does with the answer, as &lt;code&gt;tests/caddy/run.sh&lt;/code&gt; saw it with Caddy&#xA;2.11.9:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A backend that fails its checks leaves the list, and one that recovers&#xA;returns; a &lt;code&gt;drain = true&lt;/code&gt; backend leaves it too, without a reload of&#xA;Caddy and with no request lost (with the retries above).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weights are not followed&lt;/strong&gt;: a 3 to 1 service gave about half each (41 and 39 of 80 in one run).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Priorities are not followed by the default policy&lt;/strong&gt;: a &lt;code&gt;failover&lt;/code&gt; service&#xA;sent requests to the backup as well. With &lt;code&gt;lb_policy first&lt;/code&gt; Caddy takes the&#xA;backends in the order of the answer, which is sorted by priority, so&#xA;&lt;code&gt;balance = &amp;quot;failover&amp;quot;&lt;/code&gt; with &lt;code&gt;lb_policy first&lt;/code&gt; sends everything to the&#xA;first backend while it is healthy, to the second when it is not, and back&#xA;when it returns.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/tests/caddy/run.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;tests/caddy/&lt;/a&gt; runs these checks on a given Caddy&#xA;image (&lt;code&gt;tests/caddy/run.sh caddy:2&lt;/code&gt;); the Integrations workflow runs it every&#xA;week and when the DNS or candidate code of emfor changes. Not tried: HTTPS&#xA;upstreams, Caddy&amp;rsquo;s own active health checks next to emfor&amp;rsquo;s, load balancing&#xA;policies other than &lt;code&gt;first&lt;/code&gt; and the default, several Caddy instances, other&#xA;versions than 2.11.9, FreeBSD.&lt;/p&gt;&#xA;&lt;h2 id=&#34;using-it-with-prometheus&#34;&gt;Using it with Prometheus&#xA;&lt;/h2&gt;&lt;p&gt;Prometheus can find its scrape targets from SRV answers with&#xA;&lt;code&gt;dns_sd_configs&lt;/code&gt; of type &lt;code&gt;SRV&lt;/code&gt;: the targets are the backends of an emfor&#xA;service that are healthy, and the list follows the checks without a reload&#xA;of Prometheus.&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-yaml&#34; data-lang=&#34;yaml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#f92672&#34;&gt;scrape_configs&lt;/span&gt;:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  - &lt;span style=&#34;color:#f92672&#34;&gt;job_name&lt;/span&gt;: &lt;span style=&#34;color:#ae81ff&#34;&gt;api&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#f92672&#34;&gt;dns_sd_configs&lt;/span&gt;:&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;      - &lt;span style=&#34;color:#f92672&#34;&gt;names&lt;/span&gt;: [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;_http._tcp.api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;type&lt;/span&gt;: &lt;span style=&#34;color:#ae81ff&#34;&gt;SRV&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;        &lt;span style=&#34;color:#f92672&#34;&gt;refresh_interval&lt;/span&gt;: &lt;span style=&#34;color:#ae81ff&#34;&gt;5s&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;dns_sd_configs&lt;/code&gt; asks the host&amp;rsquo;s resolver and has no setting for a name&#xA;server or a port (&lt;code&gt;promtool check config&lt;/code&gt; rejects a &lt;code&gt;server&lt;/code&gt; field), so&#xA;the host resolver must forward the zone to emfor, as in &lt;a class=&#34;link&#34; href=&#34;../07-dns/#forwarding-from-the-host-resolver&#34; &gt;Forwarding from the&#xA;host resolver&lt;/a&gt;. It must forward the whole&#xA;zone, not only the SRV query: the targets of the answer are names such as&#xA;&lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt;, which Prometheus resolves again for each scrape&#xA;and which emfor answers. The &lt;code&gt;instance&lt;/code&gt; label is that name and the port, not&#xA;the address; &lt;code&gt;__meta_dns_srv_record_target&lt;/code&gt; and &lt;code&gt;__meta_dns_srv_record_port&lt;/code&gt;&#xA;are there for &lt;code&gt;relabel_configs&lt;/code&gt;. The port of an SRV record is the port of the&#xA;backend in emfor, so the application&amp;rsquo;s metrics have to be on it; for another&#xA;port, make a second service whose backends are that port.&lt;/p&gt;&#xA;&lt;p&gt;What the targets are, as &lt;code&gt;tests/prometheus/run.sh&lt;/code&gt; saw it with Prometheus&#xA;3.16.0 (&lt;code&gt;refresh_interval&lt;/code&gt; 1 second, emfor as the only resolver of the&#xA;container):&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;Both backends of a &lt;code&gt;round-robin&lt;/code&gt; service are targets and &lt;code&gt;up&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;A backend that fails its checks &lt;strong&gt;is not a target any more&lt;/strong&gt;: it is not&#xA;shown as down, and its &lt;code&gt;up&lt;/code&gt; series ends. So &lt;code&gt;up == 0&lt;/code&gt; never fires for it;&#xA;alert on emfor&amp;rsquo;s own metrics instead (&lt;code&gt;emfor_backend_state&lt;/code&gt;, see&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/prometheus/emfor-rules.yml&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/prometheus/&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;A drained backend is not a target either, though it runs and answers on&#xA;&lt;code&gt;/metrics&lt;/code&gt;: during a &lt;a class=&#34;link&#34; href=&#34;../08-operations/#rolling-deploy-with-drain&#34; &gt;deploy with drain&lt;/a&gt;&#xA;its metrics are not collected.&lt;/li&gt;&#xA;&lt;li&gt;A &lt;code&gt;failover&lt;/code&gt; service lists its backup too (priority 1), and Prometheus&#xA;takes every record, so it scrapes both. A backend with &lt;code&gt;backup = true&lt;/code&gt;&#xA;is a target only while no primary is healthy.&lt;/li&gt;&#xA;&lt;li&gt;A target appears and disappears within about &lt;code&gt;refresh_interval&lt;/code&gt; plus the&#xA;time emfor needs to see the change (&lt;code&gt;interval&lt;/code&gt;, &lt;code&gt;fall&lt;/code&gt;, &lt;code&gt;rise&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;So an emfor service is a good source for &amp;ldquo;the backends that are serving&#xA;now&amp;rdquo;. To scrape every backend whatever its health, and to see &lt;code&gt;up == 0&lt;/code&gt;,&#xA;keep a static list or file-based discovery for the scrape and use emfor for&#xA;the alerts.&lt;/p&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/tests/prometheus/run.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;tests/prometheus/&lt;/a&gt; runs these checks (the&#xA;Integrations workflow runs it every week and when the DNS or candidate&#xA;code of emfor changes). On FreeBSD 15.1 the same script passed by hand with&#xA;the Prometheus 2.55.1 package (nginx 1.30.6 from the packages; &lt;code&gt;local_unbound&lt;/code&gt;&#xA;stopped, emfor on port 53). Not tried: dnsmasq or unbound in front of emfor&#xA;(the tests used emfor as the only resolver), other Prometheus versions, HTTPS&#xA;targets, many targets, the default &lt;code&gt;refresh_interval&lt;/code&gt; of 30 seconds.&lt;/p&gt;&#xA;&lt;h2 id=&#34;using-it-with-pf-freebsd&#34;&gt;Using it with pf (FreeBSD)&#xA;&lt;/h2&gt;&lt;p&gt;On FreeBSD, pf can spread connections over a table of addresses without a&#xA;proxy in the data path: an &lt;code&gt;rdr&lt;/code&gt; rule redirects the connections that come to&#xA;an address to the members of a table, in turn. emfor has no hook that runs a&#xA;command when its answer changes, so a small loop asks emfor&amp;rsquo;s DNS server for&#xA;the SRV records of the service every second and makes the table equal to what&#xA;it offers:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;table &amp;lt;api&amp;gt; persist&#xA;rdr pass on $ext_if inet proto tcp from any to $vip port 80 -&amp;gt; &amp;lt;api&amp;gt; port 8000 round-robin&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/pf/emfor-pf-table.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/pf/emfor-pf-table.sh&lt;/a&gt; is that&#xA;loop (&lt;code&gt;emfor-pf-table.sh loop api.svc.emfor api&lt;/code&gt;, as root; for example under&#xA;&lt;code&gt;daemon(8)&lt;/code&gt;, not tried). It takes the SRV records with the lowest priority,&#xA;reads the address out of each target name (&lt;code&gt;10-0-0-11.api.svc.emfor&lt;/code&gt; is&#xA;&lt;code&gt;10.0.0.11&lt;/code&gt;) and runs &lt;code&gt;pfctl -t api -T replace&lt;/code&gt; when the set changed. It needs&#xA;&lt;code&gt;drill&lt;/code&gt;, which the base system has. All backends of the service must have the&#xA;same port, the one in the &lt;code&gt;rdr&lt;/code&gt; rule.&lt;/p&gt;&#xA;&lt;p&gt;What happens, as &lt;code&gt;tests/pf/run.sh&lt;/code&gt; saw it on FreeBSD 15.1, with the backends&#xA;and the redirected address on the loopback:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Health&lt;/strong&gt;: a backend that fails its checks leaves the table, and returns&#xA;when it recovers. &lt;strong&gt;&lt;code&gt;drain = true&lt;/code&gt;&lt;/strong&gt; removes it the same way&#xA;(&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;A connection that is already open is not cut&lt;/strong&gt; when its backend leaves the&#xA;table, because pf keeps its state: a 6 second answer on a connection to a&#xA;drained backend arrived whole (1200 of 1200 bytes), while new connections&#xA;went to the other backend.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Priorities&lt;/strong&gt;: only the lowest priority is in the table, so a &lt;code&gt;failover&lt;/code&gt;&#xA;service has its first backend there, and the second when the first is&#xA;down. A &lt;code&gt;backup = true&lt;/code&gt; backend is out while a primary is healthy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;Weights are not used&lt;/strong&gt;: pf&amp;rsquo;s &lt;code&gt;round-robin&lt;/code&gt; shares equally (40 and 40 of 80&#xA;for weights 3 and 1).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When emfor has nothing to offer&lt;/strong&gt; (&lt;code&gt;SERVFAIL&lt;/code&gt;: every backend drained, or&#xA;down with &lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt;) the table is emptied, and pf drops a&#xA;connection that an empty &lt;code&gt;rdr&lt;/code&gt; table cannot redirect: the client waits for&#xA;its own timeout (seen from the loopback and from another host). Set &lt;code&gt;EMPTY&lt;/code&gt;&#xA;to an address on the host where nothing listens on the &lt;code&gt;rdr&lt;/code&gt; port, and the&#xA;table holds that instead of nothing; a client on another host was refused at&#xA;once by a table entry where nothing listened (&lt;code&gt;rc=7&lt;/code&gt; in a second). The&#xA;script&amp;rsquo;s &lt;code&gt;EMPTY&lt;/code&gt; itself was only checked for what it puts in the table.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;When emfor does not answer&lt;/strong&gt; (stopped, not reachable) the table is left as&#xA;it is, and follows emfor again when it is back.&lt;/li&gt;&#xA;&lt;li&gt;Between the moment a backend fails and the next pass of the loop (a second,&#xA;after emfor&amp;rsquo;s own checks), pf still sends new connections to it: from another&#xA;host a connection to a backend where nothing listened was refused, so the&#xA;client should retry.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Not tried: IPv6 (the script reads IPv6 targets but they were not run),&#xA;&lt;code&gt;source-hash&lt;/code&gt; and &lt;code&gt;sticky-address&lt;/code&gt;, &lt;code&gt;rdr&lt;/code&gt; on a real interface with many&#xA;clients (the checks of other hosts were made by hand from one machine: refused&#xA;and dropped connections), &lt;code&gt;daemon(8)&lt;/code&gt; or an rc script for the loop, other&#xA;firewalls. &lt;code&gt;tests/pf/run.sh&lt;/code&gt; needs root and a FreeBSD with the pf module; it&#xA;loads its own rules and enables pf, so it refuses to run when pf is already&#xA;on. It is not part of the CI.&lt;/p&gt;&#xA;&lt;h2 id=&#34;clients-that-ask-for-srv-records-themselves&#34;&gt;Clients that ask for SRV records themselves&#xA;&lt;/h2&gt;&lt;p&gt;Some clients find their servers with SRV records on their own, so no proxy is&#xA;needed: they ask &lt;code&gt;_&amp;lt;service&amp;gt;._tcp.&amp;lt;name&amp;gt;&lt;/code&gt;, and emfor answers&#xA;&lt;code&gt;_&amp;lt;anything&amp;gt;._tcp.&amp;lt;service&amp;gt;&lt;/code&gt; with the healthy backends, their ports, and the&#xA;priorities and weights of &lt;a class=&#34;link&#34; href=&#34;../07-dns/#srv-records&#34; &gt;SRV records&lt;/a&gt;. A server that fails&#xA;emfor&amp;rsquo;s checks, or is drained, is not in the answer. They ask the host&amp;rsquo;s&#xA;resolver, so it must forward the zone to emfor&#xA;(&lt;a class=&#34;link&#34; href=&#34;../07-dns/#forwarding-from-the-host-resolver&#34; &gt;Forwarding from the host resolver&lt;/a&gt;); the&#xA;test below made emfor the only resolver of the container, on port 53. Only&#xA;&lt;code&gt;_tcp&lt;/code&gt; names are answered: a &lt;code&gt;_udp&lt;/code&gt; name gets &lt;code&gt;NXDOMAIN&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;Kerberos (MIT).&lt;/strong&gt; With &lt;code&gt;dns_lookup_kdc = true&lt;/code&gt; in &lt;code&gt;krb5.conf&lt;/code&gt; and the&#xA;realm equal to the service name in capitals (&lt;code&gt;KDC.SVC.EMFOR&lt;/code&gt; for the service&#xA;&lt;code&gt;kdc.svc.emfor&lt;/code&gt;), &lt;code&gt;kinit&lt;/code&gt; asked &lt;code&gt;_kerberos._udp.KDC.SVC.EMFOR&lt;/code&gt; (no answer, as&#xA;said), then &lt;code&gt;_kerberos._tcp.KDC.SVC.EMFOR&lt;/code&gt;, and went to both KDCs over TCP. A KDC&#xA;that failed its checks was not offered, and &lt;code&gt;kinit&lt;/code&gt; went to the other only.&#xA;With &lt;code&gt;balance = &amp;quot;failover&amp;quot;&lt;/code&gt; the answer has priorities 0 and 1 and &lt;code&gt;kinit&lt;/code&gt; tried&#xA;the primary first. emfor&amp;rsquo;s &lt;code&gt;tcp&lt;/code&gt; check looks at the TCP port only.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;OpenLDAP.&lt;/strong&gt; &lt;code&gt;ldapsearch -H &#39;ldap:///dc=ldap%2Cdc=svc%2Cdc=emfor&#39;&lt;/code&gt; (no host,&#xA;a DN) makes libldap ask &lt;code&gt;_ldap._tcp.ldap.svc.emfor&lt;/code&gt; and use the servers of&#xA;the answer. &lt;strong&gt;Write the commas of the DN as &lt;code&gt;%2C&lt;/code&gt;&lt;/strong&gt;: with plain commas&#xA;&lt;code&gt;ldapsearch&lt;/code&gt; says &lt;code&gt;Could not parse LDAP URI(s)&lt;/code&gt;. A server that failed its&#xA;checks was not used (3 of 3 attempts went to the other).&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;MongoDB.&lt;/strong&gt; &lt;code&gt;mongodb+srv://db.svc.emfor/&lt;/code&gt; gave PyMongo the two servers of&#xA;&lt;code&gt;_mongodb._tcp.db.svc.emfor&lt;/code&gt;. emfor has no TXT records and answers the TXT&#xA;query that &lt;code&gt;mongodb+srv&lt;/code&gt; also makes with &lt;code&gt;NOERROR&lt;/code&gt; and no records, which&#xA;PyMongo accepted. &lt;code&gt;mongodb+srv&lt;/code&gt; turns TLS on: add &lt;code&gt;tls=false&lt;/code&gt; to the URI when&#xA;the servers do not speak TLS. A server that failed its checks, and one that&#xA;was drained, was not in the list.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/tests/srv/run.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;tests/srv/&lt;/a&gt; checks these with the Debian bookworm&#xA;packages &lt;code&gt;krb5-user&lt;/code&gt; (MIT Kerberos), &lt;code&gt;ldap-utils&lt;/code&gt; (OpenLDAP 2.5.13) and&#xA;&lt;code&gt;python3-pymongo&lt;/code&gt; (3.11.0); the servers are plain listeners, so what is&#xA;checked is which servers the client is told about and goes to. Not tried: a&#xA;real KDC, LDAP or MongoDB server behind them (&lt;code&gt;kinit&lt;/code&gt; ended with &lt;code&gt;Cannot contact any KDC&lt;/code&gt;), LDAPS, sssd (&lt;code&gt;ldap_uri = _srv_&lt;/code&gt;), nslcd, Samba, Heimdal, Active&#xA;Directory names such as &lt;code&gt;_msdcs&lt;/code&gt;, a realm or domain outside &lt;code&gt;svc.emfor&lt;/code&gt; (the&#xA;&lt;code&gt;[dns] zone&lt;/code&gt; of its own that it would need), other drivers than PyMongo, other&#xA;versions. The Integrations workflow runs the test every week and when the DNS&#xA;code of emfor or the test changes.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>8. Operations</title>
            <link>https://embark.systems/en/emfor/08-operations/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/08-operations/</guid>
            <description>&lt;h2 id=&#34;commands&#34;&gt;Commands&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Command&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What it does&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor server&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Runs the daemon in the foreground.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor check-config [--show]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Validates the config and exits 0 (valid) or 2 (invalid). With &lt;code&gt;--show&lt;/code&gt;, prints the merged config and the file that set each value (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#checking-a-config&#34; &gt;chapter 3&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor status&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Shows what the running daemon has published.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor lookup &amp;lt;service&amp;gt;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Prints the backend a client would get, and with &lt;code&gt;--explain&lt;/code&gt; why (&lt;a class=&#34;link&#34; href=&#34;../06-clients/#from-the-shell-emfor-lookup&#34; &gt;chapter 6&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor --help&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Lists the commands and the default config path.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;code&gt;--config &amp;lt;path&amp;gt;&lt;/code&gt; goes before or after the command. &lt;code&gt;emfor&lt;/code&gt; with no&#xA;command only prints its usage and exits 2; it never starts a daemon.&lt;/p&gt;&#xA;&lt;h2 id=&#34;signals&#34;&gt;Signals&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Signal&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Effect&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SIGHUP&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Reload the config (the main file and the files of &lt;code&gt;include_dir&lt;/code&gt;). Applied only when the whole config is valid; otherwise the errors are logged and the old config keeps running. See &lt;a class=&#34;link&#34; href=&#34;../03-configuration/#what-a-reload-changes&#34; &gt;what a reload changes&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SIGTERM&lt;/code&gt;, &lt;code&gt;SIGINT&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Stop: checks end, the heartbeat is set to 0 (clients see a stale store at once), plugins are stopped, exit 0. A reload or start that is still loading the config is abandoned.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Use your service manager rather than signals: &lt;code&gt;systemctl reload emfor&lt;/code&gt; and&#xA;&lt;code&gt;service emfor reload&lt;/code&gt; check the config first and do not signal the daemon&#xA;when it is invalid.&lt;/p&gt;&#xA;&lt;h2 id=&#34;emfor-status&#34;&gt;emfor status&#xA;&lt;/h2&gt;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-console&#34; data-lang=&#34;console&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor status&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;store: /var/lib/emfor/state.mdb&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;daemon: running, heartbeat 0.4s ago, up 3h12m, site tpe1, dns 127.0.0.1:5300, metrics 127.0.0.1:5301&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#960050;background-color:#1e0010&#34;&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;api.svc.emfor  round-robin  on_all_down=open&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.11:8080  healthy    0.8s ago  1.2ms&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  10.0.0.12:8080  unhealthy  0.9s ago  -      connect: Connection refused&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;code&gt;emfor status&lt;/code&gt; reads only the &lt;code&gt;[store]&lt;/code&gt; table of the config (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#store&#34; &gt;chapter 3&lt;/a&gt;) and opens the&#xA;store read-only, like a client. Opening the store registers a reader in its&#xA;lock file, so the user needs read access to the store and write access to&#xA;the lock file (&lt;a class=&#34;link&#34; href=&#34;#who-can-use-the-store&#34; &gt;who can use the store&lt;/a&gt;); with the&#xA;lock file at mode 0444, &lt;code&gt;emfor status&lt;/code&gt; stops with &lt;code&gt;cannot open store …: Permission denied&lt;/code&gt;.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;daemon&lt;/strong&gt;: &lt;code&gt;running&lt;/code&gt; while the heartbeat is at most 3 seconds old,&#xA;&lt;code&gt;stale&lt;/code&gt; when it is older (the daemon hangs or died), &lt;code&gt;stopped&lt;/code&gt; after a&#xA;clean shutdown. With &lt;code&gt;stale&lt;/code&gt; or &lt;code&gt;stopped&lt;/code&gt;, the services shown are only&#xA;what the daemon wrote last.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;service&lt;/strong&gt;: &lt;code&gt;balance&lt;/code&gt; and &lt;code&gt;on_all_down&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#balance-methods&#34; &gt;chapter 5&lt;/a&gt;), and &lt;code&gt;ALL DOWN&lt;/code&gt; when no backend is&#xA;healthy.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;backend&lt;/strong&gt;: state, time since the last check, smoothed latency, then&#xA;these markers, in this order, when they apply: &lt;code&gt;pool&lt;/code&gt; for the fast&#xA;backends of a &lt;code&gt;fastest&lt;/code&gt; service; &lt;code&gt;backup&lt;/code&gt; for a backup backend&#xA;(&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#backup-backends&#34; &gt;chapter 5&lt;/a&gt;); &lt;code&gt;local-up&lt;/code&gt; and&#xA;&lt;code&gt;voted&lt;/code&gt; in a peer group (see &lt;a class=&#34;link&#34; href=&#34;#peer-groups&#34; &gt;Peer groups&lt;/a&gt;); &lt;code&gt;FLAPPING (&amp;lt;n&amp;gt; changes in 10m)&lt;/code&gt; for a backend that changed between healthy and&#xA;unhealthy 5 or more times in 10 minutes; &lt;code&gt;last check failed&lt;/code&gt; for a backend&#xA;that is still &lt;code&gt;healthy&lt;/code&gt; but whose last check failed (it turns unhealthy&#xA;after &lt;code&gt;fall&lt;/code&gt; failed checks in a row; until then lookups usually skip it,&#xA;see &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#a-backend-whose-checks-start-failing&#34; &gt;a backend whose checks start&#xA;failing&lt;/a&gt;,&#xA;but not when every healthy backend is failing); and the reason of the last&#xA;failed check. A backend drained in the config&#xA;(&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;) shows &lt;code&gt;drained&lt;/code&gt;&#xA;in place of its state, and &lt;code&gt;(checks: &amp;lt;state&amp;gt;)&lt;/code&gt;, the state the checks give,&#xA;as the last marker, after &lt;code&gt;backup&lt;/code&gt;, &lt;code&gt;local-up&lt;/code&gt;, &lt;code&gt;voted&lt;/code&gt; and &lt;code&gt;FLAPPING&lt;/code&gt;; it&#xA;has no &lt;code&gt;last check failed&lt;/code&gt; and no reason of the last failed check. A row with every marker and a short reason is about 100&#xA;columns wide (97 in a group of ten hosts with 40 backends); &lt;code&gt;--watch&lt;/code&gt; cuts&#xA;lines wider than the terminal.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;  10.0.1.3:5432   healthy  0.8s ago  1.2ms  voted  last check failed  connect: Connection refused&#xA;  10.0.1.40:5432  healthy  0.8s ago  1.2ms  backup  voted&#xA;  10.0.0.13:8080  drained  0.8s ago  1.2ms  (checks: healthy)&#xA;  127.0.0.1:9     drained  0.4s ago  -      (checks: unhealthy)&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Option&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Effect&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;--service &amp;lt;name&amp;gt;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Show one service.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;--json&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One JSON object, for scripts.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;--watch&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Redraw every second, highlight changes, and list the last 10 changes. The &lt;code&gt;store:&lt;/code&gt; and &lt;code&gt;daemon:&lt;/code&gt; lines stay at the top; scroll a long list with PgUp/PgDn, Up/Down and Home/End. Lines wider than the terminal are cut. &lt;code&gt;q&lt;/code&gt; or Ctrl-C ends it.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;--history&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;List each backend&amp;rsquo;s last 20 state changes (kept across restarts).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Exit status&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;When&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon is running and every shown service has a healthy backend.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon is stale or stopped, or a shown service is &lt;code&gt;ALL DOWN&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The config or store cannot be read, the store has another format version, &lt;code&gt;--service&lt;/code&gt; names no service, or the output cannot be written.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;A pipe closed early (&lt;code&gt;emfor status | head -1&lt;/code&gt;) does not change the exit&#xA;status. &lt;code&gt;--watch&lt;/code&gt; opens the store again when the file is replaced (as the&#xA;daemon does with a damaged store at start).&lt;/p&gt;&#xA;&lt;p&gt;One unhealthy backend does not change the status, so &lt;code&gt;emfor status&lt;/code&gt; (or&#xA;&lt;code&gt;emfor status --service &amp;lt;name&amp;gt;&lt;/code&gt;) works as a monitoring check from cron or&#xA;a Nagios-style system as it is. More in&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/daemon.md#emfor-status&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/daemon.md&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;metrics&#34;&gt;Metrics&#xA;&lt;/h2&gt;&lt;p&gt;With a &lt;code&gt;[metrics]&lt;/code&gt; table the daemon serves Prometheus metrics at&#xA;&lt;code&gt;http://127.0.0.1:5301/metrics&lt;/code&gt;. The endpoint has no authentication and&#xA;no TLS; keep it on loopback or firewall it.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Metric&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_state{service,backend,state}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 for the current state&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_check_latency_seconds&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Smoothed check latency (healthy backends)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_last_check_timestamp_seconds&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;When the last check finished&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_in_pool&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 in the pool of a &lt;code&gt;fastest&lt;/code&gt; service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_drain&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 for a backend drained in the config, else 0; &lt;code&gt;emfor_backend_state&lt;/code&gt; keeps the state the checks give&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_service_all_down&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1 when no backend is healthy&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_checks_total{result}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Checks by result: &lt;code&gt;up&lt;/code&gt;, &lt;code&gt;down&lt;/code&gt;, &lt;code&gt;error&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_backend_transitions_total{to}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;State changes by new state&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_store_commit_failures_total&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Failed store commits&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_store_last_commit_timestamp_seconds&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Last successful commit; should be under a few seconds old&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_store_used_bytes&lt;/code&gt;, &lt;code&gt;emfor_store_map_size_bytes&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Store use against &lt;code&gt;[store] map_size&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_plugin_up&lt;/code&gt;, &lt;code&gt;emfor_plugin_restarts_total&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Plugin health&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_dns_rate_limited_total{action}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[dns]&lt;/code&gt;: UDP answers over &lt;code&gt;rate_limit&lt;/code&gt;, by what was sent instead: &lt;code&gt;truncated&lt;/code&gt; or &lt;code&gt;dropped&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#rate-limit&#34; &gt;chapter 7&lt;/a&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_up{peer}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: 1 while that member is live&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_messages_rejected_total{reason}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: dropped messages by reason: &lt;code&gt;malformed&lt;/code&gt;, &lt;code&gt;auth&lt;/code&gt; (wrong secret, or a message that does not decrypt), &lt;code&gt;sender&lt;/code&gt; (not a member, or a spoofed address), &lt;code&gt;site&lt;/code&gt;, &lt;code&gt;stale&lt;/code&gt; (clocks more than 5 seconds apart), &lt;code&gt;replay&lt;/code&gt;, &lt;code&gt;plaintext&lt;/code&gt; (an unencrypted message while &lt;code&gt;accept_plaintext = false&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_notifications_total{direction}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: failure reports &lt;code&gt;sent&lt;/code&gt; (one per member) and &lt;code&gt;received&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_results_total{direction}&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: check results of assigned checkers &lt;code&gt;sent&lt;/code&gt; (one per member) and &lt;code&gt;received&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_disagreements&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: backends the group votes down while this host&amp;rsquo;s own check passes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_checks_skipped_total&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: checks this host did not make because two assigned checkers agreed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_peer_queue_dropped_total&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;With &lt;code&gt;[peers]&lt;/code&gt;: &lt;code&gt;down&lt;/code&gt; and &lt;code&gt;result&lt;/code&gt; messages not sent because the peer task fell more than 4096 messages behind (logged as &lt;code&gt;the peer task is behind&lt;/code&gt;); should stay 0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor_build_info{version}&lt;/code&gt;, &lt;code&gt;emfor_start_time_seconds&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/prometheus/emfor-rules.yml&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/prometheus/emfor-rules.yml&lt;/a&gt;&#xA;has alerting rules: the endpoint is down, a service is &lt;code&gt;ALL DOWN&lt;/code&gt;, store&#xA;commits fail, a backend is unhealthy for 10 minutes, flaps, mostly errors,&#xA;or is no longer checked. &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/daemon.md#metrics&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;docs/deploy/daemon.md&lt;/a&gt; has the metrics with&#xA;their types and labels.&lt;/p&gt;&#xA;&lt;h3 id=&#34;peer-groups&#34;&gt;Peer groups&#xA;&lt;/h3&gt;&lt;p&gt;Members of a peer group need UDP between each other on their &lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;&lt;code&gt;[peers]&lt;/code&gt;&lt;/a&gt;&#xA;&lt;code&gt;listen&lt;/code&gt; port (5302 in the examples) and clocks within 5 seconds of each&#xA;other (NTP). If &lt;code&gt;emfor_peer_up&lt;/code&gt; stays 0 for a member, look at&#xA;&lt;code&gt;emfor_peer_messages_rejected_total&lt;/code&gt; on both sides: &lt;code&gt;auth&lt;/code&gt; means the&#xA;secrets differ, &lt;code&gt;stale&lt;/code&gt; that the clocks drift, &lt;code&gt;sender&lt;/code&gt; that &lt;code&gt;members&lt;/code&gt; or&#xA;&lt;code&gt;listen&lt;/code&gt; do not match, &lt;code&gt;site&lt;/code&gt; that the hosts have different &lt;code&gt;site&lt;/code&gt; values.&#xA;&lt;code&gt;malformed&lt;/code&gt; on a member that runs 0.3.0 means it cannot read the&#xA;encrypted messages of upgraded members: set &lt;code&gt;encrypt = false&lt;/code&gt; on those&#xA;until it is upgraded (see&#xA;&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;&lt;code&gt;[peers]&lt;/code&gt;&lt;/a&gt;). A warning &lt;code&gt;peer &amp;lt;address&amp;gt; sends plaintext messages (EMP1); set encrypt = true there&lt;/code&gt; means that member&#xA;still sends readable messages; they are accepted unless&#xA;&lt;code&gt;accept_plaintext = false&lt;/code&gt;, in which case they count as &lt;code&gt;plaintext&lt;/code&gt; and&#xA;that member is not live here.&lt;/p&gt;&#xA;&lt;p&gt;Reasons tell where a &lt;code&gt;down&lt;/code&gt; came from: &lt;code&gt;peers 2/3: &amp;lt;reason&amp;gt;&lt;/code&gt; means two of&#xA;the three assigned checkers saw the backend down (the reason is theirs),&#xA;&lt;code&gt;local: &amp;lt;reason&amp;gt;&lt;/code&gt; that only this host sees it down while the vote did not&#xA;say down, and a reason without a prefix that there was no vote (fewer than&#xA;three live members, or no recent result). &lt;code&gt;emfor status&lt;/code&gt; shows the group&#xA;under the daemon line:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 2/3 live, not live 10.0.0.7:5302&#xA;peers: check definition of api.svc.emfor differs at 10.0.0.6:5302&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;and marks backends with &lt;code&gt;local-up&lt;/code&gt; when the group votes them down while&#xA;this host&amp;rsquo;s own check passes. Those are worth a look: this host reaches a&#xA;backend the others do not (often a network partition or a firewall rule on&#xA;the other hosts), and &lt;code&gt;emfor_peer_disagreements&lt;/code&gt; counts them. To see which members decide&#xA;a backend, read its &lt;code&gt;assigned_checkers&lt;/code&gt; in &lt;code&gt;emfor status --json&lt;/code&gt; (below).&#xA;A &amp;ldquo;check&#xA;definition differs&amp;rdquo; line means that member&amp;rsquo;s &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;check_options&lt;/code&gt; or&#xA;&lt;code&gt;timeout&lt;/code&gt; for the service differ from this host&amp;rsquo;s, so its results are not&#xA;counted; make the configs match. The same data is in the store key&#xA;&lt;code&gt;meta/peers&lt;/code&gt; and in &lt;code&gt;emfor status --json&lt;/code&gt; as &lt;code&gt;peers&lt;/code&gt;. Its &lt;code&gt;services_off&lt;/code&gt;&#xA;lists the services this host leaves out of the group (&lt;code&gt;peers = false&lt;/code&gt;);&#xA;their backends are never &lt;code&gt;voted&lt;/code&gt; and have no &lt;code&gt;assigned_checkers&lt;/code&gt;. Each&#xA;entry of &lt;code&gt;members&lt;/code&gt; has its own &lt;code&gt;services_off&lt;/code&gt;: the services of this host&#xA;that the member announces it leaves out. That member is not counted as&#xA;an assigned checker of those services here, and &lt;code&gt;emfor status&lt;/code&gt; leaves it&#xA;out the same way. The text output shows this as a line per member after&#xA;the &lt;code&gt;peers:&lt;/code&gt; line, listing only the services shown:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 10.0.0.6:5302 leaves out api.svc.emfor db.svc.emfor&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;When this host and the live members that have not left a service out number&#xA;fewer than three, the group cannot vote for it: no backend of it is voted&#xA;and every reason is this host&amp;rsquo;s own, without a &lt;code&gt;peers d/f:&lt;/code&gt; prefix. If that&#xA;is so for every shown service that takes part in the group, the &lt;code&gt;peers:&lt;/code&gt;&#xA;line ends with &lt;code&gt;too few live members to vote&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 1/2 live, not live 10.0.0.7:5302, too few live members to vote&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;If some services can vote and others cannot, the line says nothing; a&#xA;backend with no &lt;code&gt;checkers&lt;/code&gt; in &lt;code&gt;assigned_checkers&lt;/code&gt; (below) is decided by&#xA;this host alone.&lt;/p&gt;&#xA;&lt;p&gt;In a group of four or more, &lt;code&gt;emfor_peer_checks_skipped_total&lt;/code&gt; grows on&#xA;hosts that are not an assigned checker of some backends: they let the&#xA;votes decide most check times and check those backends themselves about&#xA;every third &lt;code&gt;interval&lt;/code&gt;. &lt;code&gt;emfor status&lt;/code&gt; marks those backends &lt;code&gt;voted&lt;/code&gt; and&#xA;ends the &lt;code&gt;peers:&lt;/code&gt; line with how many of the shown backends this host&#xA;checks every time:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;peers: 4/4 live, checks 12/40 backends here&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;&lt;code&gt;emfor status --json&lt;/code&gt; lists each backend&amp;rsquo;s assigned checkers under&#xA;&lt;code&gt;assigned_checkers&lt;/code&gt;, in rank order:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-json&#34; data-lang=&#34;json&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;assigned_checkers&amp;#34;&lt;/span&gt;&lt;span style=&#34;color:#960050;background-color:#1e0010&#34;&gt;:&lt;/span&gt; [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  {&lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;service&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;backend&amp;#34;&lt;/span&gt;: &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;   &lt;span style=&#34;color:#f92672&#34;&gt;&amp;#34;checkers&amp;#34;&lt;/span&gt;: [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.5:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.8:5302&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.6:5302&amp;#34;&lt;/span&gt;]}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It is &lt;code&gt;null&lt;/code&gt; while the daemon is not running or &lt;code&gt;meta/peers&lt;/code&gt; is missing.&#xA;status works these out from the live members in&#xA;&lt;code&gt;meta/peers&lt;/code&gt;, which the daemon updates every second, so right after a&#xA;member comes or goes they can be up to a second behind. &lt;code&gt;voted&lt;/code&gt; only says&#xA;that this host is not an assigned checker; whether a given check time was&#xA;decided by votes depends on the votes at that moment.&#xA;&lt;code&gt;emfor_backend_checks_total&lt;/code&gt; still counts every check time, decided by&#xA;votes or not. A backend&amp;rsquo;s latency is measured only&#xA;by this host&amp;rsquo;s own checks, so on such a host it changes about every third&#xA;&lt;code&gt;interval&lt;/code&gt;. The &lt;code&gt;&amp;lt;age&amp;gt; ago&lt;/code&gt; of a voted backend is the time its result was&#xA;applied, whoever decided it, so it can be recent while this host last&#xA;connected to the backend up to about three &lt;code&gt;interval&lt;/code&gt;s ago.&lt;/p&gt;&#xA;&lt;h2 id=&#34;logs&#34;&gt;Logs&#xA;&lt;/h2&gt;&lt;p&gt;The daemon logs to stderr: the journal under systemd, syslog (tag &lt;code&gt;emfor&lt;/code&gt;)&#xA;under the FreeBSD rc.d script. Set the level with &lt;code&gt;[log] level&lt;/code&gt;, or for&#xA;one run with &lt;code&gt;RUST_LOG&lt;/code&gt; (for example &lt;code&gt;RUST_LOG=debug&lt;/code&gt;). Colors are used&#xA;only when stderr is a terminal and &lt;code&gt;NO_COLOR&lt;/code&gt; is not set.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Level&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;What you see&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;error&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Store commit failures, rejected reloads&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;warn&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Plugin exits, damaged store replaced, services outside the DNS zone or with mixed ports, unparsable plugin output, dropped peer messages (one line per source and reason a minute), a member sending plaintext (&lt;code&gt;EMP1&lt;/code&gt;) messages while &lt;code&gt;encrypt = true&lt;/code&gt; (one line per member a minute), an unreadable &lt;code&gt;[peers] secret_file&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;info&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Start and stop, every backend state change with its reason, plugin stderr lines, peer members becoming live or not live, &lt;code&gt;peer … reports backend … down (…); checking now&lt;/code&gt;, &lt;code&gt;backend … of … is down at 2/3 peers but up here&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;debug&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The duration of every passed check, peer reports that were ignored and why, results from a member with another check definition&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;exit-codes&#34;&gt;Exit codes&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Status&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Meaning&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Retry helps?&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;0&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Clean stop (&lt;code&gt;SIGTERM&lt;/code&gt; / &lt;code&gt;SIGINT&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;—&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;1&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Cannot start: a listen address is in use, the store cannot be opened, or another emfor holds the store (&lt;code&gt;&amp;lt;store&amp;gt;.writer&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No; fix the cause&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;2&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Invalid config, or no command given&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No; run &lt;code&gt;emfor check-config&lt;/code&gt; first (with the binary you are about to start) and fix the config&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;70&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The state writer or a DNS listener stopped while running&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Yes; the systemd unit restarts it&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;the-store-on-disk&#34;&gt;The store on disk&#xA;&lt;/h2&gt;&lt;p&gt;The daemon commits to the store at least once a second and on every state&#xA;change. By default (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#store&#34; &gt;&lt;code&gt;[store] sync = false&lt;/code&gt;&lt;/a&gt;) a commit does not wait for the&#xA;disk, so a stalled disk cannot make the heartbeat late and the store look&#xA;stale to every application on the host. Nothing in the store needs to&#xA;survive a host crash: the daemon rewrites every service at start. Only the&#xA;state history (&lt;code&gt;emfor status --history&lt;/code&gt;) is kept across restarts.&lt;/p&gt;&#xA;&lt;p&gt;After a power loss the store may be damaged. The daemon then logs a&#xA;warning, moves it to &lt;code&gt;&amp;lt;store&amp;gt;.corrupt&lt;/code&gt;, and starts with a new store;&#xA;running applications follow the new file by themselves. The only thing&#xA;lost is the state history: the last 20 state changes of each backend&#xA;that &lt;code&gt;emfor status --history&lt;/code&gt; shows. Every state change is also in the&#xA;daemon&amp;rsquo;s log (&lt;code&gt;changed from healthy to unhealthy&lt;/code&gt; and the like), so look&#xA;there for what happened before the crash. Set &lt;code&gt;sync = true&lt;/code&gt; to flush&#xA;every commit, at the risk described above.&lt;/p&gt;&#xA;&lt;p&gt;Never change the store file in place: no &lt;code&gt;cp&lt;/code&gt; over it, no &lt;code&gt;truncate&lt;/code&gt;, no&#xA;shell redirection (&lt;code&gt;&amp;gt; state.mdb&lt;/code&gt;). The daemon and every application read&#xA;it through a memory map, and a file that becomes shorter while they have&#xA;it open ends each of those processes with &lt;code&gt;SIGBUS&lt;/code&gt;. To put another file&#xA;in its place, write it next to the store and &lt;code&gt;mv&lt;/code&gt; it over the store path:&#xA;running applications then follow the new file. A store that is already&#xA;shorter than LMDB records when it is opened, as after a power loss, is&#xA;handled like other damage: the daemon moves it to &lt;code&gt;&amp;lt;store&amp;gt;.corrupt&lt;/code&gt;,&#xA;lookups fail with &lt;code&gt;UNAVAILABLE&lt;/code&gt; (the message says &lt;code&gt;truncated&lt;/code&gt;) until&#xA;there is a good store, and &lt;code&gt;emfor status&lt;/code&gt; exits 2. LMDB itself can leave&#xA;a store a few pages shorter than it records (pages it freed again before&#xA;writing them); the daemon extends the file after every commit, so this&#xA;only matters for a store an older emfor wrote: the first start after the&#xA;upgrade may move it aside, losing only the state history.&lt;/p&gt;&#xA;&lt;p&gt;Commits run on their own thread, so a slow disk does not delay DNS&#xA;answers, metrics or checks. Watch &lt;code&gt;emfor_store_used_bytes&lt;/code&gt; against&#xA;&lt;code&gt;map_size&lt;/code&gt;: when the store is full, commits fail and clients soon see a&#xA;stale store.&lt;/p&gt;&#xA;&lt;h3 id=&#34;keeping-the-store-in-memory&#34;&gt;Keeping the store in memory&#xA;&lt;/h3&gt;&lt;p&gt;Since nothing in the store has to outlive a reboot, it can live on a&#xA;tmpfs. Commits, about one a second, then never reach a disk, a stalled&#xA;disk cannot delay them even with &lt;code&gt;sync = true&lt;/code&gt;, and a power loss leaves&#xA;no damaged file behind: the daemon simply starts with a new store. The&#xA;cost is the state history, which is then lost at every reboot, not only&#xA;after a crash (a restart of the daemon alone keeps it).&lt;/p&gt;&#xA;&lt;p&gt;Mount the tmpfs on the store&amp;rsquo;s directory itself (&lt;code&gt;/var/lib/emfor&lt;/code&gt;, or&#xA;&lt;code&gt;/var/db/emfor&lt;/code&gt; with the FreeBSD port), so that neither &lt;code&gt;[store] path&lt;/code&gt;&#xA;nor &lt;code&gt;EMFOR_STORE&lt;/code&gt; changes. Make it at least&#xA;as large as &lt;code&gt;[store] map_size&lt;/code&gt; (&lt;code&gt;64MiB&lt;/code&gt; by default); only the pages in&#xA;use take memory, a few tens of KiB for a few services.&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;systemd: add to &lt;code&gt;/etc/fstab&lt;/code&gt;&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;tmpfs  /var/lib/emfor  tmpfs  size=64m,mode=0755,nosuid,nodev,noexec  0 0&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;and, with &lt;code&gt;systemctl edit emfor&lt;/code&gt;, &lt;code&gt;RequiresMountsFor=/var/lib/emfor&lt;/code&gt;&#xA;under &lt;code&gt;[Unit]&lt;/code&gt;, so the daemon starts only once it is mounted.&#xA;&lt;code&gt;StateDirectory=emfor&lt;/code&gt; gives the directory to the &lt;code&gt;emfor&lt;/code&gt; user at&#xA;every start.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;li&gt;&#xA;&lt;p&gt;FreeBSD: add to &lt;code&gt;/etc/fstab&lt;/code&gt;&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;tmpfs  /var/db/emfor  tmpfs  rw,size=64m,mode=0755,uid=&amp;lt;uid of emfor&amp;gt;  0 0&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;with the number &lt;code&gt;id -u emfor&lt;/code&gt; prints, so the directory belongs to&#xA;&lt;code&gt;emfor&lt;/code&gt; after every boot.&lt;/p&gt;&#xA;&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Do not use systemd&amp;rsquo;s &lt;code&gt;RuntimeDirectory=&lt;/code&gt; (&lt;code&gt;/run/emfor&lt;/code&gt;) for the store: it&#xA;deletes the directory whenever the daemon stops, which loses the history&#xA;at every restart and leaves running applications on a deleted file.&lt;/p&gt;&#xA;&lt;p&gt;This was checked in a Debian container with a tmpfs on &lt;code&gt;/var/lib/emfor&lt;/code&gt;:&#xA;the store takes 32 KiB for one service, &lt;code&gt;emfor status&lt;/code&gt; reads it, and the&#xA;history survives a restart of the daemon. The systemd and FreeBSD&#xA;settings above have not been run on a host.&lt;/p&gt;&#xA;&lt;h2 id=&#34;who-can-use-the-store&#34;&gt;Who can use the store&#xA;&lt;/h2&gt;&lt;p&gt;Applications read the data file and register their reads in the lock file&#xA;&lt;code&gt;&amp;lt;store&amp;gt;-lock&lt;/code&gt;, so every application user needs read access to the store&#xA;and write access to the lock file. The defaults (&lt;code&gt;file_mode = &amp;quot;0644&amp;quot;&lt;/code&gt;,&#xA;&lt;code&gt;lock_file_mode = &amp;quot;0666&amp;quot;&lt;/code&gt;) let every local user do this, and also let any&#xA;local user who writes the lock file stall the daemon&amp;rsquo;s commits or disturb&#xA;lookups. Keep the defaults only on hosts where every local user is trusted.&#xA;Otherwise, give your applications a group:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Create a group, for example &lt;code&gt;emfor-clients&lt;/code&gt;, and add each application&#xA;user to it.&lt;/li&gt;&#xA;&lt;li&gt;Make it the daemon&amp;rsquo;s group. Under systemd: &lt;code&gt;systemctl edit emfor&lt;/code&gt;, add&#xA;&lt;code&gt;Group=emfor-clients&lt;/code&gt; and set &lt;code&gt;StateDirectoryMode=0750&lt;/code&gt;. Under FreeBSD&#xA;with the port:&#xA;&lt;code&gt;chgrp emfor-clients /var/db/emfor &amp;amp;&amp;amp; chmod 0750 /var/db/emfor&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;In &lt;code&gt;[store]&lt;/code&gt;, set &lt;code&gt;file_mode = &amp;quot;0640&amp;quot;&lt;/code&gt; and &lt;code&gt;lock_file_mode = &amp;quot;0660&amp;quot;&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;chgrp emfor-clients&lt;/code&gt; the existing store and lock files, and restart the&#xA;daemon (it sets the modes at every start).&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2 id=&#34;rolling-deploy-with-drain&#34;&gt;Rolling deploy with drain&#xA;&lt;/h2&gt;&lt;p&gt;To deploy to the backends of a service one at a time without cutting the&#xA;requests that are running, take the backend out of the choice with&#xA;&lt;code&gt;drain = true&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;),&#xA;wait until its requests are over, stop, deploy and start it, wait until its&#xA;checks pass, and put it back. emfor does the first and the last step and&#xA;shows the checks in between; the waiting is yours, because emfor is not in&#xA;the data path and does not see a request.&lt;/p&gt;&#xA;&lt;p&gt;A &lt;code&gt;conf.d&lt;/code&gt; file cannot switch one backend of a service that another file&#xA;defines: a list is replaced as a whole, so the file would have to repeat the&#xA;whole &lt;code&gt;backends&lt;/code&gt; list. Give the service its own file with one backend per&#xA;line, and change that line:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# /etc/emfor/conf.d/20-api.toml&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;tcp&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:8080&amp;#34;&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/deploy/emfor-drain.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/deploy/emfor-drain.sh&lt;/a&gt; does&#xA;the steps for one backend. Before you start:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;No package installs the script. Copy it to the emfor host (for &lt;code&gt;on&lt;/code&gt;, &lt;code&gt;off&lt;/code&gt;&#xA;and &lt;code&gt;wait-ready&lt;/code&gt;) and to the backend host (for &lt;code&gt;wait-idle&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Run it as root on the emfor host: &lt;code&gt;emfor check-config&lt;/code&gt; run as root accepts&#xA;only conf.d files that root owns, and the script reloads emfor.&lt;/li&gt;&#xA;&lt;li&gt;It needs &lt;code&gt;jq&lt;/code&gt;, and &lt;code&gt;ss&lt;/code&gt; (Linux, iproute2) for &lt;code&gt;wait-idle&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;The reload command is &lt;code&gt;systemctl reload emfor&lt;/code&gt;. Set &lt;code&gt;EMFOR_RELOAD&lt;/code&gt; for&#xA;another one (FreeBSD: &lt;code&gt;service emfor reload&lt;/code&gt;, with the files in&#xA;&lt;code&gt;/usr/local/etc/emfor/conf.d&lt;/code&gt;).&lt;/li&gt;&#xA;&lt;li&gt;It calls &lt;code&gt;emfor check-config&lt;/code&gt; and &lt;code&gt;emfor status&lt;/code&gt; without &lt;code&gt;--config&lt;/code&gt;, so&#xA;they use the config path built into emfor. For another path set&#xA;&lt;code&gt;EMFOR_CONFIG=/path/to/emfor.toml&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Each backend must be one line, written exactly &lt;code&gt;{ addr = &amp;quot;10.0.0.11:8080&amp;quot; },&lt;/code&gt;&#xA;with a space before the &lt;code&gt;}&lt;/code&gt;. For a line it cannot change the script stops&#xA;and leaves the file alone. It writes the new line to a copy, moves it&#xA;over the file, runs &lt;code&gt;check-config&lt;/code&gt;, and puts the old file back if the check&#xA;refuses; only then does it reload.&lt;/li&gt;&#xA;&lt;li&gt;It finds the backend in &lt;code&gt;emfor status --json&lt;/code&gt; by address, in the first&#xA;service (in name order) that has that address. If two services share an&#xA;address, set &lt;code&gt;EMFOR_SERVICE=&amp;lt;name&amp;gt;&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-sh&#34; data-lang=&#34;sh&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# On the host that runs emfor: add drain = true to the line of the backend,&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# run emfor check-config, reload, and wait until `emfor status` shows it drained.&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor-drain.sh on /etc/emfor/conf.d/20-api.toml 10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# On 10.0.0.11: wait until no connection to port 8080 has moved data for 3 seconds.&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor-drain.sh wait-idle &lt;span style=&#34;color:#ae81ff&#34;&gt;8080&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;deploy-and-restart-api&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# On the emfor host again: wait until the checks of the backend have said&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#75715e&#34;&gt;# healthy for 10 seconds, then remove drain and reload.&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor-drain.sh wait-ready 10.0.0.11:8080 &lt;span style=&#34;color:#ae81ff&#34;&gt;120&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;emfor-drain.sh off /etc/emfor/conf.d/20-api.toml 10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;What &lt;code&gt;wait-ready&lt;/code&gt; guarantees. A backend that has just stopped keeps the state&#xA;&lt;code&gt;healthy&lt;/code&gt; until &lt;code&gt;fall&lt;/code&gt; checks have failed (with the defaults, &lt;code&gt;interval&lt;/code&gt; 2&#xA;seconds and &lt;code&gt;fall&lt;/code&gt; 3, about 6 seconds), and a drained backend hides the&#xA;result of its last check (the store shows only the state of the checks). So&#xA;&lt;code&gt;wait-ready ADDR [SECONDS [SETTLE]]&lt;/code&gt; does not return at the first &lt;code&gt;healthy&lt;/code&gt;:&#xA;it returns when a check newer than the command has finished and the checks&#xA;have said &lt;code&gt;healthy&lt;/code&gt; for SETTLE seconds in a row (default 10), and for a&#xA;backend that is not drained also when the last check passed. SETTLE must be&#xA;longer than &lt;code&gt;fall&lt;/code&gt; x &lt;code&gt;interval&lt;/code&gt; plus &lt;code&gt;timeout&lt;/code&gt; of the service; raise it if&#xA;you changed those. It does not count &lt;code&gt;rise&lt;/code&gt; and does not look at what the&#xA;application does with a request: a backend that passes a &lt;code&gt;tcp&lt;/code&gt; check and&#xA;still answers 500 is ready for it.&lt;/p&gt;&#xA;&lt;p&gt;Do the backends one at a time, and look at &lt;code&gt;emfor status&lt;/code&gt; between them (its&#xA;exit status is 1 when a service is &lt;code&gt;ALL DOWN&lt;/code&gt;): a service whose backends are&#xA;all drained answers nothing. &lt;code&gt;drain&lt;/code&gt; is a setting of one host&amp;rsquo;s config&#xA;(&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;), so when emfor runs&#xA;on every application host, run the first and the last step on each of them.&#xA;A HAProxy that has the emfor servers of several hosts as name servers needs&#xA;them to agree on &lt;code&gt;drain&lt;/code&gt; too, and is better off with one local name server&#xA;(&lt;a class=&#34;link&#34; href=&#34;../07-dns-recipes/#using-it-with-haproxy&#34; &gt;HAProxy in chapter 7&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;What the waiting is for, as &lt;code&gt;tests/deploy/run.sh&lt;/code&gt; saw it with nginx 1.30.6&#xA;and 1.31.8 taking its upstream from emfor&amp;rsquo;s SRV answers&#xA;(&lt;a class=&#34;link&#34; href=&#34;../07-dns-recipes/#using-it-with-nginx&#34; &gt;chapter 7&lt;/a&gt;):&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;A request that is running when the backend is drained finishes (a 1200&#xA;byte answer, sent over about 6 seconds, arrived whole). Stopping the&#xA;backend without a drain cut the same request after 146 bytes.&lt;/li&gt;&#xA;&lt;li&gt;New requests stop reaching a drained backend within a few seconds (nginx&#xA;asks again every second). For a client that reads the store it is the&#xA;next lookup; a client that asks DNS may cache the answer for its TTL.&lt;/li&gt;&#xA;&lt;li&gt;nginx retried a request that a stopped backend had refused, so a stop&#xA;without a drain lost no new request in the test (0 of 64 to 96 POSTs).&#xA;The drain matters for the requests that are running, and for clients that&#xA;do not retry; the second was not tried.&lt;/li&gt;&#xA;&lt;li&gt;Connections that a proxy keeps open for the next request stay open after&#xA;the drain, so &amp;ldquo;no connection to the port&amp;rdquo; does not come: nginx 1.30&#xA;keeps idle connections to an upstream by default, and 1 or 2 were still&#xA;open when the wait was over. &lt;code&gt;wait-idle&lt;/code&gt; therefore waits for a quiet&#xA;time, 3 seconds by default (a third argument, in milliseconds), read from&#xA;&lt;code&gt;ss -i&lt;/code&gt;. A request that stays silent for longer than that, a long poll or&#xA;an event stream, looks idle to it: raise the time, or count the requests&#xA;in the application.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;wait-ready&lt;/code&gt; with the default &lt;code&gt;interval&lt;/code&gt; and &lt;code&gt;fall&lt;/code&gt;: the test stops a&#xA;drained backend and looks at &lt;code&gt;wait-ready&lt;/code&gt; at once; the checks still said&#xA;&lt;code&gt;healthy&lt;/code&gt; then, and &lt;code&gt;wait-ready&lt;/code&gt; returned only after the backend had been&#xA;started again and its checks had passed for 10 seconds (26 seconds after&#xA;the stop, the backend being started 12 seconds after it). The rest of the&#xA;test config has &lt;code&gt;interval = &amp;quot;200ms&amp;quot;&lt;/code&gt; and &lt;code&gt;fall = 1&lt;/code&gt;, with which a stopped&#xA;backend shows &lt;code&gt;unhealthy&lt;/code&gt; within a fraction of a second; those timings hide the&#xA;problem, so the case runs with the defaults. The test also checks that the&#xA;script leaves a file alone when it cannot change the line, and puts the old&#xA;file back when &lt;code&gt;check-config&lt;/code&gt; refuses the new one.&lt;/li&gt;&#xA;&lt;li&gt;&lt;code&gt;off&lt;/code&gt; puts the backend back at once, without counting &lt;code&gt;rise&lt;/code&gt; again, and&#xA;without slow start; a backend that was restarted while it was drained&#xA;ramps from the moment its checks passed, so after a short deploy it may&#xA;still be inside &lt;code&gt;slow_start&lt;/code&gt;&#xA;(&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#slow-start&#34; &gt;chapter 5&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Not tried: HAProxy, which also retries, and its own sessions; HTTPS;&#xA;clients that keep their own connection pool to the backends (a drain does not&#xA;close a pooled connection); several emfor hosts. The script and the test ran&#xA;on Linux only. An earlier version of the script (before &lt;code&gt;wait-ready&lt;/code&gt; settled&#xA;and before &lt;code&gt;EMFOR_CONFIG&lt;/code&gt;, &lt;code&gt;EMFOR_SERVICE&lt;/code&gt; and the restore of the old file)&#xA;passed on FreeBSD 15.1 (nginx 1.30.6 and &lt;code&gt;jq&lt;/code&gt; from the packages) when run&#xA;by hand with &lt;code&gt;wait-idle&lt;/code&gt; replaced by a fixed 8-second wait; the present&#xA;script was not run there. &lt;code&gt;wait-idle&lt;/code&gt; needs &lt;code&gt;ss&lt;/code&gt;, a Linux tool, and stops&#xA;with a message elsewhere: wait a fixed time, at least the longest request&#xA;and the time a proxy needs to ask again, or count the requests in the&#xA;application. &lt;code&gt;tests/deploy/inside.sh&lt;/code&gt; itself calls &lt;code&gt;ss&lt;/code&gt; and writes&#xA;&lt;code&gt;/etc/nginx/nginx.conf&lt;/code&gt;, so a FreeBSD run needs edits to it. The Integrations&#xA;workflow runs the test on Linux every week and when the test or the script&#xA;changes.&lt;/p&gt;&#xA;&lt;h2 id=&#34;watching-emfor-from-nagios-icinga-or-zabbix&#34;&gt;Watching emfor from Nagios, Icinga or Zabbix&#xA;&lt;/h2&gt;&lt;p&gt;Two small scripts connect emfor to a monitoring system that speaks the Nagios&#xA;plugin convention (exit status 0 OK, 1 WARNING, 2 CRITICAL, 3 UNKNOWN, a first&#xA;line of text, perfdata after a &lt;code&gt;|&lt;/code&gt;). They are examples to copy, in&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/examples/nagios/check_emfor.sh&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;examples/nagios/&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;p&gt;&lt;strong&gt;Watching emfor: &lt;code&gt;check_emfor.sh&lt;/code&gt;.&lt;/strong&gt; It reads &lt;code&gt;emfor status --json&lt;/code&gt; (it needs&#xA;&lt;code&gt;jq&lt;/code&gt;) and gives the Nagios status of it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-console&#34; data-lang=&#34;console&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ check_emfor.sh -s web.svc.emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;EMFOR WARNING - web.svc.emfor: 2 of 3 healthy (127.0.0.1:7002 WARNING - slow on 7002) | &amp;#39;web.svc.emfor_healthy&amp;#39;=2;3;1;0;3&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ echo $?&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;1&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;ul&gt;&#xA;&lt;li&gt;CRITICAL: the daemon is stale or stopped, or a service has fewer than &lt;code&gt;-c&lt;/code&gt;&#xA;healthy backends (default 1: all down, or all drained).&lt;/li&gt;&#xA;&lt;li&gt;WARNING: fewer than &lt;code&gt;-w&lt;/code&gt; are healthy (default: every backend that is not&#xA;drained).&lt;/li&gt;&#xA;&lt;li&gt;UNKNOWN: &lt;code&gt;emfor&lt;/code&gt; or &lt;code&gt;jq&lt;/code&gt; cannot be run, the store cannot be read, or the&#xA;service does not exist.&lt;/li&gt;&#xA;&lt;li&gt;A drained backend is left out of the counts (it is out on purpose) and is&#xA;named in the text. &lt;code&gt;-s&lt;/code&gt; picks a service (default every service), &lt;code&gt;-C&lt;/code&gt; the&#xA;config that names the store.&lt;/li&gt;&#xA;&lt;li&gt;The user the monitoring runs as must be able to read the store and write&#xA;its lock file (&lt;a class=&#34;link&#34; href=&#34;#who-can-use-the-store&#34; &gt;who can use the store&lt;/a&gt;); with&#xA;the recommended &lt;code&gt;lock_file_mode = &amp;quot;0660&amp;quot;&lt;/code&gt; that user must be in the group.&#xA;With NRPE or SSH the script runs on the host that has the store.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;Using a Nagios plugin as a check: &lt;code&gt;emfor-exec.sh&lt;/code&gt;.&lt;/strong&gt; emfor&amp;rsquo;s &lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#exec&#34; &gt;&lt;code&gt;exec&lt;/code&gt;&#xA;check&lt;/a&gt; starts a program without a shell and passes&#xA;the backend in the environment (&lt;code&gt;EMFOR_IP&lt;/code&gt;, &lt;code&gt;EMFOR_PORT&lt;/code&gt;), so a plugin cannot&#xA;be given &lt;code&gt;-H $EMFOR_IP&lt;/code&gt; in the config. &lt;code&gt;emfor-exec.sh&lt;/code&gt; does that and runs the&#xA;plugin:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;web.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;exec&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:80&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:80&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;timeout&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;5s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt; = { &lt;span style=&#34;color:#a6e22e&#34;&gt;command&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/usr/local/libexec/emfor-exec.sh&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/usr/local/libexec/nagios/check_http&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;-u&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;] }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;It runs &lt;code&gt;check_http -H &amp;lt;ip&amp;gt; -p &amp;lt;port&amp;gt; -u /healthz&lt;/code&gt; and returns the plugin&amp;rsquo;s&#xA;exit status; &lt;code&gt;--host-flag&lt;/code&gt; and &lt;code&gt;--port-flag&lt;/code&gt; change the two flags for a plugin&#xA;that uses others. What emfor makes of it: 0 is up, &lt;strong&gt;1 (WARNING) is down&lt;/strong&gt;, 2&#xA;(CRITICAL) is down, and 3 (UNKNOWN) is an error: the state does not change at&#xA;first, and after &lt;code&gt;fall&lt;/code&gt; x &lt;code&gt;interval&lt;/code&gt; of nothing but errors it becomes&#xA;&lt;code&gt;unknown&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#rise-fall-and-errors&#34; &gt;chapter 4&lt;/a&gt;); the first line of the plugin&amp;rsquo;s output is the reason, and a&#xA;plugin that has not finished at &lt;code&gt;timeout&lt;/code&gt; is down. &lt;code&gt;--warning up&lt;/code&gt; turns a&#xA;WARNING into up. A process is started for every check of every backend, so&#xA;choose &lt;code&gt;interval&lt;/code&gt; with that in mind (&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#exec&#34; &gt;chapter 4&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;p&gt;&lt;code&gt;tests/nagios/run.sh&lt;/code&gt; checks both scripts in Docker, with a stand-in for the&#xA;plugin that says what a file tells it (ok, warn, crit, unknown, hang). Not&#xA;tried: the real plugins of the Monitoring Plugins or Nagios Plugins packages,&#xA;a Nagios or Icinga core, NRPE, Zabbix (its &lt;code&gt;UserParameter&lt;/code&gt; can run&#xA;&lt;code&gt;check_emfor.sh&lt;/code&gt; and use its text), FreeBSD.&lt;/p&gt;&#xA;&lt;h2 id=&#34;running-more-than-one-daemon&#34;&gt;Running more than one daemon&#xA;&lt;/h2&gt;&lt;p&gt;Each store has one writer: the daemon locks &lt;code&gt;&amp;lt;store&amp;gt;.writer&lt;/code&gt;, and a second&#xA;daemon with the same &lt;code&gt;[store] path&lt;/code&gt; exits with status 1. To run two&#xA;daemons on one host (for testing), give them different store paths, DNS&#xA;ports and metrics ports.&lt;/p&gt;&#xA;&lt;h2 id=&#34;upgrading&#34;&gt;Upgrading&#xA;&lt;/h2&gt;&lt;p&gt;Install the daemon and every client library from the same release, in this&#xA;order:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Upgrade the client libraries (and &lt;code&gt;emfor status&lt;/code&gt;) on the host, and restart&#xA;the applications so that they load them: a running application keeps the&#xA;library it started with. A client older than the daemon can fail every&#xA;lookup with &lt;code&gt;INCOMPATIBLE&lt;/code&gt; (the store format is version 2 since 0.4.0, see&#xA;&lt;a class=&#34;link&#34; href=&#34;../06-clients/#store-format-version&#34; &gt;store format version&lt;/a&gt;), and an older&#xA;client can still choose a drained backend in the last tier of an open&#xA;service (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;draining a backend&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Run &lt;code&gt;emfor check-config&lt;/code&gt; with the new binary, before you restart. 0.5.0&#xA;refuses options that 0.4.0 accepted (an &lt;code&gt;interval&lt;/code&gt; above &lt;code&gt;24h&lt;/code&gt;, a&#xA;&lt;code&gt;timeout&lt;/code&gt; below &lt;code&gt;10ms&lt;/code&gt;, and more: &lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/CHANGELOG.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;CHANGELOG.md&lt;/a&gt;).&#xA;With an invalid config, &lt;code&gt;systemctl restart emfor&lt;/code&gt; leaves the daemon down&#xA;(the check before the start fails, and the unit ends &lt;code&gt;failed&lt;/code&gt; until&#xA;&lt;code&gt;systemctl reset-failed emfor&lt;/code&gt;): see&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/docs/deploy/daemon.md#running-under-systemd&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;Restarts&lt;/a&gt; in the daemon guide.&lt;/li&gt;&#xA;&lt;li&gt;Install the daemon and restart it.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;p&gt;Release notes that change what to do: 0.4.0 writes store format version 2&#xA;and encrypts peer messages by default (a group is upgraded host by host with&#xA;&lt;code&gt;encrypt = false&lt;/code&gt;, &lt;a class=&#34;link&#34; href=&#34;#peer-groups&#34; &gt;Peer groups&lt;/a&gt;); 0.5.0 validates the config&#xA;more strictly.&lt;/p&gt;&#xA;&lt;p&gt;After the restart, applications that were already running the new library&#xA;follow the restarted daemon&amp;rsquo;s store without another restart.&lt;/p&gt;&#xA;&lt;h3 id=&#34;what-applications-see-during-a-restart&#34;&gt;What applications see during a restart&#xA;&lt;/h3&gt;&lt;p&gt;A lookup never waits for the daemon: it always gets an answer from the&#xA;store. But around a restart the store cannot tell good backends from bad&#xA;ones for a short time:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;strong&gt;While the daemon is stopped.&lt;/strong&gt; After a clean stop the store is stale at&#xA;once (the heartbeat is 0); after a kill, clients trust the old states for&#xA;up to 3 seconds and then treat the store as stale. In a stale store every&#xA;backend is a candidate: an &lt;code&gt;open&lt;/code&gt; service returns any of them, a &lt;code&gt;closed&lt;/code&gt;&#xA;service fails with &lt;code&gt;ALL_DOWN&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;&lt;strong&gt;For the first 0.2 to 0.4 seconds after the start.&lt;/strong&gt; The daemon marks&#xA;every backend &lt;code&gt;unknown&lt;/code&gt; and checks each one about every 200 ms. Until the&#xA;first results are in, all of them are candidates: &lt;code&gt;open&lt;/code&gt; returns any&#xA;backend, and &lt;code&gt;closed&lt;/code&gt; fails with &lt;code&gt;ALL_DOWN&lt;/code&gt; until one of its backends is&#xA;healthy.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;Measured with the daemon started again at once (after SIGTERM, and after&#xA;SIGKILL): 20 restarts per case, one host in a Debian container, loopback,&#xA;&lt;code&gt;tcp&lt;/code&gt; checks with the default settings, two services of four backends each&#xA;with two backends down, and a client that looks up every millisecond. The&#xA;last wrong answer came about 0.25 seconds after the signal (median; at most&#xA;about 0.4 seconds). In that window half of the &lt;code&gt;open&lt;/code&gt; answers were backends that&#xA;were down, and every &lt;code&gt;closed&lt;/code&gt; lookup failed with &lt;code&gt;ALL_DOWN&lt;/code&gt;. &lt;code&gt;emfor check-config&lt;/code&gt;, which the example systemd unit runs before every start,&#xA;took about 1 ms. When the daemon stayed down for 2 seconds, the window was&#xA;2.2 to 2.4 seconds. A service whose backends are all up is not affected,&#xA;because any backend is then a good answer. Other hardware, networks,&#xA;intervals and checks give other numbers.&lt;/p&gt;&#xA;&lt;p&gt;What to do about it:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;If the host is behind a load balancer, take it out of rotation first, let&#xA;the requests in flight finish, restart, and put it back when &lt;code&gt;emfor status&lt;/code&gt; exits 0 (the daemon is running and every service shown has a&#xA;healthy backend; see &lt;a class=&#34;link&#34; href=&#34;#emfor-status&#34; &gt;emfor status&lt;/a&gt;). Work that does not&#xA;come through the load balancer (batch jobs, queue workers, connection&#xA;pools that reconnect) still looks up during the restart.&lt;/li&gt;&#xA;&lt;li&gt;Services with &lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt;, with &lt;code&gt;failover&lt;/code&gt; and a bad first&#xA;backend, or with backups are the most affected; do this for hosts that&#xA;use them.&lt;/li&gt;&#xA;&lt;li&gt;Install the new packages first and restart afterwards. Do not stop the&#xA;daemon, install and start it: the window is then as long as the install.&lt;/li&gt;&#xA;&lt;li&gt;Restart hosts one at a time.&lt;/li&gt;&#xA;&lt;li&gt;A change of the config alone needs no restart: use a reload, which keeps&#xA;the state of backends whose check did not change (see &lt;a class=&#34;link&#34; href=&#34;../03-configuration/#what-a-reload-changes&#34; &gt;what a reload&#xA;changes&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;p&gt;&lt;strong&gt;From 0.1.x to 0.2.0&lt;/strong&gt;: the daemon is now started with a command. Change&#xA;your service definition before restarting:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;0.1.x&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;0.2.0&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor --config X&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor server --config X&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor --check-config --config X&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor check-config --config X&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The store format and the plugin protocol are unchanged. Replace copies of&#xA;the 0.1.x systemd unit and rc.d script with the new examples. See&#xA;&lt;a class=&#34;link&#34; href=&#34;https://github.com/embark-systems/emfor/blob/v0.6.0/CHANGELOG.md&#34;  target=&#34;_blank&#34; rel=&#34;noopener&#34;&#xA;    &gt;CHANGELOG.md&lt;/a&gt; for everything that changed.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>9. Troubleshooting</title>
            <link>https://embark.systems/en/emfor/09-troubleshooting/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/09-troubleshooting/</guid>
            <description>&lt;p&gt;Start with &lt;code&gt;emfor status&lt;/code&gt;: it shows whether the daemon is running, each&#xA;backend&amp;rsquo;s state, and the reason of the last failed check. Then look at the&#xA;daemon&amp;rsquo;s log (&lt;code&gt;journalctl -u emfor&lt;/code&gt;, or syslog on FreeBSD).&lt;/p&gt;&#xA;&lt;h2 id=&#34;the-daemon-does-not-start&#34;&gt;The daemon does not start&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Message or exit status&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Cause&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Fix&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Exit 2, config errors listed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The config is invalid.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;emfor check-config&lt;/code&gt; lists every error with its TOML path.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Exit 2, usage printed&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No command given (0.1.x style).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Use &lt;code&gt;emfor server&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;--check-config was replaced by &amp;quot;emfor check-config&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A 0.1.x command line.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See &lt;a class=&#34;link&#34; href=&#34;../08-operations/#upgrading&#34; &gt;upgrading&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Exit 1, names an address&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A &lt;code&gt;[dns]&lt;/code&gt; or &lt;code&gt;[metrics]&lt;/code&gt; listen address is in use, or needs privileges (port 53).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Free the port, or choose another one.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Exit 1, names &lt;code&gt;&amp;lt;store&amp;gt;.writer&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Another emfor already runs with this store.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Stop it, or use another &lt;code&gt;[store] path&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Exit 1, cannot open the store&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The directory is missing or not writable for the daemon&amp;rsquo;s user, or the lock file belongs to another user.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Create the directory with the right owner; see &lt;a class=&#34;link&#34; href=&#34;#lock-file-owned-by-another-user&#34; &gt;lock file owned by another user&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;plugins.&amp;quot;x&amp;quot;.command: ...&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A plugin failed to start or to answer the handshake.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Run the command by hand; check the path and its permissions.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;every-lookup-fails&#34;&gt;Every lookup fails&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Error&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Cause&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Fix&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;, file not found&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The daemon has never run, or &lt;code&gt;EMFOR_STORE&lt;/code&gt; and &lt;code&gt;[store] path&lt;/code&gt; differ.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Start the daemon; set &lt;code&gt;EMFOR_STORE&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;, permission denied&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The application user cannot read the store or write &lt;code&gt;&amp;lt;store&amp;gt;-lock&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See &lt;a class=&#34;link&#34; href=&#34;../08-operations/#who-can-use-the-store&#34; &gt;who can use the store&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;, mentions the LMDB format or the release&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;libemfor&lt;/code&gt; is from another emfor release than the daemon.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install both from the same release.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;, mentions &lt;code&gt;EMFOR_MAX_STALENESS&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The variable has an invalid value.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Use a whole number with &lt;code&gt;ms&lt;/code&gt; or &lt;code&gt;s&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;UNAVAILABLE&lt;/code&gt;, says &lt;code&gt;truncated&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store file is shorter than LMDB records it: it was changed in place, or lost its tail in a power loss.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Restart the daemon, which moves it to &lt;code&gt;&amp;lt;store&amp;gt;.corrupt&lt;/code&gt; and starts a new store; never change the store in place (see &lt;a class=&#34;link&#34; href=&#34;../08-operations/#the-store-on-disk&#34; &gt;the store on disk&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;INCOMPATIBLE&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The store has another format version.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Install the client from the daemon&amp;rsquo;s release.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NOT_FOUND&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No service of that name in the running config.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Check the spelling, and that the reload was applied (&lt;code&gt;emfor status&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;a-service-is-all-down&#34;&gt;A service is ALL DOWN&#xA;&lt;/h2&gt;&lt;ol&gt;&#xA;&lt;li&gt;Look at the reasons in &lt;code&gt;emfor status --service &amp;lt;name&amp;gt;&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;If every backend shows the same local-looking reason (TLS, a CA file,&#xA;&lt;code&gt;connect: Network is unreachable&lt;/code&gt;), the problem is likely on the&#xA;checking host, not the backends.&lt;/li&gt;&#xA;&lt;li&gt;If the daemon line says &lt;code&gt;stale&lt;/code&gt; or &lt;code&gt;stopped&lt;/code&gt;, the backends are not&#xA;down: the daemon is. Every &lt;code&gt;closed&lt;/code&gt; service fails until it is running&#xA;again. Restart it and look at its log.&lt;/li&gt;&#xA;&lt;li&gt;Check the check itself: run &lt;code&gt;curl&lt;/code&gt; or &lt;code&gt;nc&lt;/code&gt; from the host to the&#xA;backend with the same path, port and options.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h2 id=&#34;a-backend-flaps&#34;&gt;A backend flaps&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;FLAPPING&lt;/code&gt; in &lt;code&gt;emfor status&lt;/code&gt; or the &lt;code&gt;EmforBackendFlapping&lt;/code&gt; alert means&#xA;frequent changes between healthy and unhealthy. Common causes: a &lt;code&gt;timeout&lt;/code&gt;&#xA;close to the backend&amp;rsquo;s normal response time, or an overloaded backend.&#xA;Raise &lt;code&gt;timeout&lt;/code&gt; (up to &lt;code&gt;interval&lt;/code&gt;), raise &lt;code&gt;fall&lt;/code&gt; and &lt;code&gt;rise&lt;/code&gt;, or check the&#xA;backend. &lt;code&gt;emfor status --history --service &amp;lt;name&amp;gt;&lt;/code&gt; shows when and why it&#xA;changed.&lt;/p&gt;&#xA;&lt;h2 id=&#34;checks-end-in-error&#34;&gt;Checks end in error&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;error&lt;/code&gt; results do not change a backend&amp;rsquo;s state, but a backend that has&#xA;only errors for &lt;code&gt;fall × interval&lt;/code&gt; returns to &lt;code&gt;unknown&lt;/code&gt;. Typical reasons:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Reason starts with&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Cause&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;plugin &amp;lt;name&amp;gt; unavailable&lt;/code&gt;, &lt;code&gt;plugin exited&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The plugin crashed, stopped answering, or is restarting; see the log for its exit status or &lt;code&gt;stopped answering&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;plugin &amp;lt;name&amp;gt; busy: it is not reading its requests&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The plugin stopped reading its stdin and 1024 requests are waiting for it. It is restarted once 3 requests in a row go unanswered; if this lasts, the plugin is stuck outside its request loop.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;plugin did not answer within timeout&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The plugin is too slow or handles checks one at a time.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;password_file: &lt;/code&gt;, &lt;code&gt;ca_file: &lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The file cannot be read by the daemon&amp;rsquo;s user.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;exit code N&lt;/code&gt; (exec, N not 1 or 2)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The script failed in an unexpected way.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;A change to &lt;code&gt;unknown&lt;/code&gt; with reason &lt;code&gt;check task restarted&lt;/code&gt; in&#xA;&lt;code&gt;emfor status --history&lt;/code&gt; is not a check result: the backend&amp;rsquo;s check task&#xA;stopped on a bug and was started again, and the log has an error line&#xA;&lt;code&gt;check task of backend ... stopped&lt;/code&gt;. Please report it with that line.&#xA;A backend that was already &lt;code&gt;unknown&lt;/code&gt; gets no history entry, but&#xA;&lt;code&gt;emfor status&lt;/code&gt; shows the same reason until its next check.&lt;/p&gt;&#xA;&lt;h2 id=&#34;dns-does-not-resolve&#34;&gt;DNS does not resolve&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Symptom&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Cause&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Fix&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;dig @127.0.0.1 -p 5300&lt;/code&gt; gets no answer&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No &lt;code&gt;[dns]&lt;/code&gt; table, or another listen address.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Check &lt;code&gt;emfor status&lt;/code&gt; (the daemon line shows &lt;code&gt;dns ...&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;REFUSED&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The name is outside the zone, or the query class is not &lt;code&gt;IN&lt;/code&gt; (&lt;code&gt;dig ... CH&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Name the service under the zone, or change &lt;code&gt;zone&lt;/code&gt;; query class &lt;code&gt;IN&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;NXDOMAIN&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No service of that name.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Check the name.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;SERVFAIL&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No healthy backend and &lt;code&gt;on_all_down = &amp;quot;closed&amp;quot;&lt;/code&gt;; or a validating unbound without &lt;code&gt;domain-insecure&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See &lt;a class=&#34;link&#34; href=&#34;#a-service-is-all-down&#34; &gt;ALL DOWN&lt;/a&gt;; fix the unbound config.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Direct query works, &lt;code&gt;getent hosts&lt;/code&gt; does not&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The host resolver does not forward the zone.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Follow &lt;a class=&#34;link&#34; href=&#34;../07-dns/#forwarding-from-the-host-resolver&#34; &gt;chapter 7&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A failed backend is still returned&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Resolver or application caching.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See &lt;a class=&#34;link&#34; href=&#34;../07-dns/#caching&#34; &gt;caching&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;h2 id=&#34;the-store-looks-stale-while-the-daemon-runs&#34;&gt;The store looks stale while the daemon runs&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor status&lt;/code&gt; says &lt;code&gt;stale&lt;/code&gt;, or clients act as if every backend were down,&#xA;while the daemon process is alive:&lt;/p&gt;&#xA;&lt;ul&gt;&#xA;&lt;li&gt;&lt;code&gt;emfor_store_commit_failures_total&lt;/code&gt; rising and &lt;code&gt;MDB_MAP_FULL&lt;/code&gt; in the log:&#xA;the store is full. Raise &lt;code&gt;[store] map_size&lt;/code&gt; and restart.&lt;/li&gt;&#xA;&lt;li&gt;Commits failing for other reasons: the log names them.&lt;/li&gt;&#xA;&lt;li&gt;The system clock jumped forward: the heartbeat looks old until the next&#xA;commit, about a second later.&lt;/li&gt;&#xA;&lt;li&gt;A local user holds the store&amp;rsquo;s write lock (a writable lock file); see&#xA;&lt;a class=&#34;link&#34; href=&#34;../08-operations/#who-can-use-the-store&#34; &gt;who can use the store&lt;/a&gt;.&lt;/li&gt;&#xA;&lt;/ul&gt;&#xA;&lt;h2 id=&#34;lock-file-owned-by-another-user&#34;&gt;Lock file owned by another user&#xA;&lt;/h2&gt;&lt;p&gt;If the lock file &lt;code&gt;&amp;lt;store&amp;gt;-lock&lt;/code&gt; was deleted while applications ran, the&#xA;first process to open the store recreated it with its own owner, and the&#xA;daemon (another user) may fail to open the store. Stop the applications or&#xA;fix the file&amp;rsquo;s owner and mode (&lt;code&gt;chown emfor&lt;/code&gt;, &lt;code&gt;chmod&lt;/code&gt; to&#xA;&lt;code&gt;lock_file_mode&lt;/code&gt;), then start the daemon. Never delete the lock file alone.&lt;/p&gt;&#xA;&lt;h2 id=&#34;a-reload-did-not-take-effect&#34;&gt;A reload did not take effect&#xA;&lt;/h2&gt;&lt;p&gt;A reload is all or nothing. Look for &lt;code&gt;reload rejected&lt;/code&gt; and the error lines&#xA;in the log, or run &lt;code&gt;emfor check-config&lt;/code&gt;. &lt;code&gt;[store]&lt;/code&gt;, &lt;code&gt;[dns] listen&lt;/code&gt;,&#xA;&lt;code&gt;[metrics] listen&lt;/code&gt; and &lt;code&gt;[peers] listen&lt;/code&gt; changes always need a restart.&lt;/p&gt;&#xA;&lt;h2 id=&#34;peers-drop-each-others-messages-after-a-restart&#34;&gt;Peers drop each other&amp;rsquo;s messages after a restart&#xA;&lt;/h2&gt;&lt;p&gt;&lt;code&gt;emfor_peer_messages_rejected_total{reason=&amp;quot;auth&amp;quot;}&lt;/code&gt; rises and&#xA;&lt;code&gt;emfor_peer_up&lt;/code&gt; drops to 0 for a while right after a daemon restarted: the&#xA;peer secret was being changed, and the restarted daemon knows only the new&#xA;one. It stops once every member&amp;rsquo;s &lt;code&gt;secret_file&lt;/code&gt; holds the new secret. Set&#xA;&lt;code&gt;[peers] previous_secret_file&lt;/code&gt; to avoid it (see&#xA;&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;&lt;code&gt;[peers]&lt;/code&gt;&lt;/a&gt;). For other reasons see&#xA;&lt;a class=&#34;link&#34; href=&#34;../08-operations/#peer-groups&#34; &gt;Peer groups&lt;/a&gt;.&lt;/p&gt;&#xA;</description>
        </item><item>
            <title>10. Moving from HAProxy or Consul</title>
            <link>https://embark.systems/en/emfor/10-migrating/</link>
            <pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate>
            <guid>https://embark.systems/en/emfor/10-migrating/</guid>
            <description>&lt;p&gt;This chapter is for people who already run HAProxy or Consul to keep&#xA;applications away from broken backends, and want to see what the same job&#xA;looks like in emfor. It translates a real-looking configuration of each&#xA;into &lt;code&gt;emfor.toml&lt;/code&gt;, says what has no equivalent, and ends with a way to try&#xA;emfor next to what you have, without touching traffic.&lt;/p&gt;&#xA;&lt;p&gt;Read &lt;a class=&#34;link&#34; href=&#34;../01-concepts/&#34; &gt;chapter 1&lt;/a&gt; first: the picture is different enough&#xA;that a line-by-line translation alone will mislead you.&lt;/p&gt;&#xA;&lt;h2 id=&#34;is-emfor-the-right-replacement&#34;&gt;Is emfor the right replacement?&#xA;&lt;/h2&gt;&lt;p&gt;emfor does one thing of what HAProxy and Consul do: it checks a fixed list of&#xA;backends from the host of the application and tells the application which&#xA;one to use. It is not in the path of the traffic, and it has no central&#xA;server.&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;HAProxy&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Consul&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;emfor&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Where the traffic goes&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Through HAProxy (one more hop)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Straight to the backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Straight to the backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Who decides which backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;HAProxy, for every connection&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The application, from a DNS answer&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The application, from the local store (client library or &lt;code&gt;emfor lookup&lt;/code&gt;) or from a DNS answer&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Where the checks run&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;In HAProxy&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;In the Consul agent of the node that runs the service&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;On every application host, against every backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Which backends exist&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The config file&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Services register themselves&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The config file of each host&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A central server&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Yes (servers with a quorum)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;So emfor fits when the backends are a known list of addresses (virtual&#xA;machines, bare metal, databases, AAA servers), and the application can ask&#xA;emfor where to connect, either with the client library, with a shell&#xA;command, or through DNS.&lt;/p&gt;&#xA;&lt;p&gt;It does not replace what a proxy does with the traffic itself: least&#xA;connections, retries, queueing, TLS termination, routing by URL or header,&#xA;connection limits, and passive health checks (marking a server down because&#xA;real requests failed). If you need those, keep HAProxy and see&#xA;&lt;a class=&#34;link&#34; href=&#34;#what-does-not-carry-over&#34; &gt;what does not carry over&lt;/a&gt;.&lt;/p&gt;&#xA;&lt;h2 id=&#34;from-haproxy&#34;&gt;From HAProxy&#xA;&lt;/h2&gt;&lt;p&gt;An HAProxy backend like this one:&lt;/p&gt;&#xA;&lt;pre tabindex=&#34;0&#34;&gt;&lt;code&gt;backend api&#xA;    balance roundrobin&#xA;    option httpchk GET /healthz&#xA;    http-check expect status 200&#xA;    default-server inter 2s fall 3 rise 2&#xA;    server api1 10.0.0.11:8080 check&#xA;    server api2 10.0.0.12:8080 check weight 3&#xA;    server api3 10.0.1.11:8080 check backup slowstart 30s&#xA;&lt;/code&gt;&lt;/pre&gt;&lt;p&gt;becomes this service in &lt;code&gt;emfor.toml&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;balance&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;weighted&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;2s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;fall&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;rise&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;2&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;slow_start&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;30s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;,&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;weight&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  { &lt;span style=&#34;color:#a6e22e&#34;&gt;addr&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.1.11:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#a6e22e&#34;&gt;backup&lt;/span&gt; = &lt;span style=&#34;color:#66d9ef&#34;&gt;true&lt;/span&gt; },&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.svc.emfor&amp;#34;&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;expect_status&lt;/span&gt; = [&lt;span style=&#34;color:#ae81ff&#34;&gt;200&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;Line by line:&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;HAProxy&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;emfor&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Notes&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;backend api&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;[services.&amp;quot;api.svc.emfor&amp;quot;]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The service name is host-name-like and is also the DNS name (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#servicesname&#34; &gt;chapter 3&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;server api1 10.0.0.11:8080&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;an entry of &lt;code&gt;backends&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;IP addresses only, never host names: emfor does not depend on DNS to find its backends.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check = &amp;quot;http&amp;quot;&lt;/code&gt; (or &lt;code&gt;&amp;quot;tcp&amp;quot;&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;One &lt;code&gt;check&lt;/code&gt; for the whole service.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;inter 2s&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval = &amp;quot;2s&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Both default to 2 seconds.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fall 3&lt;/code&gt; / &lt;code&gt;rise 2&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fall = 3&lt;/code&gt; / &lt;code&gt;rise = 2&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Both default to 3 and 2 (&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#rise-fall-and-errors&#34; &gt;rise, fall and errors&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;option httpchk GET /healthz&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;path = &amp;quot;/healthz&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor always sends &lt;code&gt;GET&lt;/code&gt;. HAProxy without arguments sends &lt;code&gt;OPTIONS /&lt;/code&gt;; emfor sends &lt;code&gt;GET /&lt;/code&gt; unless you set &lt;code&gt;path&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#http&#34; &gt;http check&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;http-check expect status 200&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;expect_status = [200]&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Without it, any status from 200 to 399 is &lt;code&gt;up&lt;/code&gt;. HAProxy&amp;rsquo;s default accepts 2xx and 3xx too. Ranges are written &lt;code&gt;&amp;quot;200-299&amp;quot;&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;http-check send ... hdr Host x&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;host_header = &amp;quot;x&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Only the &lt;code&gt;Host&lt;/code&gt; header can be set.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;weight 3&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;{ addr = &amp;quot;...&amp;quot;, weight = 3 }&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;From 1 to 100, and only used by &lt;code&gt;balance = &amp;quot;weighted&amp;quot;&lt;/code&gt; (and by &lt;code&gt;sticky&lt;/code&gt; with a key). HAProxy weights go to 256, and 0 takes a server out: emfor refuses &lt;code&gt;weight = 0&lt;/code&gt; and &lt;code&gt;weight = 256&lt;/code&gt; (&lt;code&gt;weight must be from 1 to 100&lt;/code&gt;), so rescale the weights to 1 to 100 and use &lt;code&gt;drain = true&lt;/code&gt; for 0 (see the table at the end).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;backup&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;backup = true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A backup is used only while no other backend passes its checks (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#backup-backends&#34; &gt;backup backends&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;slowstart 30s&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;slow_start = &amp;quot;30s&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A recovered backend gets a growing share of the traffic, in ten steps (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#slow-start&#34; &gt;slow start&lt;/a&gt;). Set on the service, not on a backend.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance roundrobin&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance = &amp;quot;round-robin&amp;quot;&lt;/code&gt; or &lt;code&gt;&amp;quot;weighted&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;HAProxy&amp;rsquo;s round robin takes the weights into account; emfor&amp;rsquo;s &lt;code&gt;round-robin&lt;/code&gt; does not. Use &lt;code&gt;weighted&lt;/code&gt; when you use weights.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance first&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance = &amp;quot;failover&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The first backend that is healthy, in config order. Without a &lt;code&gt;maxconn&lt;/code&gt;, HAProxy&amp;rsquo;s &lt;code&gt;first&lt;/code&gt; does the same; with one it moves on when the first server is full, and emfor has no connection limit.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance source&lt;/code&gt;, &lt;code&gt;balance uri&lt;/code&gt;, &lt;code&gt;balance hdr&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance = &amp;quot;sticky&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The application passes the key: &lt;code&gt;lookup(name, key)&lt;/code&gt;. The same key gets the same backend while it is healthy (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;balance leastconn&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor does not see the connections; see below.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check port 8081&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor checks the backend&amp;rsquo;s own address and port.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;agent-check&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;a plugin or an &lt;code&gt;exec&lt;/code&gt; check&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;There is no agent protocol; write the check as a &lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#exec&#34; &gt;plugin or an &lt;code&gt;exec&lt;/code&gt; check&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;maxconn&lt;/code&gt;, queues, &lt;code&gt;observe layer7&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See below.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;The application then connects to what emfor tells it instead of to HAProxy.&#xA;Pick the way that suits it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-console&#34; data-lang=&#34;console&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ emfor lookup api.svc.emfor&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;10.0.0.11:8080&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;$ curl &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http://&lt;/span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;$(&lt;/span&gt;emfor lookup api.svc.emfor&lt;span style=&#34;color:#66d9ef&#34;&gt;)&lt;/span&gt;&lt;span style=&#34;color:#e6db74&#34;&gt;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;or the client library of its language&#xA;(&lt;a class=&#34;link&#34; href=&#34;../06-clients/&#34; &gt;chapter 6&lt;/a&gt;), or the DNS name &lt;code&gt;api.svc.emfor&lt;/code&gt;: for that add&#xA;a &lt;code&gt;[dns]&lt;/code&gt; table to the config (the example above has none, and the DNS&#xA;server runs only when the table is present) and forward the zone from the&#xA;host resolver (&lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;). Limits to know about: an A or AAAA&#xA;answer carries no port, so the application keeps the port itself (or asks&#xA;for SRV); a runtime or resolver that caches answers delays a change by its&#xA;cache time (&lt;a class=&#34;link&#34; href=&#34;../07-dns/#caching&#34; &gt;caching&lt;/a&gt;); and &lt;code&gt;emfor lookup&lt;/code&gt; is a command&#xA;that prints one backend per process, so a program that connects to&#xA;&lt;code&gt;$(emfor lookup ...)&lt;/code&gt; needs a small wrapper that runs it for each connection&#xA;(&lt;a class=&#34;link&#34; href=&#34;../06-clients/#from-the-shell-emfor-lookup&#34; &gt;chapter 6&lt;/a&gt;).&lt;/p&gt;&#xA;&lt;h3 id=&#34;trying-it-next-to-haproxy&#34;&gt;Trying it next to HAProxy&#xA;&lt;/h3&gt;&lt;p&gt;emfor does not touch the traffic, so you can run it beside HAProxy first:&lt;/p&gt;&#xA;&lt;ol&gt;&#xA;&lt;li&gt;Write the services into &lt;code&gt;emfor.toml&lt;/code&gt; (with the deb or rpm, into a file of&#xA;&lt;code&gt;/etc/emfor/conf.d/&lt;/code&gt;, which the packaged main config reads) and check the&#xA;config with &lt;code&gt;emfor check-config&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Start the daemon (the deb has started it at install; after a change to&#xA;a file, &lt;code&gt;systemctl reload emfor&lt;/code&gt;) and compare &lt;code&gt;emfor status&lt;/code&gt; with the HAProxy stats page:&#xA;the same backends should be healthy and unhealthy, and the reasons in&#xA;&lt;code&gt;emfor status&lt;/code&gt; say why a check failed.&lt;/li&gt;&#xA;&lt;li&gt;Look at what an application would get, and why, with&#xA;&lt;code&gt;emfor lookup api.svc.emfor --explain&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../06-clients/#from-the-shell-emfor-lookup&#34; &gt;chapter 6&lt;/a&gt;).&lt;/li&gt;&#xA;&lt;li&gt;Move one application at a time to emfor, watching the&#xA;&lt;a class=&#34;link&#34; href=&#34;../08-operations/#metrics&#34; &gt;metrics&lt;/a&gt; and &lt;code&gt;emfor status --watch&lt;/code&gt;.&lt;/li&gt;&#xA;&lt;li&gt;Remove a backend from HAProxy&amp;rsquo;s configuration only when no application&#xA;uses it any more.&lt;/li&gt;&#xA;&lt;/ol&gt;&#xA;&lt;h3 id=&#34;keeping-haproxy-and-letting-emfor-feed-it&#34;&gt;Keeping HAProxy and letting emfor feed it&#xA;&lt;/h3&gt;&lt;p&gt;You do not have to move the applications. HAProxy can keep the traffic, with&#xA;its retries, queues, TLS and routing, and take its list of servers from&#xA;emfor&amp;rsquo;s SRV answers, so that emfor&amp;rsquo;s checks decide who is in it&#xA;(&lt;a class=&#34;link&#34; href=&#34;../07-dns-recipes/#using-it-with-haproxy&#34; &gt;HAProxy in chapter 7&lt;/a&gt; has the&#xA;configuration: a &lt;code&gt;server-template&lt;/code&gt; and a &lt;code&gt;resolvers&lt;/code&gt; section that points at&#xA;emfor&amp;rsquo;s DNS server, so the config needs a &lt;code&gt;[dns]&lt;/code&gt; table and the services&#xA;must be named under its zone). What you get: any emfor check (for example a&#xA;database plugin) decides the health instead of &lt;code&gt;check&lt;/code&gt;, &lt;code&gt;drain = true&lt;/code&gt; takes a&#xA;server out with a reload of emfor and no reload of HAProxy, and on every host&#xA;a &lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;peer group&lt;/a&gt; can vote on a backend.&lt;/p&gt;&#xA;&lt;p&gt;What does not carry over, as tried with HAProxy 3.4.7 and 2.8.31: HAProxy&#xA;ignores the priorities and weights of the answer, so &lt;code&gt;failover&lt;/code&gt;, &lt;code&gt;weighted&lt;/code&gt;,&#xA;and &lt;code&gt;backup = true&lt;/code&gt; do nothing there (keep HAProxy&amp;rsquo;s own &lt;code&gt;backup&lt;/code&gt;&#xA;and &lt;code&gt;weight&lt;/code&gt;); the slots of the template limit the number of backends; and&#xA;after a &lt;code&gt;SERVFAIL&lt;/code&gt; HAProxy drops its servers after &lt;code&gt;hold other&lt;/code&gt; (30 seconds by&#xA;default) and answers 503. Use one name server per HAProxy, the emfor of the&#xA;same host. Not tried: more than 27 backends over UDP, and long-running use.&lt;/p&gt;&#xA;&lt;h2 id=&#34;from-consul&#34;&gt;From Consul&#xA;&lt;/h2&gt;&lt;p&gt;A Consul service with an HTTP check, as the agent of the node that runs it&#xA;registers it:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-hcl&#34; data-lang=&#34;hcl&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#66d9ef&#34;&gt;service&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  name &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  port &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;8080&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  &lt;span style=&#34;color:#66d9ef&#34;&gt;check&lt;/span&gt; {&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    http                     &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http://localhost:8080/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    interval                 &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    timeout                  &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;1s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    success_before_passing   &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;2&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;    failures_before_critical &lt;span style=&#34;color:#f92672&#34;&gt;=&lt;/span&gt; &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;  }&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;}&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The same job in emfor is a service that lists the nodes, and checks them&#xA;from the host that asks:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.service.consul&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;interval&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;timeout&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;1s&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;rise&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;2&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;fall&lt;/span&gt; = &lt;span style=&#34;color:#ae81ff&#34;&gt;3&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.11:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.12:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.1.11:8080&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;api.service.consul&amp;#34;&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;expect_status&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;200-299&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;dns&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;zone&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;service.consul&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The service is named &lt;code&gt;api.service.consul&lt;/code&gt; and the zone is &lt;code&gt;service.consul&lt;/code&gt;,&#xA;so a program that looks up &lt;code&gt;api.service.consul&lt;/code&gt; keeps working once the host&#xA;resolver forwards that zone to emfor instead of to Consul (&lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;).&#xA;(You can also keep emfor&amp;rsquo;s own zone, &lt;code&gt;svc.emfor&lt;/code&gt;, and change the names.)&lt;/p&gt;&#xA;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Consul&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;emfor&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;Notes&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;service { name, port }&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;a service and its &lt;code&gt;backends&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;There is no registration: the list of backends is in the config of each host, written by whatever installs the config (Ansible, Puppet, a template). Adding a node means changing the files and reloading (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#what-a-reload-changes&#34; &gt;what a reload changes&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check { http = ... }&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check = &amp;quot;http&amp;quot;&lt;/code&gt; and &lt;code&gt;check_options&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The URL is split: the address is the backend&amp;rsquo;s, the path is &lt;code&gt;path&lt;/code&gt;. Consul checks usually run on the node itself (&lt;code&gt;localhost&lt;/code&gt;); emfor connects to the backend&amp;rsquo;s address from the application host, so a firewall must allow it.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check { tcp = ... }&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check = &amp;quot;tcp&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check { grpc = ... }&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;the &lt;code&gt;grpc&lt;/code&gt; plugin of &lt;code&gt;emfor-check&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See &lt;a class=&#34;link&#34; href=&#34;../04-health-checks/#plugins-and-emfor-check&#34; &gt;plugins&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check { args = [...] }&lt;/code&gt; (script)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;check = &amp;quot;exec&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;interval&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consul has no default; emfor&amp;rsquo;s is 2 seconds.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;timeout&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;timeout&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consul&amp;rsquo;s default for HTTP and TCP is 10 seconds; emfor&amp;rsquo;s is 1 second and cannot exceed &lt;code&gt;interval&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;success_before_passing&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;rise&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consul&amp;rsquo;s default is 0, where the first result decides. emfor refuses &lt;code&gt;rise = 0&lt;/code&gt; (&lt;code&gt;rise must be at least 1&lt;/code&gt;): &lt;code&gt;rise = 1&lt;/code&gt; is the same, and emfor&amp;rsquo;s default is 2.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;failures_before_critical&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;fall&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Consul&amp;rsquo;s default is 0, where the first result decides. emfor refuses &lt;code&gt;fall = 0&lt;/code&gt; (&lt;code&gt;fall must be at least 1&lt;/code&gt;): &lt;code&gt;fall = 1&lt;/code&gt; is the same, and emfor&amp;rsquo;s default is 3.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;method&lt;/code&gt;, &lt;code&gt;header&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor sends &lt;code&gt;GET&lt;/code&gt;; of the headers only &lt;code&gt;Host&lt;/code&gt; can be set (&lt;code&gt;host_header&lt;/code&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tls_skip_verify = true&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;tls_verify = false&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Together with &lt;code&gt;scheme = &amp;quot;https&amp;quot;&lt;/code&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Status &lt;code&gt;passing&lt;/code&gt;, &lt;code&gt;warning&lt;/code&gt;, &lt;code&gt;critical&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;healthy&lt;/code&gt;, &lt;code&gt;unhealthy&lt;/code&gt;, &lt;code&gt;unknown&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A Consul HTTP check is &lt;code&gt;passing&lt;/code&gt; for 200 to 299, &lt;code&gt;warning&lt;/code&gt; for 429 and &lt;code&gt;critical&lt;/code&gt; otherwise; emfor has no &lt;code&gt;warning&lt;/code&gt;: set &lt;code&gt;expect_status&lt;/code&gt; to what you accept.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;status&lt;/code&gt; (initial &lt;code&gt;critical&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;starts &lt;code&gt;unknown&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A backend becomes &lt;code&gt;healthy&lt;/code&gt; after &lt;code&gt;rise&lt;/code&gt; passes. Until then an &lt;code&gt;open&lt;/code&gt; service still returns an address (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#when-no-backend-is-healthy&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;consul maint -enable&lt;/code&gt;, &lt;code&gt;enable_maintenance&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;drain = true&lt;/code&gt; on the backend&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Then reload; see &lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;draining a backend&lt;/a&gt;.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;deregister_critical_service_after&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The list is static; nothing deregisters a backend.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;TTL and alias checks&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;none&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor only runs its own checks.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;api.service.consul&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;the service name, with &lt;code&gt;[dns] zone&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;See above.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;_api._tcp.service.consul&lt;/code&gt; (SRV)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;_&amp;lt;anything&amp;gt;._tcp.&amp;lt;service name&amp;gt;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;emfor&amp;rsquo;s SRV owner names contain the whole service name, for example &lt;code&gt;_http._tcp.api.service.consul&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../07-dns/&#34; &gt;chapter 7&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;tags, datacenters&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;site&lt;/code&gt; and &lt;code&gt;balance = &amp;quot;locality&amp;quot;&lt;/code&gt;&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A backend gets a &lt;code&gt;site&lt;/code&gt;, and a host prefers the backends of its own site (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Consul reads a directory of service files, one per service. emfor can do&#xA;the same with &lt;code&gt;include_dir&lt;/code&gt;: put &lt;code&gt;include_dir = &amp;quot;conf.d&amp;quot;&lt;/code&gt; as the first line of&#xA;&lt;code&gt;emfor.toml&lt;/code&gt;, before any table (after &lt;code&gt;[dns]&lt;/code&gt; it is an error,&#xA;&lt;code&gt;dns.include_dir: unknown key&lt;/code&gt;), create the directory, and put one file per&#xA;service in it, for example &lt;code&gt;/etc/emfor/conf.d/10-web.toml&lt;/code&gt;:&lt;/p&gt;&#xA;&lt;div class=&#34;highlight&#34;&gt;&lt;pre tabindex=&#34;0&#34; style=&#34;color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;&#34;&gt;&lt;code class=&#34;language-toml&#34; data-lang=&#34;toml&#34;&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;web.svc.emfor&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;check&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;http&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;backends&lt;/span&gt; = [&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.21:8080&amp;#34;&lt;/span&gt;, &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;10.0.0.22:8080&amp;#34;&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;[&lt;span style=&#34;color:#a6e22e&#34;&gt;services&lt;/span&gt;.&lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;web.svc.emfor&amp;#34;&lt;/span&gt;.&lt;span style=&#34;color:#a6e22e&#34;&gt;check_options&lt;/span&gt;]&#xA;&lt;/span&gt;&lt;/span&gt;&lt;span style=&#34;display:flex;&#34;&gt;&lt;span&gt;&lt;span style=&#34;color:#a6e22e&#34;&gt;path&lt;/span&gt; = &lt;span style=&#34;color:#e6db74&#34;&gt;&amp;#34;/healthz&amp;#34;&lt;/span&gt;&#xA;&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;&lt;p&gt;The directory and its files must not be writable by group or others and&#xA;must be owned by root or the user that runs emfor&#xA;(&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#splitting-the-config-over-a-directory&#34; &gt;permissions&lt;/a&gt;).&#xA;&lt;code&gt;emfor check-config&lt;/code&gt; lists the files it read, and a reload re-reads them&#xA;all (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#splitting-the-config-over-a-directory&#34; &gt;chapter 3&lt;/a&gt;).&#xA;Unlike Consul, emfor does not register anything: the file is the list of&#xA;backends.&lt;/p&gt;&#xA;&lt;h3 id=&#34;trying-it-next-to-consul&#34;&gt;Trying it next to Consul&#xA;&lt;/h3&gt;&lt;p&gt;The steps are those of the HAProxy section: write the config, run&#xA;&lt;code&gt;emfor check-config&lt;/code&gt;, start the daemon beside the Consul agent, compare the&#xA;health &lt;code&gt;emfor status&lt;/code&gt; shows with what &lt;code&gt;consul catalog&lt;/code&gt; and the Consul UI&#xA;show, then move one application at a time. A difference between the two&#xA;is the useful part: it means the backend looks different from where the&#xA;application is than from where the Consul agent is.&lt;/p&gt;&#xA;&lt;h2 id=&#34;what-does-not-carry-over&#34;&gt;What does not carry over&#xA;&lt;/h2&gt;&lt;table&gt;&#xA;&#x9;&lt;thead&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;You use&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;th&gt;In emfor&lt;/th&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/thead&gt;&#xA;&#x9;&lt;tbody&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Least connections, connection limits, queues&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No. emfor is not in the data path and does not see connections. Keep HAProxy for this.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Retries and timeouts of the proxy&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;The application retries. The client returns one address per lookup, so a retry gets the same backend until its next check fails, which takes about one &lt;code&gt;interval&lt;/code&gt;, if the service is &lt;code&gt;sticky&lt;/code&gt; or &lt;code&gt;failover&lt;/code&gt;; with &lt;code&gt;round-robin&lt;/code&gt; a retry in the same process takes the next candidate. A backend is left out of the candidates at its first failed check, not only once &lt;code&gt;fall&lt;/code&gt; checks have marked it down (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#a-backend-whose-checks-start-failing&#34; &gt;chapter 5&lt;/a&gt;).&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Passive health checks (&lt;code&gt;observe layer7&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No: only active checks. A failing backend is noticed by the next checks; with a &lt;a class=&#34;link&#34; href=&#34;../03-configuration/#peers&#34; &gt;peer group&lt;/a&gt;, when one member sees a backend go from &lt;code&gt;healthy&lt;/code&gt; to &lt;code&gt;unhealthy&lt;/code&gt; (after &lt;code&gt;fall&lt;/code&gt; failed checks) it tells the others, which recheck it at once and every 200ms, up to &lt;code&gt;fall&lt;/code&gt; checks.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Taking a backend out for maintenance (&lt;code&gt;disabled&lt;/code&gt;, &lt;code&gt;maint&lt;/code&gt;, &lt;code&gt;drain&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;&lt;code&gt;drain = true&lt;/code&gt; on the backend, then reload (&lt;a class=&#34;link&#34; href=&#34;../05-backend-selection/#draining-a-backend&#34; &gt;chapter 5&lt;/a&gt;): it is never chosen and its checks go on. &lt;code&gt;weight = 0&lt;/code&gt; is refused (&lt;code&gt;weight must be from 1 to 100&lt;/code&gt;), and there is no run-time command (&lt;code&gt;set server ... state drain&lt;/code&gt;): the config is the only way.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Routing by URL, header or cookie, TLS termination&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Registration, tags and key-value store of Consul&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;No. The list of backends is config.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Several configuration files (&lt;code&gt;haproxy -f dir&lt;/code&gt;)&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A main file and a directory of fragments with &lt;code&gt;include_dir&lt;/code&gt; (&lt;a class=&#34;link&#34; href=&#34;../03-configuration/#splitting-the-config-over-a-directory&#34; &gt;chapter 3&lt;/a&gt;). A fragment cannot include another directory, and a list such as &lt;code&gt;backends&lt;/code&gt; is replaced as a whole, not merged.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&#x9;&#x9;&lt;tr&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;A central view of all checks&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&#x9;&#x9;&lt;td&gt;Each host has its own. Use the &lt;a class=&#34;link&#34; href=&#34;../08-operations/#metrics&#34; &gt;metrics&lt;/a&gt; with Prometheus.&lt;/td&gt;&#xA;&#x9;&#x9;&#x9;&lt;/tr&gt;&#xA;&#x9;&lt;/tbody&gt;&#xA;&lt;/table&gt;&#xA;&lt;p&gt;Where a row says &amp;ldquo;keep HAProxy&amp;rdquo;, it is also fine to use both: emfor chooses&#xA;among several HAProxy instances, and each HAProxy chooses among the&#xA;servers.&lt;/p&gt;&#xA;</description>
        </item></channel>
</rss>
