10. Moving from HAProxy or Consul

This chapter is for people who already run HAProxy or Consul to keep applications away from broken backends, and want to see what the same job looks like in emfor. It translates a real-looking configuration of each into emfor.toml, says what has no equivalent, and ends with a way to try emfor next to what you have, without touching traffic.

Read chapter 1 first: the picture is different enough that a line-by-line translation alone will mislead you.

Is emfor the right replacement?

emfor does one thing of what HAProxy and Consul do: it checks a fixed list of backends from the host of the application and tells the application which one to use. It is not in the path of the traffic, and it has no central server.

HAProxyConsulemfor
Where the traffic goesThrough HAProxy (one more hop)Straight to the backendStraight to the backend
Who decides which backendHAProxy, for every connectionThe application, from a DNS answerThe application, from the local store (client library or emfor lookup) or from a DNS answer
Where the checks runIn HAProxyIn the Consul agent of the node that runs the serviceOn every application host, against every backend
Which backends existThe config fileServices register themselvesThe config file of each host
A central serverNoYes (servers with a quorum)No

So emfor fits when the backends are a known list of addresses (virtual machines, bare metal, databases, AAA servers), and the application can ask emfor where to connect, either with the client library, with a shell command, or through DNS.

It does not replace what a proxy does with the traffic itself: least connections, retries, queueing, TLS termination, routing by URL or header, connection limits, and passive health checks (marking a server down because real requests failed). If you need those, keep HAProxy and see what does not carry over.

From HAProxy

An HAProxy backend like this one:

backend api
    balance roundrobin
    option httpchk GET /healthz
    http-check expect status 200
    default-server inter 2s fall 3 rise 2
    server api1 10.0.0.11:8080 check
    server api2 10.0.0.12:8080 check weight 3
    server api3 10.0.1.11:8080 check backup slowstart 30s

becomes this service in emfor.toml:

[services."api.svc.emfor"]
check = "http"
balance = "weighted"
interval = "2s"
fall = 3
rise = 2
slow_start = "30s"
backends = [
  "10.0.0.11:8080",
  { addr = "10.0.0.12:8080", weight = 3 },
  { addr = "10.0.1.11:8080", backup = true },
]

[services."api.svc.emfor".check_options]
path = "/healthz"
expect_status = [200]

Line by line:

HAProxyemforNotes
backend api[services."api.svc.emfor"]The service name is host-name-like and is also the DNS name (chapter 3).
server api1 10.0.0.11:8080an entry of backendsIP addresses only, never host names: emfor does not depend on DNS to find its backends.
checkcheck = "http" (or "tcp")One check for the whole service.
inter 2sinterval = "2s"Both default to 2 seconds.
fall 3 / rise 2fall = 3 / rise = 2Both default to 3 and 2 (rise, fall and errors).
option httpchk GET /healthzpath = "/healthz"emfor always sends GET. HAProxy without arguments sends OPTIONS /; emfor sends GET / unless you set path (http check).
http-check expect status 200expect_status = [200]Without it, any status from 200 to 399 is up. HAProxy’s default accepts 2xx and 3xx too. Ranges are written "200-299".
http-check send ... hdr Host xhost_header = "x"Only the Host header can be set.
weight 3{ addr = "...", weight = 3 }From 1 to 100, and only used by balance = "weighted" (and by sticky with a key). HAProxy weights go to 256, and 0 takes a server out: emfor refuses weight = 0 and weight = 256 (weight must be from 1 to 100), so rescale the weights to 1 to 100 and use drain = true for 0 (see the table at the end).
backupbackup = trueA backup is used only while no other backend passes its checks (backup backends).
slowstart 30sslow_start = "30s"A recovered backend gets a growing share of the traffic, in ten steps (slow start). Set on the service, not on a backend.
balance roundrobinbalance = "round-robin" or "weighted"HAProxy’s round robin takes the weights into account; emfor’s round-robin does not. Use weighted when you use weights.
balance firstbalance = "failover"The first backend that is healthy, in config order. Without a maxconn, HAProxy’s first does the same; with one it moves on when the first server is full, and emfor has no connection limit.
balance source, balance uri, balance hdrbalance = "sticky"The application passes the key: lookup(name, key). The same key gets the same backend while it is healthy (chapter 5).
balance leastconnnoneemfor does not see the connections; see below.
check port 8081noneemfor checks the backend’s own address and port.
agent-checka plugin or an exec checkThere is no agent protocol; write the check as a plugin or an exec check.
maxconn, queues, observe layer7noneSee below.

The application then connects to what emfor tells it instead of to HAProxy. Pick the way that suits it:

$ emfor lookup api.svc.emfor
10.0.0.11:8080
$ curl "http://$(emfor lookup api.svc.emfor)/healthz"

or the client library of its language (chapter 6), or the DNS name api.svc.emfor: for that add a [dns] table to the config (the example above has none, and the DNS server runs only when the table is present) and forward the zone from the host resolver (chapter 7). Limits to know about: an A or AAAA answer carries no port, so the application keeps the port itself (or asks for SRV); a runtime or resolver that caches answers delays a change by its cache time (caching); and emfor lookup is a command that prints one backend per process, so a program that connects to $(emfor lookup ...) needs a small wrapper that runs it for each connection (chapter 6).

Trying it next to HAProxy

emfor does not touch the traffic, so you can run it beside HAProxy first:

  1. Write the services into emfor.toml (with the deb or rpm, into a file of /etc/emfor/conf.d/, which the packaged main config reads) and check the config with emfor check-config.
  2. Start the daemon (the deb has started it at install; after a change to a file, systemctl reload emfor) and compare emfor status with the HAProxy stats page: the same backends should be healthy and unhealthy, and the reasons in emfor status say why a check failed.
  3. Look at what an application would get, and why, with emfor lookup api.svc.emfor --explain (chapter 6).
  4. Move one application at a time to emfor, watching the metrics and emfor status --watch.
  5. Remove a backend from HAProxy’s configuration only when no application uses it any more.

Keeping HAProxy and letting emfor feed it

You do not have to move the applications. HAProxy can keep the traffic, with its retries, queues, TLS and routing, and take its list of servers from emfor’s SRV answers, so that emfor’s checks decide who is in it (HAProxy in chapter 7 has the configuration: a server-template and a resolvers section that points at emfor’s DNS server, so the config needs a [dns] table and the services must be named under its zone). What you get: any emfor check (for example a database plugin) decides the health instead of check, drain = true takes a server out with a reload of emfor and no reload of HAProxy, and on every host a peer group can vote on a backend.

What does not carry over, as tried with HAProxy 3.4.7 and 2.8.31: HAProxy ignores the priorities and weights of the answer, so failover, weighted, and backup = true do nothing there (keep HAProxy’s own backup and weight); the slots of the template limit the number of backends; and after a SERVFAIL HAProxy drops its servers after hold other (30 seconds by default) and answers 503. Use one name server per HAProxy, the emfor of the same host. Not tried: more than 27 backends over UDP, and long-running use.

From Consul

A Consul service with an HTTP check, as the agent of the node that runs it registers it:

service {
  name = "api"
  port = 8080
  check {
    http                     = "http://localhost:8080/healthz"
    interval                 = "10s"
    timeout                  = "1s"
    success_before_passing   = 2
    failures_before_critical = 3
  }
}

The same job in emfor is a service that lists the nodes, and checks them from the host that asks:

[services."api.service.consul"]
check = "http"
interval = "10s"
timeout = "1s"
rise = 2
fall = 3
backends = ["10.0.0.11:8080", "10.0.0.12:8080", "10.0.1.11:8080"]

[services."api.service.consul".check_options]
path = "/healthz"
expect_status = ["200-299"]

[dns]
zone = "service.consul"

The service is named api.service.consul and the zone is service.consul, so a program that looks up api.service.consul keeps working once the host resolver forwards that zone to emfor instead of to Consul (chapter 7). (You can also keep emfor’s own zone, svc.emfor, and change the names.)

ConsulemforNotes
service { name, port }a service and its backendsThere is no registration: the list of backends is in the config of each host, written by whatever installs the config (Ansible, Puppet, a template). Adding a node means changing the files and reloading (what a reload changes).
check { http = ... }check = "http" and check_optionsThe URL is split: the address is the backend’s, the path is path. Consul checks usually run on the node itself (localhost); emfor connects to the backend’s address from the application host, so a firewall must allow it.
check { tcp = ... }check = "tcp"
check { grpc = ... }the grpc plugin of emfor-checkSee plugins.
check { args = [...] } (script)check = "exec"
intervalintervalConsul has no default; emfor’s is 2 seconds.
timeouttimeoutConsul’s default for HTTP and TCP is 10 seconds; emfor’s is 1 second and cannot exceed interval.
success_before_passingriseConsul’s default is 0, where the first result decides. emfor refuses rise = 0 (rise must be at least 1): rise = 1 is the same, and emfor’s default is 2.
failures_before_criticalfallConsul’s default is 0, where the first result decides. emfor refuses fall = 0 (fall must be at least 1): fall = 1 is the same, and emfor’s default is 3.
method, headernoneemfor sends GET; of the headers only Host can be set (host_header).
tls_skip_verify = truetls_verify = falseTogether with scheme = "https".
Status passing, warning, criticalhealthy, unhealthy, unknownA Consul HTTP check is passing for 200 to 299, warning for 429 and critical otherwise; emfor has no warning: set expect_status to what you accept.
status (initial critical)starts unknownA backend becomes healthy after rise passes. Until then an open service still returns an address (chapter 5).
consul maint -enable, enable_maintenancedrain = true on the backendThen reload; see draining a backend.
deregister_critical_service_afternoneThe list is static; nothing deregisters a backend.
TTL and alias checksnoneemfor only runs its own checks.
api.service.consulthe service name, with [dns] zoneSee above.
_api._tcp.service.consul (SRV)_<anything>._tcp.<service name>emfor’s SRV owner names contain the whole service name, for example _http._tcp.api.service.consul (chapter 7).
tags, datacenterssite and balance = "locality"A backend gets a site, and a host prefers the backends of its own site (chapter 5).

Consul reads a directory of service files, one per service. emfor can do the same with include_dir: put include_dir = "conf.d" as the first line of emfor.toml, before any table (after [dns] it is an error, dns.include_dir: unknown key), create the directory, and put one file per service in it, for example /etc/emfor/conf.d/10-web.toml:

[services."web.svc.emfor"]
check = "http"
backends = ["10.0.0.21:8080", "10.0.0.22:8080"]

[services."web.svc.emfor".check_options]
path = "/healthz"

The directory and its files must not be writable by group or others and must be owned by root or the user that runs emfor (permissions). emfor check-config lists the files it read, and a reload re-reads them all (chapter 3). Unlike Consul, emfor does not register anything: the file is the list of backends.

Trying it next to Consul

The steps are those of the HAProxy section: write the config, run emfor check-config, start the daemon beside the Consul agent, compare the health emfor status shows with what consul catalog and the Consul UI show, then move one application at a time. A difference between the two is the useful part: it means the backend looks different from where the application is than from where the Consul agent is.

What does not carry over

You useIn emfor
Least connections, connection limits, queuesNo. emfor is not in the data path and does not see connections. Keep HAProxy for this.
Retries and timeouts of the proxyThe application retries. The client returns one address per lookup, so a retry gets the same backend until its next check fails, which takes about one interval, if the service is sticky or failover; with round-robin a retry in the same process takes the next candidate. A backend is left out of the candidates at its first failed check, not only once fall checks have marked it down (chapter 5).
Passive health checks (observe layer7)No: only active checks. A failing backend is noticed by the next checks; with a peer group, when one member sees a backend go from healthy to unhealthy (after fall failed checks) it tells the others, which recheck it at once and every 200ms, up to fall checks.
Taking a backend out for maintenance (disabled, maint, drain)drain = true on the backend, then reload (chapter 5): it is never chosen and its checks go on. weight = 0 is refused (weight must be from 1 to 100), and there is no run-time command (set server ... state drain): the config is the only way.
Routing by URL, header or cookie, TLS terminationNo.
Registration, tags and key-value store of ConsulNo. The list of backends is config.
Several configuration files (haproxy -f dir)A main file and a directory of fragments with include_dir (chapter 3). A fragment cannot include another directory, and a list such as backends is replaced as a whole, not merged.
A central view of all checksEach host has its own. Use the metrics with Prometheus.

Where a row says “keep HAProxy”, it is also fine to use both: emfor chooses among several HAProxy instances, and each HAProxy chooses among the servers.

Built with Hugo
Theme Stack designed by Jimmy