6. Client libraries

A client library has one call: look up a service, get back one backend address. It reads the local store, so it needs the emfor daemon running on the same host, and nothing else.

The lookup

Every client takes a service name and an optional affinity key, and returns a string:

  • ip:port for IPv4, such as 10.0.0.11:8080;
  • [ip]:port for IPv6, such as [2001:db8::11]:8080.

Both forms can be passed to the usual “connect to host:port” functions of most languages. The name is matched without regard to case and with one trailing dot ignored. The affinity key matters only for balance = "sticky"; an empty key is no key.

Call lookup for every new connection, not once at startup: the answer changes as backends fail and recover, and a lookup costs about a microsecond.

Errors

Every client fails with one of four codes, named the same in every language:

CodeMeaningWhat to do
UNAVAILABLEThe store cannot be used: it does not exist, cannot be opened, was written by another emfor release’s LMDB, is not initialized, or EMFOR_MAX_STALENESS is invalid.Check that the daemon runs, EMFOR_STORE, file permissions, and that libemfor comes from the daemon’s release. The message says which.
INCOMPATIBLEThe store has another format version, or a record cannot be read.Install the client from the daemon’s release.
NOT_FOUNDNo service of that name.Check the name against the config.
ALL_DOWNNo healthy backend, and the service has on_all_down = "closed".Fail the request, or retry later.

A stale store is not an error of its own: it makes every backend count as unhealthy, so an open service still returns an address and a closed one fails with ALL_DOWN.

Configuration

Clients have no configuration call. Two environment variables are read on the first lookup in a process (and again in a child after fork):

VariableDefaultMeaning
EMFOR_STORE/var/lib/emfor/state.mdbPath of the store; must match the daemon’s [store] path.
EMFOR_MAX_STALENESS3sHow old the heartbeat may be: a positive whole number with ms or s, such as 1500ms. An invalid value makes every lookup fail with UNAVAILABLE.

Examples

Rust

match emfor_client::lookup("api.svc.emfor", Some("user-42")) {
    Ok(addr) => println!("connect to {addr}"),
    Err(e) => eprintln!("{e}"), // "ALL_DOWN: ..."; e.code is an ErrorCode
}

C

#include <emfor.h>

char addr[EMFOR_ADDR_MAX];
emfor_status st = emfor_lookup("api.svc.emfor", "user-42", addr, sizeof addr);
if (st == EMFOR_OK)
    printf("connect to %s\n", addr);
else
    fprintf(stderr, "%s: %s\n", emfor_status_name(st), emfor_last_error());

Build with cc app.c $(pkg-config --cflags --libs emfor). C also has EMFOR_INVALID_ARGUMENT for a NULL or non-UTF-8 argument or a buffer that is too small; EMFOR_ADDR_MAX bytes always fit.

Go

import "github.com/embark-systems/emfor/clients/go"

addr, err := emfor.Lookup("api.svc.emfor", "user-42")
var e *emfor.Error
if errors.As(err, &e) && e.Code == emfor.AllDown {
	// no healthy backend
}

Python

import emfor

try:
    addr = emfor.lookup("api.svc.emfor", affinity_key="user-42")
except emfor.AllDownError:
    ...
except emfor.Error as e:
    log.warning("%s: %s", e.code, e.message)

Perl

use Emfor;

my $addr = eval { Emfor::lookup('api.svc.emfor', 'user-42') };
warn "$@\n" if $@;   # an Emfor::Error with ->code and ->message

PHP

try {
    $addr = Emfor\lookup('api.svc.emfor', 'user-42');
} catch (Emfor\LookupException $e) {
    error_log($e->getMessage());   // $e->errorCode is "ALL_DOWN" etc.
}

Each client’s README covers installing it and its exact API: Rust, C, Go, Python, Perl, PHP.

From the shell: emfor lookup

emfor lookup answers from the shell what a client library would answer, for scripts and cron jobs that have no client, and with --explain why.

$ emfor lookup api.svc.emfor
10.0.0.11:8080
$ emfor lookup api.svc.emfor --key user-42

It reads the store at the [store] path of the config (--config, as emfor status does) and applies EMFOR_MAX_STALENESS like a client. It never writes the store. --key is the affinity key of a sticky service.

Every run is a new process, so a rotating balance (round-robin, weighted, locality, fastest, and sticky without a key) gives one of the candidates, starting at a random point: two runs can print different backends. A script that needs the same answer every time uses a failover service, or a sticky one with --key.

--explain says why: whether the store is stale, which tier the candidates come from (the tiers of chapter 5), every backend that is out and the reason, and how balance chooses.

$ emfor lookup api.svc.emfor --explain
service: api.svc.emfor
answering: as a client library answers
store: /var/lib/emfor/state.mdb (heartbeat 0.4s ago, fresh)
balance: round-robin  on_all_down: open
tier: healthy primary backends whose last check passed
candidates:
  10.0.0.11:8080
excluded:
  10.0.0.12:8080  last check failed
  10.0.0.13:8080  backup held back
  10.0.0.14:8080  unhealthy
selection: round-robin: the candidates in turn
  the actual choice depends on the rotation start of the process, which is random in a new process
answer: 10.0.0.11:8080

The reasons are stale (a stale store makes no backend healthy or unknown), unhealthy, unknown (a better tier has candidates), last check failed, backup held back, not in the pool (fastest) and not at this site (locality). A candidate in slow start shows its step and effective weight. failover and sticky with a key name the backend they choose; the other methods list the candidates and the rule, because the choice depends on the rotation of the process. --explain does not advance any rotation.

--json prints one JSON object (service, via, family, store, balance, on_all_down, result, and with --explain also explain); later releases only add fields. The text is not a stable interface.

--dns A and --dns AAAA (these imply --explain) explain what the DNS server of the daemon answers to an A or an AAAA query: only the backends of that family are looked at, the store counts as fresh, sticky has no key, and weighted is the daemon’s smooth weighted round-robin. It reads the store, which gets a new reason up to about one second after the daemon decides it, so for that second it can differ from the live DNS answer; the rotation counters are in the daemon’s memory and are not shown. SRV answers are not explained.

Exit statusMeaning
0An address was printed (with --dns: the service has candidates for the family)
1No backend can be chosen: ALL_DOWN, or no backend of the family with --dns
2No such service, an invalid command line, or a config that cannot be read
3The store cannot be used (UNAVAILABLE) or has a format this command does not read (INCOMPATIBLE), or the output cannot be written (a full disk; a closed pipe is not an error)

--explain does not change the exit status, and a failed lookup still prints what is known.

Threads, processes and fork

  • Lookups are safe from many threads at once.
  • The store is opened once per process and reused. A lookup holds an LMDB reader slot only while it reads. The store has 1024 slots shared by all processes on the host; a process killed in the middle of a lookup leaves its slot taken until the daemon frees it, about once a minute.
  • After fork, the child opens the store again on its first lookup; this works also when the parent looked up before forking (php-fpm, preforking servers), and when other threads of the parent were looking up at the moment of the fork.
  • On FreeBSD, fork waits until no lookup of the process is inside the store, and keeps new lookups out until it has returned in the parent and in the child; without that, a child forked while another thread looked up could wait for ever in its first lookup, on a lock that FreeBSD’s threading library leaves held in the child. The wait is as long as one read, and lookups wait the same way while a fork runs. A thread stopped (SIGSTOP) inside a lookup holds the fork up until it continues. libemfor cannot be unloaded on FreeBSD (dlclose leaves it mapped), because its fork handlers stay registered until the process ends; link libemfor.a statically into a module that is unloaded, or depend on the Rust crate from your own cdylib, only with -Wl,-z,nodelete.
  • What the guard does not cover on FreeBSD: a fork from a signal handler that interrupted a lookup of the same thread (it returns, but the child can hang in its first lookup, as without the guard), and other users of process-shared mutexes in the same process (another LMDB environment, say), which can still leave a child waiting in its own lookup. Do not fork from signal handlers; the library cannot fix the rest.
  • Round-robin counters are per process.

When the store file is replaced or grows

If the store file is replaced while applications run (the daemon moved a damaged store aside, or someone deleted it and the daemon created a new one), clients notice within about 100 ms and switch to the new file; no application restart is needed. The same holds when the daemon is restarted with a larger [store] map_size and the store grows past the old size.

A store file that exists but cannot be opened (another LMDB format, no permission, not a store) is tried again at most once a second; lookups in between fail with the same UNAVAILABLE error. A missing store file is tried on every lookup, so applications started before the daemon find the store as soon as it exists.

Do not delete the lock file (<store>-lock) alone. Start the daemon before applications that run as root or as the daemon’s user: if the lock file is missing, the first process to open the store creates it, owned by that process’s user, and the daemon may then fail to start.

Same release as the daemon

The client library contains the daemon’s own LMDB build, and the store’s lock file format depends on it. Install libemfor (and the wrappers) from the same emfor release as the daemon, and upgrade them together. A mismatch shows up as UNAVAILABLE with a message that says so.

Store format version

The daemon writes store format version 2, which adds the backend field backup. Clients from this release read versions 1 and 2, so they work with an older daemon too; older clients fail every lookup on a version 2 store with INCOMPATIBLE. When upgrading, upgrade the clients (and emfor status) on a host before the daemon.

Performance

A lookup takes about 1–1.5 µs on one thread. Throughput grows with threads up to about four, then the shared reader lock of the store sets the pace (about two million lookups a second in total on Linux). See docs/performance.md for the measurements.

Built with Hugo
Theme Stack designed by Jimmy