9. Troubleshooting

Start with emfor status: it shows whether the daemon is running, each backend’s state, and the reason of the last failed check. Then look at the daemon’s log (journalctl -u emfor, or syslog on FreeBSD).

The daemon does not start

Message or exit statusCauseFix
Exit 2, config errors listedThe config is invalid.emfor check-config lists every error with its TOML path.
Exit 2, usage printedNo command given (0.1.x style).Use emfor server.
--check-config was replaced by "emfor check-config"A 0.1.x command line.See upgrading.
Exit 1, names an addressA [dns] or [metrics] listen address is in use, or needs privileges (port 53).Free the port, or choose another one.
Exit 1, names <store>.writerAnother emfor already runs with this store.Stop it, or use another [store] path.
Exit 1, cannot open the storeThe directory is missing or not writable for the daemon’s user, or the lock file belongs to another user.Create the directory with the right owner; see lock file owned by another user.
plugins."x".command: ...A plugin failed to start or to answer the handshake.Run the command by hand; check the path and its permissions.

Every lookup fails

ErrorCauseFix
UNAVAILABLE, file not foundThe daemon has never run, or EMFOR_STORE and [store] path differ.Start the daemon; set EMFOR_STORE.
UNAVAILABLE, permission deniedThe application user cannot read the store or write <store>-lock.See who can use the store.
UNAVAILABLE, mentions the LMDB format or the releaselibemfor is from another emfor release than the daemon.Install both from the same release.
UNAVAILABLE, mentions EMFOR_MAX_STALENESSThe variable has an invalid value.Use a whole number with ms or s.
UNAVAILABLE, says truncatedThe store file is shorter than LMDB records it: it was changed in place, or lost its tail in a power loss.Restart the daemon, which moves it to <store>.corrupt and starts a new store; never change the store in place (see the store on disk).
INCOMPATIBLEThe store has another format version.Install the client from the daemon’s release.
NOT_FOUNDNo service of that name in the running config.Check the spelling, and that the reload was applied (emfor status).

A service is ALL DOWN

  1. Look at the reasons in emfor status --service <name>.
  2. If every backend shows the same local-looking reason (TLS, a CA file, connect: Network is unreachable), the problem is likely on the checking host, not the backends.
  3. If the daemon line says stale or stopped, the backends are not down: the daemon is. Every closed service fails until it is running again. Restart it and look at its log.
  4. Check the check itself: run curl or nc from the host to the backend with the same path, port and options.

A backend flaps

FLAPPING in emfor status or the EmforBackendFlapping alert means frequent changes between healthy and unhealthy. Common causes: a timeout close to the backend’s normal response time, or an overloaded backend. Raise timeout (up to interval), raise fall and rise, or check the backend. emfor status --history --service <name> shows when and why it changed.

Checks end in error

error results do not change a backend’s state, but a backend that has only errors for fall × interval returns to unknown. Typical reasons:

Reason starts withCause
plugin <name> unavailable, plugin exitedThe plugin crashed, stopped answering, or is restarting; see the log for its exit status or stopped answering.
plugin <name> busy: it is not reading its requestsThe plugin stopped reading its stdin and 1024 requests are waiting for it. It is restarted once 3 requests in a row go unanswered; if this lasts, the plugin is stuck outside its request loop.
plugin did not answer within timeoutThe plugin is too slow or handles checks one at a time.
password_file: , ca_file:The file cannot be read by the daemon’s user.
exit code N (exec, N not 1 or 2)The script failed in an unexpected way.

A change to unknown with reason check task restarted in emfor status --history is not a check result: the backend’s check task stopped on a bug and was started again, and the log has an error line check task of backend ... stopped. Please report it with that line. A backend that was already unknown gets no history entry, but emfor status shows the same reason until its next check.

DNS does not resolve

SymptomCauseFix
dig @127.0.0.1 -p 5300 gets no answerNo [dns] table, or another listen address.Check emfor status (the daemon line shows dns ...).
REFUSEDThe name is outside the zone, or the query class is not IN (dig ... CH).Name the service under the zone, or change zone; query class IN.
NXDOMAINNo service of that name.Check the name.
SERVFAILNo healthy backend and on_all_down = "closed"; or a validating unbound without domain-insecure.See ALL DOWN; fix the unbound config.
Direct query works, getent hosts does notThe host resolver does not forward the zone.Follow chapter 7.
A failed backend is still returnedResolver or application caching.See caching.

The store looks stale while the daemon runs

emfor status says stale, or clients act as if every backend were down, while the daemon process is alive:

  • emfor_store_commit_failures_total rising and MDB_MAP_FULL in the log: the store is full. Raise [store] map_size and restart.
  • Commits failing for other reasons: the log names them.
  • The system clock jumped forward: the heartbeat looks old until the next commit, about a second later.
  • A local user holds the store’s write lock (a writable lock file); see who can use the store.

Lock file owned by another user

If the lock file <store>-lock was deleted while applications ran, the first process to open the store recreated it with its own owner, and the daemon (another user) may fail to open the store. Stop the applications or fix the file’s owner and mode (chown emfor, chmod to lock_file_mode), then start the daemon. Never delete the lock file alone.

A reload did not take effect

A reload is all or nothing. Look for reload rejected and the error lines in the log, or run emfor check-config. [store], [dns] listen, [metrics] listen and [peers] listen changes always need a restart.

Peers drop each other’s messages after a restart

emfor_peer_messages_rejected_total{reason="auth"} rises and emfor_peer_up drops to 0 for a while right after a daemon restarted: the peer secret was being changed, and the restarted daemon knows only the new one. It stops once every member’s secret_file holds the new secret. Set [peers] previous_secret_file to avoid it (see [peers]). For other reasons see Peer groups.

Built with Hugo
Theme Stack designed by Jimmy