19. Operations

This chapter gathers the things commonly done after going live. The full rules for each are in the reference document; here are only the ways to do them and what to watch out for. To find the cause of a problem from its symptom, see chapter 20.

Checklist before going live

  • Every *.secret replaced with newly generated values (see “Changing secrets” below), with file permissions letting only the account that runs rradius read them.
  • Key files (aka.key, dpsk.key, ticket.seal) generated by you, the same on every node, mode 0600, and backed up separately.
  • No demo certificates; expiry dates of the server certificate and of NAS and device certificates monitored.
  • Demo data files removed from migrations/, or replaced with real data before the first start.
  • [[radius.clients]] lists only devices that will really ask; no 0.0.0.0/0.
  • [metrics] bound only to an internal address; with several nodes, [cluster] listen bound to a WireGuard or internal network address.
  • The firewall opens only the ports in use, with limited sources: 1812, 1813 (UDP), 2083 (RadSec), 3799 (CoA), 49 and 300 (TACACS+).
  • rradius-server --check --config ... passes.
  • Started by a service manager such as systemd, stopped with SIGTERM (not SIGKILL) with 30 seconds or more allowed, and the data directory and accounting spool directory on a persistent local disk.
  • Automatic restarts set up (systemd Restart=on-failure, Docker restart: unless-stopped): when a node’s database breaks, rradius exits with status 1 and relies on the restart to recover (chapter 8).
  • After startup, the log has a listening line for every port that should be open. When TACACS+, RadSec or metrics ports cannot be bound, only an error line is logged, and the process does not exit.
  • One login, accounting, and any CoA you will use tried with real devices (not just radtest).
  • Monitoring connected at least to “the node is alive” (Status-Server) and the metrics listed below.

Accounts and data

When accounts live in rradius’s own tables, manage them with the rradius-server user command; when they live elsewhere, have rradius ask there:

ApproachSuitsHow
The user commandAccounts that change often (broadband subscribers, staff, guests)rradius-server user add|passwd|enable|disable|delete|show, seen by the next login
The client, dpsk and aka commandsDynamic client NAS, per-person Wi-Fi keys, SIM cardsrradius-server client|dpsk|aka ..., with SQL set in [accounts.clients], [accounts.dpsk] and [accounts.aka] (chapters 12, 13, 14)
Migration filesData that rarely changes and whose history you want to keep (plans, devices, table structure)Add a SQL file with the next number, put it on every node, and restart them one by one
Verified by an existing directoryThe company already has AD/LDAPThe ldap handler queries the directory directly; rradius stores no accounts (chapter 5)
Verified by an existing systemAccounts are in a billing or membership systemAsk that system’s API with http_request in a lua handler (chapter 18)
Moved from an old systemReplacing an old authentication systemImport the old password hashes as they are, so users need not reset passwords (chapter 17)

With the user command

First add [accounts] to the configuration file, with the SQL for each action (?1 is the account, ?2 the password). For the quick start’s users table:

[accounts]
add = "INSERT INTO users (username, password) VALUES (?1, ?2)"
passwd = "UPDATE users SET password = ?2 WHERE username = ?1"
enable = "UPDATE users SET enabled = 1 WHERE username = ?1"
disable = "UPDATE users SET enabled = 0 WHERE username = ?1"
delete = "DELETE FROM users WHERE username = ?1"
show = "SELECT username, enabled FROM users WHERE username = ?1"

The server need not restart; the command reads the configuration file every time. Then:

rradius-server user add --config rradius.toml dave        # asks for the password twice, without echoing
rradius-server user disable --config rradius.toml carol   # disables; user_query must look only for enabled = 1
rradius-server user show --config rradius.toml dave
  • It changes the whole cluster, with no node-by-node work and no reload. The cluster must be running, and the host running the command must reach the nodes’ api port.
  • Passwords are read from standard input, so they do not stay in the shell history. For a batch import, one account per line, sent through a pipe.
  • How passwords are stored is set by scheme: bcrypt by default; sha512-crypt for high login volumes; clear for CHAP (PPPoE, for example); nthash for MS-CHAPv2 or PEAP. The broadband scenario’s example configuration uses clear.
  • Names are treated as at login: with [auth.username], Dave@Corp is stored as dave, according to the settings.
  • The full configuration rules, output and status codes are in usage section 2.14.
  • Protect the configuration file: these commands connect to the cluster with secret_api, which is the key to the whole database. Let only administrators read the configuration and secret files, and run the commands on a node or over WireGuard.

With migration files

-- 3_plans.sql
INSERT INTO plans (name, max_sessions) VALUES ('family-500m', 3);
UPDATE users SET enabled = 0 WHERE username = 'carol';
  • Each file runs only once, and cannot be changed once applied. So it suits a “change record”, not frequent password changes.
  • Do not change data by connecting other tools directly to the cluster’s api port.
  • The user command only handles accounts. Dynamic client NAS, per-person Wi-Fi keys and SIM cards each have their own command: client (chapter 14), dpsk (chapter 12) and aka (chapter 13), used the same way as user. Other tables use migration files.

Removing old data

Tables like accounting and tacacs_log keep growing, and rradius does not clean them up. They can be pruned regularly from the accounting script: at most once an hour, with only one request in the whole cluster actually doing it.

First create a table that records the time, with a migration:

-- 3_housekeeping.sql
CREATE TABLE housekeeping (name TEXT PRIMARY KEY, last_run INTEGER NOT NULL);
INSERT INTO housekeeping VALUES ('purge-accounting', 0);

Add this before return 'accept' in the accounting script:

-- At most once an hour: read the local replica first, so not every request writes to the cluster.
local now = math.floor(now_unix())
local due = sql_query("SELECT 1 FROM housekeeping WHERE name = 'purge-accounting' AND last_run < ?", {now - 3600})
-- The write goes through the cluster, so only one request succeeds in changing last_run, and it does the delete.
if #due > 0 and sql_execute("UPDATE housekeeping SET last_run = ? WHERE name = 'purge-accounting' AND last_run < ?",
    {now, now - 3600}) == 1 then
  -- Delete at most 10000 rows at a time; leave the rest for the next hour, so no single write takes too long.
  -- ORDER BY makes every node delete the same rows.
  sql_execute([[DELETE FROM accounting WHERE rowid IN
    (SELECT rowid FROM accounting WHERE stopped IS NOT NULL AND stopped < ? ORDER BY rowid LIMIT 10000)]], {now - 180 * 86400})
end
  • How long to keep data (180 days in the example) depends on regulations and company policy. Providers may have a minimum retention period; check before deleting.
  • The delete runs while handling an accounting request, which then gets its reply a little later. The amount deleted must finish within [lua] query_timeout_ms.
  • To keep historical data, have an external export move it away first, then delete.

Reloading (SIGHUP)

Most settings take effect with a SIGHUP after a change, without a restart:

kill -HUP $(pidof rradius-server)
systemctl reload rradius        # when the service file has ExecReload=/bin/kill -HUP $MAINPID

The log says configuration reloaded from ..., or reload failed, keeping the running configuration: ... (the old configuration then stays in use; fix it and send the signal again).

What changedHow it takes effect
Scripts, authentication chains and name normalisation, [lua] time limits, [lua.http]SIGHUP
RADIUS and TACACS+ client lists, shared secret files, TACACS+ authorization groups and enableSIGHUP
Certificates, private keys, CAs and revocation lists for EAP, RadSec and TACACS+ over TLS; other [radius.eap] settings (affecting only new conversations)SIGHUP
[proxy] (every upstream treated as healthy again, DNS results cleared)SIGHUP
[radius.dynamic_clients], [radius.dpsk] (remembered results cleared)SIGHUP
[[radius.coa.clients]], coa_timeout_ms, coa_attemptsSIGHUP
[log.trace]SIGHUP
The dictionary, [cluster] (including new migrations), other [log] settings, [metrics]Restart
Every listening address and port; adding or removing [radius], [radius.tls], [radius.coa], [tacacs] or [tacacs.tls]Restart
Concurrency limits (max_auth_inflight, max_acct_inflight, max_connections), idle_timeout_ms, [lua] max_memory_mbRestart
The two accounting spools ([radius.accounting_spool], [proxy.accounting_spool])Restart
  • When settings that need a restart have changed, SIGHUP lists them in the log (these need a restart to apply: ...) and applies the others as usual.
  • Nothing applies unless everything is valid. With one error, everything stays as it was, and service is not interrupted.
  • Requests in progress finish with the old configuration. Open TACACS+ and RadSec connections are not dropped, but their next packet is handled with the new list: a connection whose client was removed is closed.
  • SIGHUP is also the way to “clear what is remembered”: dynamic clients, the keys matched for per-person passphrases, upstreams found through DNS, and TLS sessions without shared_tickets are all cleared.

The full list is in usage section 2.9. Each node reloads on its own; in a cluster, update the files on every node and send each the signal.

Changing secrets

SecretHow to change it
NAS shared secrets (secret_file)One NAS at a time: change the NAS first, then rradius’s file and send SIGHUP; or the other way round, whichever side can change first. In between, that NAS’s requests fail. With a different secret per NAS, only one is affected at a time
ldap.secretChange the service account’s password in the directory first, then the file, and send SIGHUP
raft.secret, api.secretMust be the same on every node, so the whole cluster has to stop to change them; schedule a maintenance window
TACACS+ shared secretsAs for NAS. After SIGHUP, open connections must use the new secret from their next packet
Upstream shared secrets ([[proxy.servers]])Agree a time with the other side, change both together, then send SIGHUP
aka.key, dpsk.key (master keys protecting table contents)Without downtime: keep the old and new keys side by side (previous_key_file), re-encrypt the data, then remove the old one. See chapter 13 and chapter 12
ticket.sealOld and new side by side (previous_ticket_seal_file); see chapter 15
Certificates and private keysReplace the files and send SIGHUP. New connections and conversations use the new ones; open ones are unaffected

To generate a new secret:

(umask 077; openssl rand -base64 24 > nas.secret)

A shared secret under 16 characters gives a warning at startup, at reload and with --check (has a shared secret shorter than 16 octets), but startup is not refused. Some old devices limit the length or characters, so confirm the device accepts the generated value.

Monitoring

With [metrics] on, GET /metrics returns the Prometheus format. Look at these first:

MetricMeansAdvice
Rate of rradius_radius_responses_total{code="access_reject"}Failed loginsA sudden rise: leaked password attempts, a directory failure, a configuration error
rradius_radius_discarded_total{reason="unknown_client"}Unknown sourcesRising: a new device not added to the list, or someone scanning
rradius_radius_discarded_total{reason="invalid"}Bad signatures or formatsRising: a wrong shared secret
rradius_radius_discarded_total{reason="overload"}Dropped because too busyInvestigate when above 0: a slow database, slow scripts
rradius_acct_spool_bytesAccounting not yet written to the databaseStaying above 0: a database or cluster problem
rradius_cluster_leaderWhether this node has a usable leader0 for more than a few seconds: this node is partitioned or the cluster has no majority
Rate of rradius_radius_duplicates_totalThe NAS is retransmittingRising steadily: handling is slower than the NAS’s retransmission interval
rradius_proxy_server_aliveWhether an upstream is healthyAlert on 0
rradius_proxy_unanswered_totalRequests no upstream answeredRising: every upstream of a realm has a problem
rradius_proxy_acct_spool_bytesForwarded accounting waiting for upstreams to recoverStaying above 0: an upstream has not recovered
rradius_acct_spool_rejected_total, rradius_proxy_acct_spool_rejected_totalAccounting given up during replayAbove 0: look at rejected.log in the spool directory
rradius_radsec_connections_total{result="handshake_failed"}Failed RadSec handshakesRising: certificates expired or revoked, or someone trying
rradius_radsec_connections_openCurrent RadSec connectionsBelow the number of NAS: some NAS has disconnected
rradius_eap_sessions_total{result="timeout"}, {result="evicted"}Unfinished and evicted 802.1X conversationsRising steadily; see chapter 20
rradius_dynamic_client_lookups_total{result="not_found"}Unregistered sourcesA sudden rise: new devices not registered, or someone scanning
rradius_script_http_requests_total{result="timeout"}APIs called by scripts timing outRising: an API problem
rradius_coa_relay_total{result="failed"}Disconnect requests the NAS did not answerRising: the NAS’s CoA settings or the network
rradius_reload_total{result="error"}Failed reloadsAbove 0: read the log
rradius_log_dropped_lines_totalLog lines that could not be written in timeRising: the log destination is too slow

Status-Server is the most accurate liveness check, because it takes the same path as authentication:

echo 'Message-Authenticator=0x00' | radclient -t 2 -r 1 10.99.0.1:1812 status "$(cat nas.secret)"

All metrics are in usage section 2.8.

  • Values accumulate from startup, each node counting its own, and a reload does not reset them. Alert on rates, not absolute values.
  • /metrics has no authentication and no TLS; bind it only to an internal address.

What happens when it gets too busy

When rradius is too busy, it does not answer with errors; it does not answer, so the NAS retransmits or uses its backup:

  • The number of requests handled at once is limited: max_auth_inflight for logins and max_acct_inflight for accounting, 4096 each by default, counted separately, so an accounting surge cannot take login slots. Packets beyond the limit are dropped, with at most one log line per second, RADIUS ... overloaded: discarded N packets, and rradius_radius_discarded_total{reason="overload"} increases.
  • NAS retransmissions are not handled twice: while a request is still being handled, its retransmitted packets are dropped; retransmissions within 10 seconds after handling get the previous answer directly, without querying the database or directory again.
  • Raising the limits usually does not help. A suitable value is roughly “requests handled per second × the NAS’s retransmission timeout”; anything larger just lets more requests queue until after the NAS has given up. When overload appears, first find what makes each request slow (the database, the directory, an upstream, an API, password hashing); see chapter 20.
  • TACACS+ and RadSec are limited by connection count (max_connections, 1024 each by default), and new connections beyond it are closed at once.
  • On Linux, each UDP port opens as many receiving sockets as there are CPUs by default; physical machines dedicated to rradius can also turn on [runtime] pin_threads = true. Other platforms use a single socket. See usage section 2.4.

Everyday care of the accounting spool

With [radius.accounting_spool] (chapter 7) or [proxy.accounting_spool] (chapter 11) set:

TaskHow
Knowing whether there is a backlogrradius_acct_spool_bytes or rradius_proxy_acct_spool_bytes above 0 means there is. Alert when it stays up
Estimating how long it lastsEach record is a little over a hundred bytes plus the attributes the NAS sent. The default limit is 1 GiB; once full, new accounting gets no reply and relies on NAS retransmission
Seeing what was given uprejected.log in the spool directory: requests that would give the same result if run again (rejected by the script, SQL errors, a client that no longer exists). It is not cleaned up automatically, and never replayed
Seeing what is damagedFiles ending in .corrupt and truncated.log: left by power loss or disk problems
Maintenance downtimeSIGTERM stops replay; data stays on disk and continues at the next start
Replacing a nodeWait for this node’s spool to empty (the metric at 0) before shutting it down, or whatever it has not replayed stays on the old disk
Changing settingsBoth spools’ settings need a restart to apply. Before removing a [proxy.realms] entry, make sure the forwarding spool is empty

Spool files are raw packets, with user names and addresses. The directory’s mode is 0700; take it into account for backups and when disposing of disks.

Investigating one user’s problem

1. Look at the authentication log first. One line per login:

authentication protocol="radius" client="wlc" source=10.30.0.10 user="[email protected]" account="staff1" chain="default" handler="sql" decision="reject"

user is the name the device sent, account is the name used for the account lookup, and chain and handler say who decided.

2. When you need the full packets, turn on tracing for a while:

[log.trace]
users = ["staff1"]
minutes = 30

After SIGHUP, this user’s requests and responses are written to the log in full (passwords shown as <hidden>), stopping automatically after 30 minutes. clients = ["wlc"] traces one device instead. The log will contain personal data; remove this section when done.

3. Script problems show as Lua script error (<protocol>/<phase>): <file>:<line>: <message>. Scripts can print their own messages with log(...).

For what common messages mean, see usage section 8.

Upgrading

  1. Read the new version’s change notes, especially items marked as upgrade notes.
  2. Run --check with the new version against your current configuration.
  3. One node: stop → replace the binary → start. A cluster: one node at a time; see chapter 8.

Backups

rradius has no built-in backup command. The cluster itself keeps three copies and survives losing one host, but data deleted by mistake is replicated to every node, so keep separate copies anyway:

  • The source of accounts must be rebuildable: keep migration files in version control; when accounts come from a directory or another system, that is the original.
  • Data that exists only in rradius, such as accounting, is exported regularly by an external process to the billing system or a data warehouse.
  • Key files are not in the database and need separate backups:
    • aka.key (chapter 13): losing it means importing every card again.
    • dpsk.key (chapter 12): losing it means everyone has to reset their Wi-Fi passphrase.
    • ticket.seal (chapter 15): losing it loses no data; generate a new one, and devices do one full handshake.
  • Keep key file backups apart from database backups. Kept together, whoever gets the backup has both the ciphertext and the key.
  • Configuration files, scripts and certificates go into version control or a configuration management tool; private keys and *.secret are kept separately, however you manage secrets.
Built with Hugo
Theme Stack designed by Jimmy