This chapter gathers the things commonly done after going live. The full rules for each are in the reference document; here are only the ways to do them and what to watch out for. To find the cause of a problem from its symptom, see chapter 20.
Checklist before going live
- Every
*.secretreplaced with newly generated values (see “Changing secrets” below), with file permissions letting only the account that runs rradius read them. - Key files (
aka.key,dpsk.key,ticket.seal) generated by you, the same on every node, mode 0600, and backed up separately. - No demo certificates; expiry dates of the server certificate and of NAS and device certificates monitored.
- Demo data files removed from
migrations/, or replaced with real data before the first start. -
[[radius.clients]]lists only devices that will really ask; no0.0.0.0/0. -
[metrics]bound only to an internal address; with several nodes,[cluster] listenbound to a WireGuard or internal network address. - The firewall opens only the ports in use, with limited sources: 1812, 1813 (UDP), 2083 (RadSec), 3799 (CoA), 49 and 300 (TACACS+).
-
rradius-server --check --config ...passes. - Started by a service manager such as systemd, stopped with SIGTERM (not SIGKILL) with 30 seconds or more allowed, and the data directory and accounting spool directory on a persistent local disk.
- Automatic restarts set up (systemd
Restart=on-failure, Dockerrestart: unless-stopped): when a node’s database breaks, rradius exits with status 1 and relies on the restart to recover (chapter 8). - After startup, the log has a
listeningline for every port that should be open. When TACACS+, RadSec or metrics ports cannot be bound, only an error line is logged, and the process does not exit. - One login, accounting, and any CoA you will use tried with real devices (not just
radtest). - Monitoring connected at least to “the node is alive” (Status-Server) and the metrics listed below.
Accounts and data
When accounts live in rradius’s own tables, manage them with the rradius-server user command; when they live elsewhere, have rradius ask there:
| Approach | Suits | How |
|---|---|---|
The user command | Accounts that change often (broadband subscribers, staff, guests) | rradius-server user add|passwd|enable|disable|delete|show, seen by the next login |
The client, dpsk and aka commands | Dynamic client NAS, per-person Wi-Fi keys, SIM cards | rradius-server client|dpsk|aka ..., with SQL set in [accounts.clients], [accounts.dpsk] and [accounts.aka] (chapters 12, 13, 14) |
| Migration files | Data that rarely changes and whose history you want to keep (plans, devices, table structure) | Add a SQL file with the next number, put it on every node, and restart them one by one |
| Verified by an existing directory | The company already has AD/LDAP | The ldap handler queries the directory directly; rradius stores no accounts (chapter 5) |
| Verified by an existing system | Accounts are in a billing or membership system | Ask that system’s API with http_request in a lua handler (chapter 18) |
| Moved from an old system | Replacing an old authentication system | Import the old password hashes as they are, so users need not reset passwords (chapter 17) |
With the user command
First add [accounts] to the configuration file, with the SQL for each action (?1 is the account, ?2 the password). For the quick start’s users table:
[accounts]
add = "INSERT INTO users (username, password) VALUES (?1, ?2)"
passwd = "UPDATE users SET password = ?2 WHERE username = ?1"
enable = "UPDATE users SET enabled = 1 WHERE username = ?1"
disable = "UPDATE users SET enabled = 0 WHERE username = ?1"
delete = "DELETE FROM users WHERE username = ?1"
show = "SELECT username, enabled FROM users WHERE username = ?1"
The server need not restart; the command reads the configuration file every time. Then:
rradius-server user add --config rradius.toml dave # asks for the password twice, without echoing
rradius-server user disable --config rradius.toml carol # disables; user_query must look only for enabled = 1
rradius-server user show --config rradius.toml dave
- It changes the whole cluster, with no node-by-node work and no reload. The cluster must be running, and the host running the command must reach the nodes’ api port.
- Passwords are read from standard input, so they do not stay in the shell history. For a batch import, one account per line, sent through a pipe.
- How passwords are stored is set by
scheme: bcrypt by default;sha512-cryptfor high login volumes;clearfor CHAP (PPPoE, for example);nthashfor MS-CHAPv2 or PEAP. The broadband scenario’s example configuration usesclear. - Names are treated as at login: with
[auth.username],Dave@Corpis stored asdave, according to the settings. - The full configuration rules, output and status codes are in usage section 2.14.
- Protect the configuration file: these commands connect to the cluster with
secret_api, which is the key to the whole database. Let only administrators read the configuration and secret files, and run the commands on a node or over WireGuard.
With migration files
-- 3_plans.sql
INSERT INTO plans (name, max_sessions) VALUES ('family-500m', 3);
UPDATE users SET enabled = 0 WHERE username = 'carol';
- Each file runs only once, and cannot be changed once applied. So it suits a “change record”, not frequent password changes.
- Do not change data by connecting other tools directly to the cluster’s api port.
- The
usercommand only handles accounts. Dynamic client NAS, per-person Wi-Fi keys and SIM cards each have their own command:client(chapter 14),dpsk(chapter 12) andaka(chapter 13), used the same way asuser. Other tables use migration files.
Removing old data
Tables like accounting and tacacs_log keep growing, and rradius does not clean them up. They can be pruned regularly from the accounting script: at most once an hour, with only one request in the whole cluster actually doing it.
First create a table that records the time, with a migration:
-- 3_housekeeping.sql
CREATE TABLE housekeeping (name TEXT PRIMARY KEY, last_run INTEGER NOT NULL);
INSERT INTO housekeeping VALUES ('purge-accounting', 0);
Add this before return 'accept' in the accounting script:
-- At most once an hour: read the local replica first, so not every request writes to the cluster.
local now = math.floor(now_unix())
local due = sql_query("SELECT 1 FROM housekeeping WHERE name = 'purge-accounting' AND last_run < ?", {now - 3600})
-- The write goes through the cluster, so only one request succeeds in changing last_run, and it does the delete.
if #due > 0 and sql_execute("UPDATE housekeeping SET last_run = ? WHERE name = 'purge-accounting' AND last_run < ?",
{now, now - 3600}) == 1 then
-- Delete at most 10000 rows at a time; leave the rest for the next hour, so no single write takes too long.
-- ORDER BY makes every node delete the same rows.
sql_execute([[DELETE FROM accounting WHERE rowid IN
(SELECT rowid FROM accounting WHERE stopped IS NOT NULL AND stopped < ? ORDER BY rowid LIMIT 10000)]], {now - 180 * 86400})
end
- How long to keep data (180 days in the example) depends on regulations and company policy. Providers may have a minimum retention period; check before deleting.
- The delete runs while handling an accounting request, which then gets its reply a little later. The amount deleted must finish within
[lua] query_timeout_ms. - To keep historical data, have an external export move it away first, then delete.
Reloading (SIGHUP)
Most settings take effect with a SIGHUP after a change, without a restart:
kill -HUP $(pidof rradius-server)
systemctl reload rradius # when the service file has ExecReload=/bin/kill -HUP $MAINPID
The log says configuration reloaded from ..., or reload failed, keeping the running configuration: ... (the old configuration then stays in use; fix it and send the signal again).
| What changed | How it takes effect |
|---|---|
Scripts, authentication chains and name normalisation, [lua] time limits, [lua.http] | SIGHUP |
RADIUS and TACACS+ client lists, shared secret files, TACACS+ authorization groups and enable | SIGHUP |
Certificates, private keys, CAs and revocation lists for EAP, RadSec and TACACS+ over TLS; other [radius.eap] settings (affecting only new conversations) | SIGHUP |
[proxy] (every upstream treated as healthy again, DNS results cleared) | SIGHUP |
[radius.dynamic_clients], [radius.dpsk] (remembered results cleared) | SIGHUP |
[[radius.coa.clients]], coa_timeout_ms, coa_attempts | SIGHUP |
[log.trace] | SIGHUP |
The dictionary, [cluster] (including new migrations), other [log] settings, [metrics] | Restart |
Every listening address and port; adding or removing [radius], [radius.tls], [radius.coa], [tacacs] or [tacacs.tls] | Restart |
Concurrency limits (max_auth_inflight, max_acct_inflight, max_connections), idle_timeout_ms, [lua] max_memory_mb | Restart |
The two accounting spools ([radius.accounting_spool], [proxy.accounting_spool]) | Restart |
- When settings that need a restart have changed, SIGHUP lists them in the log (
these need a restart to apply: ...) and applies the others as usual. - Nothing applies unless everything is valid. With one error, everything stays as it was, and service is not interrupted.
- Requests in progress finish with the old configuration. Open TACACS+ and RadSec connections are not dropped, but their next packet is handled with the new list: a connection whose client was removed is closed.
- SIGHUP is also the way to “clear what is remembered”: dynamic clients, the keys matched for per-person passphrases, upstreams found through DNS, and TLS sessions without
shared_ticketsare all cleared.
The full list is in usage section 2.9. Each node reloads on its own; in a cluster, update the files on every node and send each the signal.
Changing secrets
| Secret | How to change it |
|---|---|
NAS shared secrets (secret_file) | One NAS at a time: change the NAS first, then rradius’s file and send SIGHUP; or the other way round, whichever side can change first. In between, that NAS’s requests fail. With a different secret per NAS, only one is affected at a time |
ldap.secret | Change the service account’s password in the directory first, then the file, and send SIGHUP |
raft.secret, api.secret | Must be the same on every node, so the whole cluster has to stop to change them; schedule a maintenance window |
| TACACS+ shared secrets | As for NAS. After SIGHUP, open connections must use the new secret from their next packet |
Upstream shared secrets ([[proxy.servers]]) | Agree a time with the other side, change both together, then send SIGHUP |
aka.key, dpsk.key (master keys protecting table contents) | Without downtime: keep the old and new keys side by side (previous_key_file), re-encrypt the data, then remove the old one. See chapter 13 and chapter 12 |
ticket.seal | Old and new side by side (previous_ticket_seal_file); see chapter 15 |
| Certificates and private keys | Replace the files and send SIGHUP. New connections and conversations use the new ones; open ones are unaffected |
To generate a new secret:
(umask 077; openssl rand -base64 24 > nas.secret)
A shared secret under 16 characters gives a warning at startup, at reload and with --check (has a shared secret shorter than 16 octets), but startup is not refused. Some old devices limit the length or characters, so confirm the device accepts the generated value.
Monitoring
With [metrics] on, GET /metrics returns the Prometheus format. Look at these first:
| Metric | Means | Advice |
|---|---|---|
Rate of rradius_radius_responses_total{code="access_reject"} | Failed logins | A sudden rise: leaked password attempts, a directory failure, a configuration error |
rradius_radius_discarded_total{reason="unknown_client"} | Unknown sources | Rising: a new device not added to the list, or someone scanning |
rradius_radius_discarded_total{reason="invalid"} | Bad signatures or formats | Rising: a wrong shared secret |
rradius_radius_discarded_total{reason="overload"} | Dropped because too busy | Investigate when above 0: a slow database, slow scripts |
rradius_acct_spool_bytes | Accounting not yet written to the database | Staying above 0: a database or cluster problem |
rradius_cluster_leader | Whether this node has a usable leader | 0 for more than a few seconds: this node is partitioned or the cluster has no majority |
Rate of rradius_radius_duplicates_total | The NAS is retransmitting | Rising steadily: handling is slower than the NAS’s retransmission interval |
rradius_proxy_server_alive | Whether an upstream is healthy | Alert on 0 |
rradius_proxy_unanswered_total | Requests no upstream answered | Rising: every upstream of a realm has a problem |
rradius_proxy_acct_spool_bytes | Forwarded accounting waiting for upstreams to recover | Staying above 0: an upstream has not recovered |
rradius_acct_spool_rejected_total, rradius_proxy_acct_spool_rejected_total | Accounting given up during replay | Above 0: look at rejected.log in the spool directory |
rradius_radsec_connections_total{result="handshake_failed"} | Failed RadSec handshakes | Rising: certificates expired or revoked, or someone trying |
rradius_radsec_connections_open | Current RadSec connections | Below the number of NAS: some NAS has disconnected |
rradius_eap_sessions_total{result="timeout"}, {result="evicted"} | Unfinished and evicted 802.1X conversations | Rising steadily; see chapter 20 |
rradius_dynamic_client_lookups_total{result="not_found"} | Unregistered sources | A sudden rise: new devices not registered, or someone scanning |
rradius_script_http_requests_total{result="timeout"} | APIs called by scripts timing out | Rising: an API problem |
rradius_coa_relay_total{result="failed"} | Disconnect requests the NAS did not answer | Rising: the NAS’s CoA settings or the network |
rradius_reload_total{result="error"} | Failed reloads | Above 0: read the log |
rradius_log_dropped_lines_total | Log lines that could not be written in time | Rising: the log destination is too slow |
Status-Server is the most accurate liveness check, because it takes the same path as authentication:
echo 'Message-Authenticator=0x00' | radclient -t 2 -r 1 10.99.0.1:1812 status "$(cat nas.secret)"
All metrics are in usage section 2.8.
- Values accumulate from startup, each node counting its own, and a reload does not reset them. Alert on rates, not absolute values.
/metricshas no authentication and no TLS; bind it only to an internal address.
What happens when it gets too busy
When rradius is too busy, it does not answer with errors; it does not answer, so the NAS retransmits or uses its backup:
- The number of requests handled at once is limited:
max_auth_inflightfor logins andmax_acct_inflightfor accounting, 4096 each by default, counted separately, so an accounting surge cannot take login slots. Packets beyond the limit are dropped, with at most one log line per second,RADIUS ... overloaded: discarded N packets, andrradius_radius_discarded_total{reason="overload"}increases. - NAS retransmissions are not handled twice: while a request is still being handled, its retransmitted packets are dropped; retransmissions within 10 seconds after handling get the previous answer directly, without querying the database or directory again.
- Raising the limits usually does not help. A suitable value is roughly “requests handled per second × the NAS’s retransmission timeout”; anything larger just lets more requests queue until after the NAS has given up. When overload appears, first find what makes each request slow (the database, the directory, an upstream, an API, password hashing); see chapter 20.
- TACACS+ and RadSec are limited by connection count (
max_connections, 1024 each by default), and new connections beyond it are closed at once. - On Linux, each UDP port opens as many receiving sockets as there are CPUs by default; physical machines dedicated to rradius can also turn on
[runtime] pin_threads = true. Other platforms use a single socket. See usage section 2.4.
Everyday care of the accounting spool
With [radius.accounting_spool] (chapter 7) or [proxy.accounting_spool] (chapter 11) set:
| Task | How |
|---|---|
| Knowing whether there is a backlog | rradius_acct_spool_bytes or rradius_proxy_acct_spool_bytes above 0 means there is. Alert when it stays up |
| Estimating how long it lasts | Each record is a little over a hundred bytes plus the attributes the NAS sent. The default limit is 1 GiB; once full, new accounting gets no reply and relies on NAS retransmission |
| Seeing what was given up | rejected.log in the spool directory: requests that would give the same result if run again (rejected by the script, SQL errors, a client that no longer exists). It is not cleaned up automatically, and never replayed |
| Seeing what is damaged | Files ending in .corrupt and truncated.log: left by power loss or disk problems |
| Maintenance downtime | SIGTERM stops replay; data stays on disk and continues at the next start |
| Replacing a node | Wait for this node’s spool to empty (the metric at 0) before shutting it down, or whatever it has not replayed stays on the old disk |
| Changing settings | Both spools’ settings need a restart to apply. Before removing a [proxy.realms] entry, make sure the forwarding spool is empty |
Spool files are raw packets, with user names and addresses. The directory’s mode is 0700; take it into account for backups and when disposing of disks.
Investigating one user’s problem
1. Look at the authentication log first. One line per login:
authentication protocol="radius" client="wlc" source=10.30.0.10 user="[email protected]" account="staff1" chain="default" handler="sql" decision="reject"
user is the name the device sent, account is the name used for the account lookup, and chain and handler say who decided.
2. When you need the full packets, turn on tracing for a while:
[log.trace]
users = ["staff1"]
minutes = 30
After SIGHUP, this user’s requests and responses are written to the log in full (passwords shown as <hidden>), stopping automatically after 30 minutes. clients = ["wlc"] traces one device instead. The log will contain personal data; remove this section when done.
3. Script problems show as Lua script error (<protocol>/<phase>): <file>:<line>: <message>. Scripts can print their own messages with log(...).
For what common messages mean, see usage section 8.
Upgrading
- Read the new version’s change notes, especially items marked as upgrade notes.
- Run
--checkwith the new version against your current configuration. - One node: stop → replace the binary → start. A cluster: one node at a time; see chapter 8.
Backups
rradius has no built-in backup command. The cluster itself keeps three copies and survives losing one host, but data deleted by mistake is replicated to every node, so keep separate copies anyway:
- The source of accounts must be rebuildable: keep migration files in version control; when accounts come from a directory or another system, that is the original.
- Data that exists only in rradius, such as accounting, is exported regularly by an external process to the billing system or a data warehouse.
- Key files are not in the database and need separate backups:
aka.key(chapter 13): losing it means importing every card again.dpsk.key(chapter 12): losing it means everyone has to reset their Wi-Fi passphrase.ticket.seal(chapter 15): losing it loses no data; generate a new one, and devices do one full handshake.
- Keep key file backups apart from database backups. Kept together, whoever gets the backup has both the ciphertext and the key.
- Configuration files, scripts and certificates go into version control or a configuration management tool; private keys and
*.secretare kept separately, however you manage secrets.