This chapter starts from “what you see” and tells you where to look first. The “Common problems” at the end of each chapter cover problems specific to that scenario; this one covers problems across scenarios. The full table of messages is in usage section 8.
Have these four ready
| Tool | What it shows | How to use it |
|---|---|---|
| The authentication log | The result of every login, and who decided | Log lines starting with authentication protocol=..., present by default |
| Metrics | What leaves no log line: dropped packets, overload, upstream state | curl -s http://127.0.0.1:9812/metrics, with [metrics] set |
| Tracing | The full requests and responses of one user or one NAS | [log.trace] plus SIGHUP; see chapter 19 |
--check | Problems in the configuration file, certificates or script syntax | rradius-server --check --config ..., without affecting the running service |
The authentication log records only requests that reached a decision. Dropped packets (unknown source, bad signature, overload, retransmissions) leave no log line; look at the metrics. This is the most common misunderstanding: nothing in the log does not mean the packet never arrived.
The NAS sent a request and got no reply at all
rradius deliberately does not reply in many cases (a reply would give an attacker information, or make the NAS think the password was wrong). First see which label of this metric is rising:
curl -s http://127.0.0.1:9812/metrics | grep -E 'rradius_radius_(discarded|duplicates|packets)_total'
| What is rising | Meaning | What to do |
|---|---|---|
| Nothing moves | The packet does not reach rradius | The firewall, routing, the address and port set on the NAS; whether [radius] bind listens only on 127.0.0.1; confirm with tcpdump -ni any udp port 1812 |
discarded{reason="unknown_client"} | The source address is not in [[radius.clients]] | The log shows unknown client with the source address. When the NAS is behind NAT, rradius sees the translated address. With dynamic clients, see chapter 14 |
discarded{reason="invalid"} | Wrong shared secret, bad format, or no Message-Authenticator | When the log shows lacking a Message-Authenticator: old devices do not send this attribute; set require_message_authenticator = "no" or "auto" on that client (chapter 1). Otherwise check the shared secret, watching for extra whitespace at the end of the file |
discarded{reason="overload"} | Requests handled at once reached the limit | See “Logins are slow, or only sometimes work” below |
duplicates | Retransmissions: the original request is still being handled | Handling is too slow for the NAS to wait. See “Logins are slow, or only sometimes work” below |
packets rises but there is no reply | The request was handled, but the decision was not to reply | Forwarded by realm and no upstream answered (chapter 9); the accounting script did not return 'accept'; EAP’s max_sessions_per_device is full |
Use Status-Server to separate “network and secret” from “the account”:
echo 'Message-Authenticator=0x00' | radclient -t 2 -r 1 <address>:1812 status <shared secret>
A reply means the network works, the source is recognised and the shared secret is right, so the problem is in the request itself. No reply means the problem is one of those three. Send this from a host in the client list.
Logins are rejected
Find that login’s line in the authentication log and look at handler and account:
authentication protocol="radius" client="wlc" source=10.30.0.10 user="[email protected]" account="staff1" chain="default" handler="sql" decision="reject"
handler | Meaning | Common causes |
|---|---|---|
sql | The sql handler rejected | Wrong password; the password column is NULL or empty; more than one row found; the password’s format cannot be used with this login method (CHAP needs plain text, MS-CHAPv2 plain text or {nthash}) |
ldap | The ldap handler rejected | Wrong password, not in required_groups; when the directory cannot be reached or the service account fails to log in, there is also a log message starting with auth ldap handler: |
lua, script | A script rejected | The script’s logic, or a script error (the log shows Lua script error, with the file and line) |
none | Every handler skipped | The account does not exist (sql found nothing, ldap did not find it). First check that account is the name you expect |
no_password_check | Nobody in an all_accept chain checked a password | The account is not in sql or ldap; only a later script said yes |
normalization | The name was stopped before the chain | The name was empty after processing, or does not match [auth.username] allowed |
eap-tls, eap-peap, eap-ttls, eap-aka-prime | The EAP conversation ended in a reject | The line before usually has EAP conversation with client ... failed or refused with the reason; see “Wi-Fi (802.1X)” below |
dpsk | A per-person passphrase | Nobody’s passphrase matched (account is empty), or it matched but that person is disabled (chapter 12) |
A realm name, with chain="proxy" | The upstream rejected | Investigate on the upstream’s side |
The password is definitely right:
- Is
accountthe same as the name in the table? Case, and whether it carries a realm. See[auth.username](usage section 3.6). - Does the log show
RADIUS response attribute error? That means a reply attribute’s name or value is wrong (reply_query,group_reply, the script’sset_attr), which rejects even with the right password. - Was the log line perhaps not written by this rradius at all? With several nodes, look on each.
- Was a name with
@forwarded by[proxy]? Thenchainisproxy.
No log line for this login at all: go back to the previous section; it was dropped. Or [log] auth = false has turned the log off.
Logins are slow, or only sometimes work
| What to look at | Means |
|---|---|
RADIUS authentication overloaded: discarded N packets in the log | Requests handled at once reached max_auth_inflight (4096 by default). This is the effect, not the cause: something is making each request slow |
rradius_radius_duplicates_total rising | The NAS is retransmitting; handling takes longer than its retransmission interval |
sql_query timed out or sql_execute timed out in the log | The database is slow, or the cluster has no majority. Such logins get no reply (rradius_radius_discarded_total{reason="database"}), and the NAS retransmits or asks another node |
Timeouts from auth ldap handler: in the log | The directory is slow or unreachable |
rradius_script_http_requests_total{result="timeout"} rising | An API called by a script is slow (chapter 18) |
rradius_proxy_requests_total{result="timeout"} rising | An upstream is slow or down |
Finding the slow part:
- Password hashing: bcrypt and argon2 are deliberately slow, verifying no more at once than there are CPU cores, with the rest queueing. At peaks, a few hundred logins a second already queue. For high login volumes, switch to
sha512-crypt(chapter 19). - Writes: scripts that write to the database on every login (quotas, IP assignment, two-step verification) go through the cluster and are slower than reads alone. Right after the cluster loses its majority, writes fail only after
[lua] query_timeout_ms; after more than 5 seconds without a leader they fail at once. These logins all get no reply rather than a reject. - The first upstream does not answer: when forwarding by realm, every request first waits for the first server to time out before trying the next, until it is marked down.
probe_interval_msnotices earlier (chapter 11). - Raising
max_auth_inflightusually does not help. The NAS gives up after a few seconds, and a longer queue just means more wasted work. Fix the cause of the slowness first. - Accounting slowing logins down: the number of accounting scripts running at once is limited (CPU cores minus 2), leaving threads for logins; but both share the database, so heavy accounting scripts still affect logins.
Accounting keeps being retransmitted, or accounting is missing
The NAS keeps retransmitting the same accounting: the accounting script did not return 'accept', so rradius did not reply.
Lua script error (radius/accounting)in the log: the script failed; look at the file and line.Lua script did not accept accountingin the log: the script ended normally but returned something else. Packets you do not want to record must also return'accept'.sql_execute timed outin the log: the database cannot take writes. With the accounting spool set up, this case is written locally first and then answered (chapter 7).
Accounting is missing:
| Check | Means |
|---|---|
rradius_acct_spool_bytes above 0 | Still in the spool waiting for replay, not yet in the database. It empties by itself once the database recovers and the script works |
rejected.log in the spool directory | Requests given up during replay: rejected by the script, SQL syntax or constraint errors, a client that no longer exists. The file is not cleaned up automatically, and never replayed |
Files ending in .corrupt or truncated.log in the spool directory | Left by power loss or disk problems; the requests in them were not replayed |
rradius_acct_spool_full_total rising | The spool is full or writing failed; these requests got no reply and rely on NAS retransmission |
RADIUS accounting overloaded in the log | max_acct_inflight was reached, and packets were dropped |
| The accounting is forwarded | Look at rradius_proxy_acct_spool_* (chapter 11) |
The spool never goes down (rradius_acct_spool_bytes stays above 0): replay is stuck on the first record. The database has not recovered, or the script fails every time. Look at log messages starting with Lua script and accounting spool; once the script is fixed and SIGHUP sent, it continues by itself.
The same accounting was recorded twice: the script is not safe to repeat. When a NAS updates Acct-Delay-Time and retransmits, it uses a new Identifier, so rradius cannot tell it is a retransmission; spool replay can also run it again. Writes must use INSERT ... ON CONFLICT or INSERT OR IGNORE (chapter 1).
Wi-Fi (802.1X)
| Symptom | Cause |
|---|---|
EAP conversation with client ... failed in the log, with a reason mentioning State | The second half of the conversation reached another node. One login must stay on one node: primary/backup on the NAS, or a load balancer pinning by source (chapter 8) |
... failed with a TLS error as the reason | The client broke off because it does not trust the server certificate (no CA on the device, a server name mismatch), or TLS versions or cipher suites do not match |
... refused | The certificate (or card) passed, but the name does not match, or the chain did not accept it: certificate_query found nothing, or ldap has no certificate_filter |
the chain did not accept the inner name | Wrong PEAP or TTLS password, or no such account |
the chain accepted the inner name without checking a password | The handler that accepted is a lua handler without verifies_password |
this client has no chain to check a password with | PEAP and TTLS need an authentication chain, and this client has none |
| PEAP keeps failing, while a PAP test works | The password is not plain text or {nthash}; or ldap uses bind mode |
| The client picks a method you did not want to offer | That client’s eap_methods lists it. To accept only certificates, list only ["tls"] |
rradius_eap_sessions_total{result="timeout"} is high | Clients did not finish: users cancelled after a certificate warning, a poor signal, or the NAS’s EAP timeout is too short |
result="evicted" keeps rising | Conversations at once exceed max_sessions or max_sessions_per_client. When every user goes through the same controller, raise the latter |
result="overload" | One device opened too many conversations at once (max_sessions_per_device), or EAP-AKA’ rate limits |
| Revocation and session resumption problems | Chapter 15 |
The inner name cannot be seen: in the authentication log for PEAP and TTLS, user is the anonymous outer name, and account is the real account. [log.trace] matches outer names, so trace the whole NAS with clients = [...], or list the outer names actually used.
TLS connections (RadSec, TACACS+ over TLS, RadSec upstreams)
| Symptom | Cause |
|---|---|
RadSec handshake with ... failed | The other side has no certificate, or it is not signed by client_ca_file, revoked or expired, or the TLS versions do not match |
| The handshake succeeds but the connection is closed at once | The certificate passed but matched no client: see whether the log says fits no client, is not valid for client ...'s tls_name or needs tls_name |
| Every device fails to connect at the same time | The server certificate expired; crl_file is missing a CA; ocsp = "hard" and the responder is down; RadSec has reject_expired_crl on and the CRL expired |
| A replaced certificate or CRL does not take effect | SIGHUP is needed, on every node |
proxy server ...: cannot connect ... TLS: | Connecting to an upstream failed. The rest of the message says whether the other side’s certificate is not trusted by ca_file, is not valid for tls_name, or the other side does not accept the certificate you presented |
accepting a connection failed N times | File descriptors ran out. Raise ulimit -n, or lower max_connections |
openssl s_client can check the handshake on its own; see chapter 10.
Forwarding by realm
| Symptom | Cause |
|---|---|
no proxy server answered a request for realm ... | No upstream gave a valid answer: down, a firewall, a wrong shared secret, or the upstream’s answer had no Message-Authenticator |
proxy server ... is not answering on ... | This server was marked down, and the next is used for now. Later a Status-Server checks whether it is back |
| An upstream is marked down after a mistyped password | timeout_ms is shorter than the upstream’s deliberate delay before rejecting |
a request for realm ... came back to this server | The forwarding configuration loops |
| Local accounts are forwarded | There is a * or an entry covering your own domain; with DNS discovery, your own realms are not listed in local_realms |
| Every request for some realm is dropped | The DNS lookup failed (chapter 11) |
Disconnects (Disconnect, CoA)
The Error-Cause of a NAK states the reason directly:
| Result | Meaning |
|---|---|
| No reply at all | The sender is not in [[radius.coa.clients]], the shared secret is wrong, or more than 256 requests at once. rradius_coa_relay_discarded_total increases |
| NAK 502 | No NAS found: the request’s NAS-IP-Address or NAS-Identifier matches no client; or a NAS recognised only by certificate has no connection open |
| NAK 503 | Only a user name was given, and session_query found no open session |
| NAK 505 | The NAS gave no valid answer (CoA not enabled, wrong port, rradius’s source not accepted, a different shared secret); session_query failed; or the request contains attributes that must be encrypted |
NAK with another Error-Cause | The NAS’s own answer, passed back as is |
send_disconnect and send_coa returning "timeout", and rradius-server disconnect printing timeout, have the same causes as the first item of 505. The command sends from the host it runs on, so the NAS must accept that source address.
TACACS+
| Symptom | Cause |
|---|---|
The connection is closed at once, with TACACS+ connection from unknown client in the log | The source is not in [[tacacs.clients]] |
Authentication answers ERROR, with a body did not decode, so the shared secret may be wrong in the log | The shared secrets differ. rradius_tacacs_malformed_total increases too |
| Logins work, but every command is refused | The authorization script refuses, or no rule of the authorization group holds (chapter 6) |
enable always fails, with TACACS+ enable request refused in the log | Enable is refused by default. Do not put rradius in the device’s enable methods, or set enable = "login" |
| CHAP and MS-CHAPv2 logins get RESTART | This client has no authentication chain, or the chain only has a script without verifies_password |
| Logins keep answering ERROR | More than 8 unfinished logins on one connection, or a single login taking more than 60 seconds |
TACACS+ overloaded: closed N connections | max_connections was reached. When devices open connections and never close them, lower idle_timeout_ms |
| TLS problems | Chapter 16 |
Startup fails
Look at the exit status:
| Status | Meaning | Common causes |
|---|---|---|
| 2 | The configuration is wrong, and nothing has started | The message names the file and field: a misspelt field, a missing file, a script syntax error, a certificate and key that do not match, key file permissions too broad, problems in migration file names or contents |
| 1 | The configuration is fine, but the environment does not allow it | The RADIUS UDP ports or the CoA port cannot be bound (in use, or no permission; the message gives the address and port); the cluster was not ready within migrations_wait_ms (other nodes not up, or unreachable) |
- When TACACS+, RadSec or metrics ports cannot be bound, the process does not exit; it logs one error line, and other services run as usual. So with TACACS+ on port 49 and no permission, the process looks healthy but devices cannot connect; after startup, confirm the log has
listeningfor every port. migration file '...' changed after it was applied: an applied file was changed. Restore its original contents, and add a file with the next number for changes. In development you can stop the server, deletedata/and start over.- A node keeps exiting with status 1, with
cluster was not ready to apply migrationsin the log: the majority of nodes is not online yet; ormigrations_wait_msis shorter than the slowest SQL file takes to run. Raft node has not been initializedkeeps appearing: the node has not joined the cluster yet. Have the other nodes started? Can they reach each other’s raft and api ports? Israft.secretthe same?--checkbefore starting catches most of the status 2 problems, but it does not connect to the database or directory, so wrong SQL or a wrong LDAP password only shows up after startup.
A reload did not take effect
reload failed, keeping the running configuration: ...in the log: the new configuration has an error, and the service keeps using the old one. Fix it and send SIGHUP again.rradius_reload_total{result="error"}increases.these need a restart to apply: ...in the log: the settings listed changed but apply only after a restart; the others have been applied. The list is in chapter 19.- Only one node took effect: each node reloads on its own; files must go on every node and the signal to every node.
- The dictionary file changed: the dictionary is not reread; restart.
- The migration directory changed: new files are applied only at startup.
The cluster
| Symptom | Cause |
|---|---|
| Login results on one node differ from the others | That node’s migrations/, scripts or configuration file differ from the others’; or it is partitioned (rradius_cluster_leader is 0) and reads stale data |
| Some logins on one node get no reply | rradius_radius_discarded_total{reason="database"}: it cannot reach the cluster database; reason="no_leader": stop_without_leader_ms is set and it has been without a leader too long (chapter 8) |
| A restarted node takes a while to start answering | Normal: it waits for its local replica to catch up with the cluster before answering, logging has caught up with the cluster. After a long downtime the catch-up takes longer, up to migrations_wait_ms |
| Data just written cannot be read | sql_query reads the local replica, which may lag slightly behind writes. To know whether a write succeeded, use sql_execute’s return value instead of querying again |
| Every write times out | The majority was lost (one of three left). Read-only logins work as usual; logins that need to write get no reply and TACACS+ answers ERROR; accounting goes to the spool when there is one, until the nodes come back |
A node exits by itself with status 1, with exiting so that a restart recovers it in the log | Its Raft core stopped (commonly because the cluster wrote more than snapshot_every while it was disconnected), or hiqlite panicked internally. Restarting it lets it catch up; with automatic restarts set up there is nothing to do (chapter 8) |
| Stopping a node takes about 10 seconds | Normal: with several nodes, hiqlite waits 9.5 seconds before closing |
| Restarted while it was the last node, it never comes up (exiting with status 1 when there are migrations) | A restarted node waits for a leader. Bring the other nodes back first (chapter 8) |
| Startup is slow after an abnormal exit | The writes since the last snapshot have to be replayed. See snapshot_every in usage section 2.2 |
| Features that behave differently on each node | The accounting spool, EAP conversations, sessions without shared_tickets, results remembered for dynamic clients and per-person passphrases, RadSec connections, and tracing timings all belong to each node |
The log itself
rradius_log_dropped_lines_totalrising: the log is written faster than its destination takes it, or writing fails (a full disk, the receiving process exited). Dropped lines are gone; the service is unaffected. Check where stderr goes; with heavy traffic, turn off[log] auth, or do not trace whole NAS.- Messages ending in
(N since the last line): messages anyone can trigger (unknown sources, failed handshakes) appear at most once per second of each kind, and N is the total for that period. - WARN and ERROR from
hiqlite: a few lines appear when the cluster starts and stops, such asNo metadata exists inside the DB yet, and are normal. - Lines starting with
script ...: ...are written by scripts withlog(), and are not errors. - Every message goes to stderr. The results of management commands (
user,disconnect,coa) go to stdout.