docs: customer doc import HIVE 3360/3361 Monitoring section - nav update, add new fileis (tools)
Updated nav and logging files
*** xref:vendors/cisco.adoc[Cisco]
*** xref:vendors/proxim.adoc[ProxIM]
-** Optimization
+** xref:monitoring/optimize.adoc[Optimization]
*** xref:monitoring/index.adoc[Monitoring]
+**** xref:monitoring/logging_examples.adoc[Log Examples]
**** xref:monitoring/statistics.adoc[Server Statistics]
*** xref:tuning/performance-testing.adoc[Performance Testing]
+*** xref:monitoring/tools/index.adoc[Tools]
+**** xref:monitoring/tools/radclient_tool.adoc[Radclient]
+**** xref:monitoring/tools/radsniff_tool.adoc[Radsniff]
+**** xref:monitoring/tools/radmin_tool.adoc[Radmin]
*** xref:tuning/tuning_guide.adoc[Tuning Guide]
Any good systems administrator will want to know how well
their systems are operating, both to catch issues before they
become a serious problem, or for long term analysis.
+
The term "monitoring" can encompass all kinds of watching how the
system is working, from generating and watching logs, gathering
statistics or ensuring that the service daemon is still running
and serving requests.
-We break the different types of monitoring down into the following
-sections.
+Monitoring is organized into the following sections:
== Service checking
* Sending Status-Server RADIUS requests
Within a proxy environment FreeRADIUS needs to know if upstream
-proxies are available. It can do this itself using the latter two
-options above.
+proxies are available. It can do this itself by issuing request as outlined in the options above.
== Logging
System logs are often the most critical part of a RADIUS system.
-They are necessary for the administrator to know who has logged in
+Logs are necessary for the administrator to know who has logged in
and when, for debugging purposes such as when an end user cannot
connect, and often for regulatory or compliance purposes.
-RADIUS server logs are also often used as a basic form of
-recording accounting requests, which are in and of themselves a
+FreeRADIUS server logs are also often used as a basic form of
+recording accounting requests, which are also a
form of logging by the NAS. Getting correct logging systems
operational is key to running an efficient and easy to maintain
-RADIUS server.
+FreeRADIUS server.
FreeRADIUS has many options for being able to generate and store
logs, including the following:
-* Main daemon logging, configured in `radiusd.conf`
-* Line-based text logging, using `rlm_linelog`
-* Detailed RADIUS packet logs, using `rlm_detail`
+* Main daemon logging, configured in xref:reference:raddb/radiusd.conf.adoc[`radiusd.conf`]
+* Line-based text logging, using xref:reference:raddb/mods-available/linelog.adoc[`rlm_linelog`]
+* Detailed RADIUS packet logs, using xref:reference:raddb/mods-available/detail.adoc[`rlm_detail`]
-As well as recording direct to disk, the above can be sent via a
+As well as recording direct to disk, the options above can be sent via a
local syslog server, which opens up many opportunities for central
logging.
It is possible to integrate FreeRADIUS into other more complicated
logging systems, some options may include:
-* To CSV files, for example via `rlm_linelog`
-* Writing entries to an SQL database using `rlm_sql`
+* To CSV files, for example via xref:reference:raddb/mods-available/linelog.adoc[`rlm_linelog`]
+* Writing entries to an SQL database using xref:reference:raddb/mods-available/sql.adoc[`rlm_sql`]
* Into a log management system such as Elasticsearch or Graylog
== Statistics gathering
It is often useful to collect statistics from a running RADIUS
-server. These are often plotted on graphs to show current load or
+server. The data collected can be displayed on graphs to show current load or
for trend analysis, as well as an indication of system operation.
Statistics are usually gathered in two ways:
--- /dev/null
+= Log Examples
+
+FreeRADIUS logging provides many sources of runtime service information that can be useful for understanding a problem.
+Issues are frequently evident in log files long before they present a problem in production.
+
+
+== Server log: /var/log/freeradius/radius.log
+
+This log contains global server events, such as attempts to add new clients, attempts to open new connections, slow requests and server faults.
+There are thousands of possible log entry messages, many of which are dynamically generated, and for the most part they are self-describing.
+Therefore only common log entry messages are documented.
+
+=== Being polled by the monitoring server
+
+Munin, a resource monitoring tool, is connecting to the freeradius process to retrieve the latest packet statistics.
+
+```
+... Info: ... adding new socket command file .../monitor.sock
+... Info: ... shutting down socket command file .../monitor.sock
+```
+
+=== Module opening a connection
+
+In this example, the sql_auth instance of the rlm_sql module is opening an additional connection to the database. Since the server was last restarted, nine connections have been opened (indicated by the connection number (8) which is based at zero).
+
+Currently 1 of 24 pending connection slots are being used. The 24 is the number of connections left until the connection pool has opened the maximum number of allowed connections.
+
+The 1 shows how many connections the server is trying to open at the same time. A high number here means that there’s likely been a large and sudden increase in load and that the server is trying to open lots of new connections simultaneously to cope with the sudden increase.
+
+```
+... Info: rlm_sql ...: Opening additional connection (8), 1 of 24 pending slots used
+```
+
+=== Module closing a connection
+
+The server recognized that there are surplus connections open to a given server and is therefore closing a spare connection.
+The way the server determines that there is a surplus is by concentrating new queries on as small a subset of the connection pool as possible and then closing connections that haven’t been used for a period.
+
+```
+... Info: rlm_ldap ...: Closing connection (1), from 7 unused connections
+
+```
+
+=== Request from unknown client
+
+This message states that the request was ignored since the client is unknown. If this client is a valid source of authentication and/or accounting requests then it should be added to the configuration.
+If the clients are added dynamically, e.g. from SQL (via the nas table) or a dynamic_clients file then it should be determined why the addition is failing.
+
+```
+... Error: Ignoring request to auth address * port 1812 bound to server \
+default from unknown client 203.0.113.7 port 54980 proto udp
+```
+
+=== Database locking issue
+The database encountered a deadlock while placing a lock on data to prevent conflicting updates or selects of duplicate/stale data. If this happens frequently then the database (or cluster) may be suffering from performance problems.
+
+```
+... ERROR: (1526654) sqlippool: ERROR: rlm_sql_mysql: \
+ERROR 1213 (Deadlock found when trying to get lock; \
+try restarting transaction): 40001
+```
--- /dev/null
+= Optimization
+
+Once the FreeRADIUS server is successfully installed, optimizing your eco-system, particularly your FreeRADIUS server, becomes a top priority. A well-tuned server is critical for ensuring high performance, robust security, and reliability. These factors are especially important in environments with many users or high authentication traffic. This process involves tuning various components such as datastore performance, network connections, and server resources.
+
+It can be challenging to identify issues or what component needs adjusting. To start, you need to gather statistics through monitoring to identify the bottlenecks in your system. This important activity allows you to determine the most critical areas for improvement.
+
+== Why Optimize?
+
+A poorly optimized FreeRADIUS server can lead to slow authentication times, timeouts, and even system instability. Enhance your FreeRADIUS server’s performance, reliability, and resource efficiency to enhance the user experience while maintaining a secure environment.
+
+=== Performance
+
+Optimization ensures fast authentication and accounting processes, preventing delays and timeouts. A well-optimized FreeRADIUS server can handle a large number of simultaneous users and authentication requests without performance degradation.
+
+=== Reliability
+
+Optimizing resources and configurations helps maintain a stable and robust RADIUS server, reducing the risk of crashes or outages. Optimizing FreeRADIUS helps ensure consistent and reliable network access, minimizing disruptions.
+
+=== Resource Efficiency
+
+Optimizing resource usage (CPU, memory, disk I/O) allows you to run FreeRADIUS effectively even on limited hardware. If FreeRADIUS is integrated with a datastore, implementing well-structured querires prevents bottlenecks.
+
+The optimization section is organized by the following topic areas:
+
+== xref:monitoring/index.adoc[Monitoring]
+
+Monitoring the FreeRADUS server and network components includes observing the AAA processes and how the overall system is operating. Several logging options help the administrator generate statistics to determine where issues are occurring such as slow connectivity or authentications.
+
+FreeRADIUS is packaged with a xref:monitoring/statistics.adoc[virtual statistics server] that enables you to select what you want to watch or help find a problem.
+
+== xref:tuning/performance-testing.adoc[Performance Testing]
+
+Performance testing helps you figure out what the maximum loads can be without affecting operations. As your network grows, a non-optimized server may struggle to handle the increased authentication and accounting loads, especially during peak hours.
+
+== xref:monitoring/tools/index.adoc[Tools]
+
+FreeRADIUS is packaged with some useful tools such as radsniff and radclient that are used in testing, monitoring, and gathering statistics. These tools give you a top-level view of the health of your server, clients, and processess.
+
+== xref:tuning/tuning_guide.adoc[Tuning Guide]
+
+Tuning the FreeRADIUS server and relevant components ensures optimal performance across the network.
=== Global server authentications
-Using `FreeRADIUS-Statistics-Type = 0x01` requests stats about
+Using `FreeRADIUS-Statistics-Type = 0x01` requests statistics about
authentications. Because, for example, no "Client" qualifier has
been added (`0x20`) the numbers are global to the server.
Data can be provided about each RADIUS client defined in the
server. Note that this is for the client definition, not for each
client that connects - if a client definition has a wide netmask
-and permits multiple clients to connect, the statistics will be
-aggregate for all clients using that definition.
+and permits multiple clients to connect, the statistics will be an
+aggregate sum for all clients using that definition.
[NOTE]
====
--- /dev/null
+= Tools
+
+FreeRADIUS comes with a set of useful tools that assist you in monitoring and collecting statistics for your system. While there are many third-party tools that offer similar functionality, the reliable and proven tools included with FreeRADIUS are the ones to use.
+
+Each tool has a specific purpose and is designed to work seamlessly together. These tools include:
+
+== xref:monitoring/tools/radclient_tool.adoc[radclient]
+Radclient enables you to setup mock clients to perform basic authentication testing. The radius.client file is used to popoulate the list that is read by the radclient tool.
+
+== xref:monitoring/tools/radsniff_tool.adoc[radsniff]
+Radsniff allows you to inspect and process any type of RADIUS packet that's on the network. This tool can be used in conjunction with th radclient tool and as well with performance testing.
+
+== xref:monitoring/tools/radmin_tool.adoc[radmin]
+Radadmin is a administration tool designed to administer and interact with a running FreeRADIUS server. It enables users to monitor statistics, view configuration, and make changes without the need to restart the server. FreeRADIUS Server is an that connects to the control socket of a running server, providing a command-line interface to manage it.
--- /dev/null
+= Using radclient
+
+Policy issues for basic authentication protocols such as PAP, CHAP and MSCHAP, as well as for accounting and CoA/Disconnect requests, can be investigated using the radclient command whilst the server is in debug mode.
+Firstly, create an authentication packet definition by specifying its attributes. For example:
+
+```
+nwkrad@radius-fe-01$ cat <<EOF > auth_request.txt
+Acct-Session-Id = "00000042"
+Service-Type = Framed-User
+Framed-Protocol = PPP
+Calling-Station-Id = "AA-BB-CC-DD-EE-FF"
+User-Name = "bob"
+User-Password = "testing123"
+NAS-Identifier = "nas01"
+NAS-IP-Address = 192.0.2.1
+NAS-Port = "12345"
+NAS-Port-Type = Virtual
+EOF
+```
+You will need to customise the above to reflect the attributes that are being sent by your NAS. You can passively determine the format of your RADIUS packets by running radsniff -x as root on the RADIUS server:
+`root@radius-fe-01# radsniff -x`
+
+Next, generate a RADIUS access request packet from this definition and send it to the RADIUS server by calling radclient with the auth option:
+```
+nwkrad@radius-fe-01$ cat auth_request.txt | \
+radclient -x 127.0.0.1 auth testing123
+```
+
+Carefully examine the output of the above command and the debug output from FreeRADIUS. Make any required changes in small careful steps, regularly committing the changes to versions control so that they can be reverted if necessary.
+
+The process for generating accounting requests is similar. Firstly, create the accounting packet definition:
+
+```
+nwkrad@radius-fe-01$ cat <<EOF > acct_request.txt
+Acct-Session-Id = "00000042"
+Acct-Status-Type = Interim
+Acct-Authentic = RADIUS
+Acct-Delay-Time = 0
+Acct-Input-Octets = 3656340066
+Acct-Output-Octets = 3424317634
+Acct-Session-Time = 1215315
+Acct-Input-Packets = 45922570
+Acct-Output-Packets = 74246191
+Acct-Input-Gigawords = 2
+Acct-Output-Gigawords = 22
+Calling-Station-Id = "AA-BB-CC-DD-EE-FF"
+Framed-Protocol = PPP
+Framed-IP-Address = 10.33.44.55
+User-Name = "bob"
+Hermod Retail - Operations guide 13
+Service-Type = Framed-User
+NAS-Identifier = "nas01"
+NAS-IP-Address = 192.0.2.1
+NAS-Port = "12345"
+NAS-Port-Type = Ethernet
+EOF
+```
+
+Then call radclient with the acct option to generate the actual accounting request:
+```
+nwkrad@radius-fe-01:~$ cat acct_request.txt | \
+radclient -x 127.0.0.1 acct testing123
+```
--- /dev/null
+= Using radmin
+
+It's possible to retrieve debug logs from FreeRADIUS while it is running in normal multi-threaded mode by using the radmin tool. This has the benefit of not interrupting the normal operations of FreeRADIUS server. It also allows for selective logging of packets which match a specified criteria.
+
+== Getting Started
+
+Enter the following commang to launch radmin:
+
+`root@radius# radmin -f /var/run/freeradius/control/control.sock`
+
+At the radmin prompt, set the debug file. By default, it's located in the FreeRADIUS log directory /var/log/freeradius/:
+
+```
+radmin> debug file tmp/debug
+radmin> show debug file
+/var/log/freeradius/tmp/debug
+radmin>
+```
+
+Next, set a condition for the packets you want to capture in the debug logs. The debug logs will contain only the traffic type that you're investigating instead of all the traffic packets. The conditions are specified using FreeRADIUS unlang conditions.
+
+=== Debug Output using radmin set with Calling-Station-ID condition
+
+```
+radmin> debug condition'Calling-Station-Id == "aa-bb-cc-dd-ee-ff"'
+radmin> show debug condition
+(Calling-Station-Id == "aa-bb-cc-dd-ee-ff")
+```
+
+=== Debug Output using radmin set with User-Name condition
+
+`radmin> debug condition'User-Name =~ /^abc/'`
+
+=== Debug Output using radmin set with multiple conditions
+
+The current condition is the last one specified; to match multiple items, the criteria must be combined into one expression.
+
+```
+radmin> debug condition'(User-Name =~ /^abc/') || \
+(Calling-Station-Id == "aa-bb-cc-dd-ee-ff")'
+```
+
+Now you can see the debug output being written to the file specified:
+
+`root@radius# tail -F /var/log/freeradius/tmp/debug`
+
+[NOTE]
+====
+The debug logs can grow quickly and get very large on busy systems. Only run debugging when needed and watch the disk space carefully.
+====
+
+=== Stop the capture
+
+Once you've have captured the debug output, turn the capture off with the following command:
+
+```
+radmin> debug condition
+radmin> show debug condition
+```
+
+Wait a few seconds for any outstanding requests to complete and the last writes to the debug file.
+
+
+=== Clear debug file
+
+Next, Clear the debug file setting to complete your session.
+
+```
+radmin> debug file
+radmin> show debug file
+```
--- /dev/null
+= Using radsniff
+
+The radsniff tool is extremely useful for debugging RADIUS packet flows, either by monitoring the live network interfaces or by processing a PCAP-based traffic dump.
+
+To see the different switches that can be used, see the xref:reference:man/radsniff.adoc[radsniff] man page included with this documentation.
+
+== Packet capture processing
+
+=== Live packet capture
+
+In it’s default mode radsniff will display incoming and outgoing RADIUS packets, one per line, annotated with timing information and “events”. To view your real-time transmission of packets, enter the following command:
+
+`root@radius# radsniff`
+
+For the first few seconds after entering the above command, you may see noreq events. These events occur because the request that solicited the response was sent before radsniff was started. If such events are seen a while after startup it most likely means that packets are not being seen because the capture is lossy which may happen in the event that the system is overloaded.
+
+.Normal Output
+
+```
+... (440) Access-Request Id 231 eth0:2.3.4.5:1812 -> 192.0.2.100:1812 +4.587
+... (441) Access-Accept Id 114 eth0:2.3.4.5:1812 <- 192.0.2.100:1812 +4.587 +0.101
+... (442) Access-Request Id 76 eth0:2.3.4.5:1812 -> 192.0.2.100:1812 +4.592
+... (443) Access-Request Id 74 eth0:2.3.4.5:1812 -> 192.0.2.100:1812 +4.601
+```
+
+Requests that do not receive a response will be marked with norsp.
+
+[NOTE]
+====
+For any servers using bonded interfaces, the -i bond<num> option must be used, otherwise radsniff marks all the packets as retransmitted, i.e. received once via the LACP member link and once via the bond interface. As well, servers using a VIP for the RADIUS service marks all packets as retransmitted unless -i <iface> is added referring to either the label for the VIP or the real interface the VIP is attached to.
+====
+
+== Viewing RADIUS attributes
+Attributes from the RADIUS packets can be shown using the -x option:
+
+`root@radius# radsniff -x`
+
+.RADIUS Attributes Display Example
+
+... (66) Access-Request Id 159 eth0:2.3.4.5:1812 -> 192.0.2.100:1812 +1.142
+User-Name = 'bob'
+CHAP-Password = 0x0128cd38aacd802092a8c46d5bb84f61ed
+NAS-IP-Address = 2.3.4.5
+NAS-Port = 17828
+Service-Type = Framed-User
+Framed-Protocol = PPP
+Calling-Station-Id ='30:b5:c2:4a:85:70'
+NAS-Identifier = 'BRAS-01'
+CHAP-Challenge = 0x84926cafcfb2446fabbce4b14d6d6337
+NAS-Port-Type = Ethernet
+Acct-Session-Id = 'd55c079771'
+NAS-Port-Id = 'MSAN-01 atm 0/1/2:3.4'
+Authenticator-Field = 0x84926cafcfb2446fabbce4b14d6d6337
+... (67) Access-Reject Id 146 eth0:2.3.4.5:1812 <- 192.0.2.100:1812 +1.152 +0.104
+Reply-Message ='Simultaneous use limit exceeded.'
+Authenticator-Field = 0x13ab6ed86c597041070250ae5dbcc920
+
+=== Live traffic statistics
+
+To view real-time traffic statistics (rate, latency, success rate) continuously, execute `radsniff -W` to capture statistics at timed or preset intervals:
+
+`root@radius# radsniff -W <sample_interval>`
+
+.Relevant counters
+[options = "1,3"]
+|===
+|Counter Name|Description
+
+|Access-Accept (Total)
+|Shows the current rate of Access-Accept packets
+
+|Accounting-Response (Total)
+|Shows the current rate for accounting packets.
+
+|Retransmits / Loss
+|Shows the number of requests that were sent but did not receive responses indicating that the RADIUS server is overwhelmed, most likely due to an authentication dependancy (e.g. SQL database, LDAP directory, credential cache) not responding in a timely manner.
+
+|RT(0)
+| means “retransmits 0”, i.e. the first time a request was sent it received a response. If requests are retransmitted these appear as RT(1), RT(2), RT(3), ... where the number after the initial RT shows how many times the request was retransmitted before it received a response.
+Lost means that the request never received a response.
+|===
+
+.Capture live stats on all interfaces output
+Muting stats for the next 5200 milliseconds (warmup)
+######### Stats Iteration 1 #########
+Interface capture rate:
+eth0 : 120.333/s
+lo : 0.000/s
+Stats muted because of warmup, or previous error
+######### Stats Iteration 2 #########
+Interface capture rate:
+eth0 : 129.333/s
+lo : 0.000/s
+Stats muted because of warmup, or previous error
+######### Stats Iteration 3 #########
+Interface capture rate:
+eth0 : 139.000/s
+lo : 0.000/s
+Access-Request counters:
+Total : 70.667/s
+Linked : 68.333/s
+Unlinked : 0.000/s
+Access-Request latency:
+High : 105.64ms
+Low : 0.108ms
+Average : 78.724ms
+MA : 78.724ms
+Access-Request retransmits & loss:
+RT (0) : 53.667/s
+Access-Accept counters:
+Total : 15.333/s
+Linked : 15.333/s
+Unlinked : 0.000/s
+Access-Accept latency:
+High : 0.439ms
+Low : 0.160ms
+Average : 0.384ms
+MA : 0.384ms
+Access-Reject counters:
+Total : 53.000/s
+Linked : 53.000/s
+Unlinked : 0.000/s
+Access-Reject latency:
+High : 1005.643ms
+Low : 1001.162ms
+Average : 1002.903ms
+MA : 1002.903ms
+
+
+== Filtering Requests
+
+The radsniff tool can filter the traffic looking for attributes with specific values.
+The attributes in the request filter below can be something like NAS-IP-Address or User-Name to filter for requests containing a specific client or user:
+
+* root@radius# radsniff -r 'User-Name==carol'
+* root@radius# radsniff -r 'User-Name==bob,NAS-IP-Address==2.3.4.5'
+
+Similarly, response attributes can be filtered with -R in which case replies are buffered until a reponse is seen or it times out.
+
+== Displaying specific attributes
+The radsniff output can be amended to display specific attributes using -l:
+
+root@radius# radsniff -l User-Name,NAS-IP-Address,Reply-Message
+
+Defaulting to capture on all interfaces
+noreq,1, ... ,,Access-Accept,eth0,192.0.2.100,1812,2.3.4.5,34633,86, \
+,,
+noreq,2, ... ,,Access-Reject,eth0,192.0.2.100,1812,2.3.4.5,77, \
+,,"Simultaneous use count exceeded"
+received,3, ... ,,Access-Request,eth0,2.3.4.5,1812,192.0.2.100,1812,207, \
+"bob",172.16.246.129,
+noreq,4, ... ,,Access-Reject,eth0,192.0.2.100,1812,2.3.4.5,1812,209, \
+,,"Subscriber carol not found"
o Others
```
-NOTE: Before moving on, you will probably want to add `/dev/null` to
+[NOTE]
+====
+Before moving on, you will probably want to add `/dev/null` to
/etc/shells _temporarily_ so that default system authentication will
work. REMEMBER TO TAKE IT OUT!
+====
== Test procedures
[arabic, start=5]
. Run radclient with `radius.test` as the input file.
-NOTE: First you need to setup a secret for your local machine in the
-`clients` file and use that secret below
+[NOTE]
+====
+First you need to setup a secret for your local machine in the
+`clients` file and use that secret below. Enter the command all on one line.
+====
```
# time /usr/local/bin/radclient -q -s -f radius.test <yourhostname> auth <secret>
```
-NOTE: The above is to be put all on one line.
-NOTE: Some systems do not have the `time` command, so you may need to
-break out the stopwatch instead :)
+[NOTE]
+====
+Some systems do not have the `time` command, so you may need to
+break out the stopwatch instead.
+====
Take note of the output of `radclient`. If there were lots of
failures, something is wrong. All authentications should succeed.
== Results
-I’d really rather not post results because they will vary tremendously
-with other system-specific configuration. This is exactly the reason you
-should run tests of this nature, to find what’s best for _your_ system.
-Good luck!
+The testing results vary tremendouslywith other system-specific configurations. This is exactly the reason you should run tests of this nature, to find what’s best for _your_ system. Good luck!
// Copyright (C) 2025 Network RADIUS SAS. Licenced under CC-by-NC 4.0.
// This documentation was developed by Network RADIUS SAS.
projects / thirdparty / freeradius-server.git / commitdiff
