sendmmsg is only available on Linux, FreeBSD.
====
-For load tests, or sending multiple packets in quick succession proto_radius_load should
-be used to generate input packets and produce packet rate information.
+For load tests, or sending multiple packets in quick succession proto_radius_load should be used to generate
+input packets and produce packet rate information.
-Do not use radperf as it introduces additional latency and multiple instances are
-required to achieve high packet rates (meaning accurate rate information is not available).
+Do not use radperf as it introduces additional latency and multiple instances are required to achieve high
+packet rates (meaning accurate rate information is not available).
For sending small numbers of test packets radclient should be used.
+[TIP]
+====
+Start with a default rlm_radius configuration for each test. Don't apply configuration changes additively.
+
+====
+
== Test packets - Sent outbound to destination
=== Access-Request
==== PAP Access-Request
User-Name = "testuser"
User-Password = "supersecret"
Service-Type = Framed-User
-Tunnel-Password = 0xff0B7375706572736563726574000000
+Tunnel-Password = "supersecret"
Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
```
-==== CHAP Access-Request
+==== CHAP Access-Request - unlang
```
User-Name = "testuser"
Chap-Password = %{chap_password:supersecret}
Service-Type = Framed-User
NAS-IP-Address = 127.0.0.1
-Tunnel-Password = 0xff0B7375706572736563726574000000
+Tunnel-Password = "supersecret"
+Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
+```
+
+==== CHAP Access-Request - radclient
+```
+User-Name = "testuser"
+Chap-Password = "supersecret"
+Service-Type = Framed-User
+NAS-IP-Address = 127.0.0.1
+Tunnel-Password = "supersecret"
Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565
```
NAS-Port-ID = 0
Message-Authenticator = 0x
```
+== Topology
+
+The same topology is used for all tests, and consists of two devices.
+
+```
+ NAS + Proxy Home Server
+ _____________ ____________
+| | | |
+| FreeRADIUS | | |
+| + radclient | <-------> | FreeRADIUS |
+| + wireshark | | |
+|_____________| |____________|
+```
+
+Packets will be generated on the NAS + Proxy server using either `proto_radius_load` or `radclient`.
+`radperf` should not be used as it's deprecated, and cannot produce the load levels required.
+
+The home server should be configured to respond to all packets from the `NAS + Proxy` with a success
+response (`Access-Accept` / `Accounting-Response`).
+
+Packet loss should be introduced on the home server. There are various facilities in IP tables to do this,
+or `%{rand:}` and `&control:Response-Packet-Type = none` may be used on the home server to randomly prevent
+replies to requests.
+
+If delay needs to be added at a network level, a script is available https://gist.github.com/arr2036/6598137[here].
+
+[NOTE]
+====
+Restart the `NAS + Proxy` instance after each test.
+====
== 1. Replicate mode
---
-* Send a <<PAP Access-Request>> - Verify that
-** No ICMP messages are generated by the operating system.
+* Send 50 <<PAP Access-Request>>s
+** Verify that no ICMP messages are generated by the operating system when responses are received.
-=== 1.2 Base packet rate
+=== 1.2 Base TX packet rate
Platforms:: MacOS, Linux, FreeBSD
-Rationale:: As there is no acknowledgement required, packet rate tests, test only the ability of the rlm_radius
-module to transmit packets. On Linux and FreeBSD we'd expect an exceptionally high packet rate in this mode,
-as we coalesce multiple outbound packets and send them with a single call to `sendmmsg.` For MacOS each outbound
-packet requires a system call, so we expect the rate to be significantly lower.
+Rationale:: As there are no ACKs required this tests mesures the rate at which rlm_radius + transport can send
+packets. On Linux and FreeBSD we'd expect an exceptionally high packet rate in this mode as we coalesce multiple
+outbound packets and send them using single system call
+(`sendmmsg` for datagrams, and aggregated buffers + `write` for streams).
+
+When sending datagrams on macOS no native `sendmmsg` function exists, so we expect the rate to be significantly lower.
---
* Configure proto_radius_load `start_pps = 10000`
* Configure proto_radius_load `duration = 5`
* Configure proto_radius_load `max_pps = 500000`
-* Configure proto_radius_load `max_backlog = 4096` - This essentially controls how many packets are "in flight"
+* Configure proto_radius_load `max_backlog = 4096` - This controls how many packets are "in flight"
within the server for any given period.
* Set `<transport>.max_packet_size = 128` to reduce memory consumption - This has a direct effect on the amount of
memory pre-allocated in trunk requests.
** Use the PAP test packet. Verify a packet rate of at least 25,000PPS on MacOS, and 50,000PPS on Linux and FreeBSD.
== 2. Proxy mode
-
-=== 2.1 Entering and exiting zombie state
+=== 2.1 Entering and exiting zombie state (no status checks)
Platforms:: Any
-Note:: You may need to increase `max_request_time` in radius.conf to 60 seconds for this test.
+Note:: You may need to increase `max_request_time` in radius.conf for this test.
+
+This test verifies behaviour when no status_check is configured.
+
+In this mode when no responses are received for `zombie_period` seconds, the connection is marked as a zombie.
+Whilst in the zombie state no additional requests will be enqueued on the connection. Any requests
+in the connection's backlog will be shifted onto other connections or into the trunk's backlog.
+
+A connection in zombie state is set to be `INACTIVE`. This removes the connection from the set of `ACTIVE`
+connections which can be used to send packets.
+
+If there are no `ACTIVE` connections, and the the last trunk event was a connection failure (i.e. a reconnection),
+new requests will immediately fail and will not enter the trunk's backlog.
+If there are no `ACTIVE` connections, and the last trunk event was a connection opening successfully,
+new requests will be enqueued on the trunk's backlog, and will be transferred to the next connection to enter the
+`ACTIVE` state.
+
+As there's no method to determine if the upstream server is alive, we need to periodically "revive" the connection.
+Reviving in this context means reconnecting the connection. Reviving the connection is done after `revive_interval`.
+When a connection is revived it enters the `ACTIVE` state, and any requests in the trunk's backlog are transferred
+to it.
+
+---
* Ensure `status_check.type` is not set.
* Set `zombie_period = 10`.
* Set packet loss rate on network link to be 100%
* Send a <<PAP Access-Request>>
** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
-** Verify that packets are resent at increasing intervals until `<transport>.max_rtx_count`
- is reached.
-** After 10 seconds, verify that the connection is marked as "inactive", and a message is emitted indicating
- that the connection has entered zombie state.
-*** Verify that a new connection is opened.
-*** Verify that the outstanding request is re-queued on this new connection.
+** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
+ or `<transport>.Access-Request.max_rtx_duration` is reached.
+** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message indicates
+ the connection is a zombie.
+*** Verify outstanding request enters the backlog _(no active connections, but no failure)_.
+* Send a <<PAP Access-Request>>
+** Verify this request enters the backlog _(no active connections, but no failure)_.
+* Set packet loss rate on network link to be 0%
+* Wait `revive_interval` (default 10s). The connection should enter the `FAILED` then `CLOSED` states.
+** Verify both outstanding requests now fail _(no active connections, last event was failure)_.
+* Wait `pool.reconnect_delay` (default 5s). The connection should enter the `INIT` then `CONNECTING` states.
+* Send a <<PAP Access-Request>>
+** Verify this request is sent on the freshly revived connection
+ _(active connections, last event was connection connected)_.
+
+=== 2.2 Entering and exiting zombie state (status check - full failure)
+
+Platforms:: Any
+
+Note:: You may need to increase `max_request_time` in radius.conf for this test.
+
+This test verifies the behaviour when a `Status-Server` packet is configured for status checks.
+
+In this mode when no responses are received for `zombie_period` seconds the connection is marked as a zombie.
+A connection marked as a zombie enters the `INACTIVE` state. This removes the connection from the set of
+`ACTIVE` connections which can be used to send packets, but does not cancel pending requests already sent
+on the connection.
+
+When a connection enters zombie state it will start sending status-check messages.
+One of the below conditions will cause the connection to exit zombie state:
+
+- `status_check.num_answers_to_alive` contiguous responses are received to status checks -
+ The connection is re-enlivened (enters the `ACTIVE` state).
+- `<transport>.Status-Server.max_rtx_duration` is reached - The connection is dead and enters the `FAILED`/`CLOSED` state.
+ All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or
+ onto another connection.
+- `<transport>.Status-Server.max_rtx_count` is hit - The connection is dead and enters the `FAILED`/`CLOSED` state.
+ All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or
+ onto another connection.
+
+Once `pool.reconnect_delay` seconds have passed, the connection enters the `INIT`/`CONNECTING` state.
+Once in the `CONNECTING` state, the connection will begin sending `Status-Server` requests.
+`status_check.num_answers_to_alive` contiguous responses are required for the connection to be considered `CONNECTED`
+and to receive new requests.
+If `max_rtx_duration` or `max_rtc_count` are hit before `status_check.num_answers_to_alive` contiguous responses are
+received, the connection will be reconnected (again) after `pool.reconnect_delay`.
+
+[NOTE]
+====
+Immediate entry into zombie state is likely incorrect and will probably change.
+====
+
+---
+
+* Set `status_check.type = Status-Server`
+* Set `zombie_period = 10`.
+* Send a <<PAP Access-Request>>
+** Verify a response is received.
+* Set packet loss rate on network link to be 100%
+* Send a <<PAP Access-Request>>
+** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
+** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
+ or `<transport>.Access-Request.max_rtx_duration` is reached after which the request should fail.
+** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message
+ is displayed indicating the connection is now a zombie.
+** Verify connection begins sending `Status-Server` requests.
+** Verify that after `max_rtx_count` (default 5) packets are sent, or after `max_rtx_duration` (default 30s) passes
+ the connection enters the `FAILED`/`CLOSED` state.
+* Send a <<PAP Access-Request>>
+** Verify the request immediately fails _(no active connections, and previous failure)_.
+* Set packet loss rate on network link to be 0%.
+* Wait `reconnect_delay` (default 5s). The connection should enter the `INIT`/`CONNECTING` state
+** Verify connection beings sending `Status-Server` requests.
+** Verify that `status_check.num_answers_to_alive` (default 3) status checks are sent with a constant delay
+ between each check.
+** Verify that after `status_check.num_answers_to_alive` responses the connection enters the `CONNECTED` + `ACTIVE`
+ state.
+* Send a <<PAP Access-Request>>
+** Verify this request is sent immediately.
+
+=== 2.3 Entering and exiting zombie state (status check - partial failure and revival)
+
+Platforms:: Any
+
+Note:: You may need to increase `max_request_time` in radius.conf for this test.
+
+See description for previous test.
+
+[NOTE]
+====
+Immediate entry into zombie state is likely incorrect and will probably change.
+====
+
+---
+* Set `status_check.type = Status-Server`
+* Set `zombie_period = 10`.
+* Send a <<PAP Access-Request>>
+** Verify a response is received.
+* Set packet loss rate on network link to be 100%
+* Send a <<PAP Access-Request>>
+** Verify that after `<transport>.initial_rtx_time` (default 2 seconds) the packet is resent.
+** Verify that packets are resent at increasing intervals until `<transport>.Access-Request.max_rtx_count`
+ or `<transport>.Access-Request.max_rtx_duration` is reached.
+** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message
+ is displayed indicating the connection is now a zombie.
+** Verify connection begins sending `Status-Server` requests.
+* Immediately send a <<PAP Access-Request>>
+** Verify request enters the trunk's backlog _(no active connections, no previous failure)_.
+* After one `Status-Server` packet, set packet loss rate on network link to be 0%.
+** Verify that `status_check.num_answers_to_alive` (default 3) additional `Status-Server` packets are sent with a
+ constant delay between each.
+** Verify that on the last response to a `Status-Server` packet the connection immediately enters the `ACTIVE` state.
+*** Verify previously backlogged request is now sent.
+* Send a <<PAP Access-Request>>
+** Verify this request is sent immediately.
+
+=== 2.4 Synchronous retransmissions
+
+Platforms:: Any
+
+rlm_radius can operate in synchronous and asynchronous retransmission modes. In asynchronous modes the rlm_radius
+module will send retransmissions itself, in synchronous mode the rlm_radius module uses retransmissions by the NAS
+to drive retransmissions to the upstream server.
+
+This test verifies synchronous retransmissions operate correctly.
+
+---
+
+* Set `status_check.type = none`.
+* Set `zombie_period = 120`.
+* Set `synchronous = yes`.
+* Set packet loss rate on network link to be 100%
+* Send a <<PAP Access-Request>> with 5 retransmissions, 5 seconds apart (use radclient).
+** Verify that as radiusd receives each retransmission, rlm_radius sends a new request.
+* After 10 seconds set packet loss rate on network link to be 0%.
+** Verify response is passed back from rlm_radius, that the response is sent to radclient, and that
+ no more retransmissions are sent.
=== 2.5 Memory usage under adverse conditions
Platforms:: Any
-Repeat with no `status_check.type` and `status_check.type = status-server`.
+Repeat with:
+
+* `status_check.type = Status-Server`, `zombie_period = 1`.
+* no `status_check.type`, `zombie_period = 1`, `revive_interval = 10`.
+
+---
* Configure a packet loss rate of 35%.
* Send <<PAP Access-Request>>s at a high rate for 30 minutes.
== 3. Both replicate and proxy modes
-Repeat these tests for `replicate = yes` and `replicate = no`.
+i.e. repeat these tests with:
+
+* `replicate = yes`.
+* `replicate = no`.
=== 3.1 Well formedness
---
-* Send a <<PAP Access-Request>> - Verify that:
-** Packet is well formed.
-** Message-Authenticator attribute is present and correct.
-* Send a <<CHAP Access-Request>> - Verify that:
-** Packet is well formed.
-** CHAP-Challenge is present and matches the Authentication Vector of the input packet.
-** Message-Authenticator attribute is present and correct.
+* Send a <<PAP Access-Request>>
+** Verify packet is well formed.
+** Verify Message-Authenticator attribute is present and correct.
+* Send a <<CHAP Access-Request - radclient>> - Verify that:
+** Verify packet is well formed.
+** Verify `CHAP-Challenge` attribute is present and matches the Authentication Vector of the input packet.
+** Verify Message-Authenticator attribute is present and correct.
* Send an <<Accounting-Request>> - Verify that:
-** Packet is well formed.
-** No Message-Authenticator attribute is present
+** Verify packet is well formed.
+** Verify No Message-Authenticator attribute is present
* Send a <<CoA-Request>>
-** Packet is well formed.
-** Message-Authenticator attribue is present.
+** Verify packet is well formed.
+** Verify Message-Authenticator attribue is present.
* Send a <<Disconnect-Message>>
-** Packet is well formed.
-** Message-Authenticator attribue is present.
+** Verify packet is well formed.
+** Verify Message-Authenticator attribue is present.
=== 3.2 Outbound packet buffer overrun
---
* Set `<transport>.max_packet_size = 64`
-* Send a <<PAP Access-Request>> - Verify that:
-** An error is produced explaining why the packet can't be encoded (out of buffer space).
-** The request fails (Look for trunk state transition `PENDING -> FAILED`).
-** The connection is *NOT* re-established.
-** The request is not re-queued and the rlm_radius returns the `fail` return code.
-* Unset `<transport>.max_packet_size`
+* Send a <<PAP Access-Request>>
+** Verify an error is produced explaining why the packet can't be encoded (out of buffer space).
+** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`).
+** Verify the connection is *NOT* reconnected and stays in the `ACTIVE` state.
+** Verify the request is not re-queued and rlm_radius returns `fail`.
=== 3.3 SNDBUF exhaustion - single packet
Rationale:: Verify correct behaviour when the packet size exceeds the `SO_SNDBUF` value
for the socket. `SO_SNDBUF` on the socket should be configured to be small enough that any
-outbound requests immediately fail, hopefully with an error which allows us to distinguish
-between a temporary buffer exhaustion condition, and a permanent error (packet too big).
+outbound requests immediately fail,
---
* set `<transport>.send_buff = 64`
-* Send a <<PAP Access-Request>> - Verify that:
-** An error is produced explaining why the packet can't be sent (`EMSGSIZE`). If a different
+* Send a <<PAP Access-Request>>
+** Verify an error is produced explaining why the packet can't be sent (`EMSGSIZE`). If a different
error is produced, count this as a test fail and record the error.
-** The request fails (Look for trunk state transition `PENDING -> FAILED`).
-** The connection is *NOT* re-established. This is a temporary condition, not a fatal one.
-** The request is not re-queued and the rlm_radius returns the `fail` return code.
-* unset `<transport>.send_buff`
+** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`).
+** Verify the connection is *NOT* re-established and stays in the `ACTIVE` state
+ _(this is a temporary condition, not a fatal one)_.
+** The request is not re-queued and rlm_radius returns `fail`.
=== 3.4 SNDBUF exhaustion - multiple packets
---
-* set `<transport>.send_buff` = 128`
-* Send multiple <<PAP Access-Request>>s at a high rate - Verify that:
-** The first request is sent successfully.
-** A subsequent request fails. You'll likely see two failure conditions, one with an explicit
+* set `<transport>.send_buff = 128`
+* Send multiple <<PAP Access-Request>>s at a high rate
+** Verify the first request is sent successfully.
+** Verify a subsequent request fails. You'll likely see two failure conditions, one with an explicit
error, and one where coalesced packets are silently re-queued. It depends on whether the
- error occurs on the first packet being processed by sendmmsg or a subsequent one.
+ error occurs on the first packet being processed by `sendmmsg` or a subsequent one.
** Verify that for the explicit failure the error message is (`ENOBUFS`, `EWOULDBLOCK` or `EAGAIN`).
** The connection is *NOT* re-established. This is a temporary condition, not a fatal one.
-** Requests do not explicitly fail (absence of `PENDING -> FAILED` transitions).
-* unset `<transport>.send_buff`
+** Requests do not explicitly fail (absence of trunk request `PENDING -> FAILED` transitions).
=== 3.5 EHOSTUNREACH - Unreachable host
projects / thirdparty / freeradius-server.git / commitdiff
