From: Arran Cudbard-Bell Date: Thu, 5 Mar 2020 19:54:38 +0000 (-0600) Subject: rlm_radius: Finish manual test plan for rlm_radius X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=c05aedf49fbac85bc78df7f2950153347d6625bd;p=thirdparty%2Ffreeradius-server.git rlm_radius: Finish manual test plan for rlm_radius --- diff --git a/src/modules/rlm_radius/test_plan.adoc b/src/modules/rlm_radius/test_plan.adoc index 4023f6f280c..ad6d64bb1a3 100644 --- a/src/modules/rlm_radius/test_plan.adoc +++ b/src/modules/rlm_radius/test_plan.adoc @@ -16,14 +16,20 @@ sendmsg on MacOS for example, produces no indication of reachability issues, and sendmmsg is only available on Linux, FreeBSD. ==== -For load tests, or sending multiple packets in quick succession proto_radius_load should -be used to generate input packets and produce packet rate information. +For load tests, or sending multiple packets in quick succession proto_radius_load should be used to generate +input packets and produce packet rate information. -Do not use radperf as it introduces additional latency and multiple instances are -required to achieve high packet rates (meaning accurate rate information is not available). +Do not use radperf as it introduces additional latency and multiple instances are required to achieve high +packet rates (meaning accurate rate information is not available). For sending small numbers of test packets radclient should be used. +[TIP] +==== +Start with a default rlm_radius configuration for each test. Don't apply configuration changes additively. + +==== + == Test packets - Sent outbound to destination === Access-Request ==== PAP Access-Request @@ -32,17 +38,27 @@ For sending small numbers of test packets radclient should be used. User-Name = "testuser" User-Password = "supersecret" Service-Type = Framed-User -Tunnel-Password = 0xff0B7375706572736563726574000000 +Tunnel-Password = "supersecret" Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565 ``` -==== CHAP Access-Request +==== CHAP Access-Request - unlang ``` User-Name = "testuser" Chap-Password = %{chap_password:supersecret} Service-Type = Framed-User NAS-IP-Address = 127.0.0.1 -Tunnel-Password = 0xff0B7375706572736563726574000000 +Tunnel-Password = "supersecret" +Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565 +``` + +==== CHAP Access-Request - radclient +``` +User-Name = "testuser" +Chap-Password = "supersecret" +Service-Type = Framed-User +NAS-IP-Address = 127.0.0.1 +Tunnel-Password = "supersecret" Class = 0x69616D616E6F706171756576616C756569616D616E6F706171756576616C7565 ``` @@ -75,6 +91,36 @@ NAS-IP-Address = 127.0.0.1 NAS-Port-ID = 0 Message-Authenticator = 0x ``` +== Topology + +The same topology is used for all tests, and consists of two devices. + +``` + NAS + Proxy Home Server + _____________ ____________ +| | | | +| FreeRADIUS | | | +| + radclient | <-------> | FreeRADIUS | +| + wireshark | | | +|_____________| |____________| +``` + +Packets will be generated on the NAS + Proxy server using either `proto_radius_load` or `radclient`. +`radperf` should not be used as it's deprecated, and cannot produce the load levels required. + +The home server should be configured to respond to all packets from the `NAS + Proxy` with a success +response (`Access-Accept` / `Accounting-Response`). + +Packet loss should be introduced on the home server. There are various facilities in IP tables to do this, +or `%{rand:}` and `&control:Response-Packet-Type = none` may be used on the home server to randomly prevent +replies to requests. + +If delay needs to be added at a network level, a script is available https://gist.github.com/arr2036/6598137[here]. + +[NOTE] +==== +Restart the `NAS + Proxy` instance after each test. +==== == 1. Replicate mode @@ -90,17 +136,19 @@ incoming RADIUS packets. --- -* Send a <> - Verify that -** No ICMP messages are generated by the operating system. +* Send 50 <>s +** Verify that no ICMP messages are generated by the operating system when responses are received. -=== 1.2 Base packet rate +=== 1.2 Base TX packet rate Platforms:: MacOS, Linux, FreeBSD -Rationale:: As there is no acknowledgement required, packet rate tests, test only the ability of the rlm_radius -module to transmit packets. On Linux and FreeBSD we'd expect an exceptionally high packet rate in this mode, -as we coalesce multiple outbound packets and send them with a single call to `sendmmsg.` For MacOS each outbound -packet requires a system call, so we expect the rate to be significantly lower. +Rationale:: As there are no ACKs required this tests mesures the rate at which rlm_radius + transport can send +packets. On Linux and FreeBSD we'd expect an exceptionally high packet rate in this mode as we coalesce multiple +outbound packets and send them using single system call +(`sendmmsg` for datagrams, and aggregated buffers + `write` for streams). + +When sending datagrams on macOS no native `sendmmsg` function exists, so we expect the rate to be significantly lower. --- @@ -110,19 +158,40 @@ packet requires a system call, so we expect the rate to be significantly lower. * Configure proto_radius_load `start_pps = 10000` * Configure proto_radius_load `duration = 5` * Configure proto_radius_load `max_pps = 500000` -* Configure proto_radius_load `max_backlog = 4096` - This essentially controls how many packets are "in flight" +* Configure proto_radius_load `max_backlog = 4096` - This controls how many packets are "in flight" within the server for any given period. * Set `.max_packet_size = 128` to reduce memory consumption - This has a direct effect on the amount of memory pre-allocated in trunk requests. ** Use the PAP test packet. Verify a packet rate of at least 25,000PPS on MacOS, and 50,000PPS on Linux and FreeBSD. == 2. Proxy mode - -=== 2.1 Entering and exiting zombie state +=== 2.1 Entering and exiting zombie state (no status checks) Platforms:: Any -Note:: You may need to increase `max_request_time` in radius.conf to 60 seconds for this test. +Note:: You may need to increase `max_request_time` in radius.conf for this test. + +This test verifies behaviour when no status_check is configured. + +In this mode when no responses are received for `zombie_period` seconds, the connection is marked as a zombie. +Whilst in the zombie state no additional requests will be enqueued on the connection. Any requests +in the connection's backlog will be shifted onto other connections or into the trunk's backlog. + +A connection in zombie state is set to be `INACTIVE`. This removes the connection from the set of `ACTIVE` +connections which can be used to send packets. + +If there are no `ACTIVE` connections, and the the last trunk event was a connection failure (i.e. a reconnection), +new requests will immediately fail and will not enter the trunk's backlog. +If there are no `ACTIVE` connections, and the last trunk event was a connection opening successfully, +new requests will be enqueued on the trunk's backlog, and will be transferred to the next connection to enter the +`ACTIVE` state. + +As there's no method to determine if the upstream server is alive, we need to periodically "revive" the connection. +Reviving in this context means reconnecting the connection. Reviving the connection is done after `revive_interval`. +When a connection is revived it enters the `ACTIVE` state, and any requests in the trunk's backlog are transferred +to it. + +--- * Ensure `status_check.type` is not set. * Set `zombie_period = 10`. @@ -131,18 +200,154 @@ Note:: You may need to increase `max_request_time` in radius.conf to 60 seconds * Set packet loss rate on network link to be 100% * Send a <> ** Verify that after `.initial_rtx_time` (default 2 seconds) the packet is resent. -** Verify that packets are resent at increasing intervals until `.max_rtx_count` - is reached. -** After 10 seconds, verify that the connection is marked as "inactive", and a message is emitted indicating - that the connection has entered zombie state. -*** Verify that a new connection is opened. -*** Verify that the outstanding request is re-queued on this new connection. +** Verify that packets are resent at increasing intervals until `.Access-Request.max_rtx_count` + or `.Access-Request.max_rtx_duration` is reached. +** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message indicates + the connection is a zombie. +*** Verify outstanding request enters the backlog _(no active connections, but no failure)_. +* Send a <> +** Verify this request enters the backlog _(no active connections, but no failure)_. +* Set packet loss rate on network link to be 0% +* Wait `revive_interval` (default 10s). The connection should enter the `FAILED` then `CLOSED` states. +** Verify both outstanding requests now fail _(no active connections, last event was failure)_. +* Wait `pool.reconnect_delay` (default 5s). The connection should enter the `INIT` then `CONNECTING` states. +* Send a <> +** Verify this request is sent on the freshly revived connection + _(active connections, last event was connection connected)_. + +=== 2.2 Entering and exiting zombie state (status check - full failure) + +Platforms:: Any + +Note:: You may need to increase `max_request_time` in radius.conf for this test. + +This test verifies the behaviour when a `Status-Server` packet is configured for status checks. + +In this mode when no responses are received for `zombie_period` seconds the connection is marked as a zombie. +A connection marked as a zombie enters the `INACTIVE` state. This removes the connection from the set of +`ACTIVE` connections which can be used to send packets, but does not cancel pending requests already sent +on the connection. + +When a connection enters zombie state it will start sending status-check messages. +One of the below conditions will cause the connection to exit zombie state: + +- `status_check.num_answers_to_alive` contiguous responses are received to status checks - + The connection is re-enlivened (enters the `ACTIVE` state). +- `.Status-Server.max_rtx_duration` is reached - The connection is dead and enters the `FAILED`/`CLOSED` state. + All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or + onto another connection. +- `.Status-Server.max_rtx_count` is hit - The connection is dead and enters the `FAILED`/`CLOSED` state. + All outstanding sent packets are cancelled, and `SENT` or `PENDING` requests are moved to the trunk's backlog or + onto another connection. + +Once `pool.reconnect_delay` seconds have passed, the connection enters the `INIT`/`CONNECTING` state. +Once in the `CONNECTING` state, the connection will begin sending `Status-Server` requests. +`status_check.num_answers_to_alive` contiguous responses are required for the connection to be considered `CONNECTED` +and to receive new requests. +If `max_rtx_duration` or `max_rtc_count` are hit before `status_check.num_answers_to_alive` contiguous responses are +received, the connection will be reconnected (again) after `pool.reconnect_delay`. + +[NOTE] +==== +Immediate entry into zombie state is likely incorrect and will probably change. +==== + +--- + +* Set `status_check.type = Status-Server` +* Set `zombie_period = 10`. +* Send a <> +** Verify a response is received. +* Set packet loss rate on network link to be 100% +* Send a <> +** Verify that after `.initial_rtx_time` (default 2 seconds) the packet is resent. +** Verify that packets are resent at increasing intervals until `.Access-Request.max_rtx_count` + or `.Access-Request.max_rtx_duration` is reached after which the request should fail. +** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message + is displayed indicating the connection is now a zombie. +** Verify connection begins sending `Status-Server` requests. +** Verify that after `max_rtx_count` (default 5) packets are sent, or after `max_rtx_duration` (default 30s) passes + the connection enters the `FAILED`/`CLOSED` state. +* Send a <> +** Verify the request immediately fails _(no active connections, and previous failure)_. +* Set packet loss rate on network link to be 0%. +* Wait `reconnect_delay` (default 5s). The connection should enter the `INIT`/`CONNECTING` state +** Verify connection beings sending `Status-Server` requests. +** Verify that `status_check.num_answers_to_alive` (default 3) status checks are sent with a constant delay + between each check. +** Verify that after `status_check.num_answers_to_alive` responses the connection enters the `CONNECTED` + `ACTIVE` + state. +* Send a <> +** Verify this request is sent immediately. + +=== 2.3 Entering and exiting zombie state (status check - partial failure and revival) + +Platforms:: Any + +Note:: You may need to increase `max_request_time` in radius.conf for this test. + +See description for previous test. + +[NOTE] +==== +Immediate entry into zombie state is likely incorrect and will probably change. +==== + +--- +* Set `status_check.type = Status-Server` +* Set `zombie_period = 10`. +* Send a <> +** Verify a response is received. +* Set packet loss rate on network link to be 100% +* Send a <> +** Verify that after `.initial_rtx_time` (default 2 seconds) the packet is resent. +** Verify that packets are resent at increasing intervals until `.Access-Request.max_rtx_count` + or `.Access-Request.max_rtx_duration` is reached. +** After 10 seconds, verify that the connection enters the `INACTIVE` state, and a log message + is displayed indicating the connection is now a zombie. +** Verify connection begins sending `Status-Server` requests. +* Immediately send a <> +** Verify request enters the trunk's backlog _(no active connections, no previous failure)_. +* After one `Status-Server` packet, set packet loss rate on network link to be 0%. +** Verify that `status_check.num_answers_to_alive` (default 3) additional `Status-Server` packets are sent with a + constant delay between each. +** Verify that on the last response to a `Status-Server` packet the connection immediately enters the `ACTIVE` state. +*** Verify previously backlogged request is now sent. +* Send a <> +** Verify this request is sent immediately. + +=== 2.4 Synchronous retransmissions + +Platforms:: Any + +rlm_radius can operate in synchronous and asynchronous retransmission modes. In asynchronous modes the rlm_radius +module will send retransmissions itself, in synchronous mode the rlm_radius module uses retransmissions by the NAS +to drive retransmissions to the upstream server. + +This test verifies synchronous retransmissions operate correctly. + +--- + +* Set `status_check.type = none`. +* Set `zombie_period = 120`. +* Set `synchronous = yes`. +* Set packet loss rate on network link to be 100% +* Send a <> with 5 retransmissions, 5 seconds apart (use radclient). +** Verify that as radiusd receives each retransmission, rlm_radius sends a new request. +* After 10 seconds set packet loss rate on network link to be 0%. +** Verify response is passed back from rlm_radius, that the response is sent to radclient, and that + no more retransmissions are sent. === 2.5 Memory usage under adverse conditions Platforms:: Any -Repeat with no `status_check.type` and `status_check.type = status-server`. +Repeat with: + +* `status_check.type = Status-Server`, `zombie_period = 1`. +* no `status_check.type`, `zombie_period = 1`, `revive_interval = 10`. + +--- * Configure a packet loss rate of 35%. * Send <>s at a high rate for 30 minutes. @@ -150,7 +355,10 @@ Repeat with no `status_check.type` and `status_check.type = status-server`. == 3. Both replicate and proxy modes -Repeat these tests for `replicate = yes` and `replicate = no`. +i.e. repeat these tests with: + +* `replicate = yes`. +* `replicate = no`. === 3.1 Well formedness @@ -161,22 +369,22 @@ You will need to use different subrequest types to produce the different packet --- -* Send a <> - Verify that: -** Packet is well formed. -** Message-Authenticator attribute is present and correct. -* Send a <> - Verify that: -** Packet is well formed. -** CHAP-Challenge is present and matches the Authentication Vector of the input packet. -** Message-Authenticator attribute is present and correct. +* Send a <> +** Verify packet is well formed. +** Verify Message-Authenticator attribute is present and correct. +* Send a <> - Verify that: +** Verify packet is well formed. +** Verify `CHAP-Challenge` attribute is present and matches the Authentication Vector of the input packet. +** Verify Message-Authenticator attribute is present and correct. * Send an <> - Verify that: -** Packet is well formed. -** No Message-Authenticator attribute is present +** Verify packet is well formed. +** Verify No Message-Authenticator attribute is present * Send a <> -** Packet is well formed. -** Message-Authenticator attribue is present. +** Verify packet is well formed. +** Verify Message-Authenticator attribue is present. * Send a <> -** Packet is well formed. -** Message-Authenticator attribue is present. +** Verify packet is well formed. +** Verify Message-Authenticator attribue is present. === 3.2 Outbound packet buffer overrun @@ -187,12 +395,11 @@ Rationale:: Verify that internal logic deals correctly with packet buffer overru --- * Set `.max_packet_size = 64` -* Send a <> - Verify that: -** An error is produced explaining why the packet can't be encoded (out of buffer space). -** The request fails (Look for trunk state transition `PENDING -> FAILED`). -** The connection is *NOT* re-established. -** The request is not re-queued and the rlm_radius returns the `fail` return code. -* Unset `.max_packet_size` +* Send a <> +** Verify an error is produced explaining why the packet can't be encoded (out of buffer space). +** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`). +** Verify the connection is *NOT* reconnected and stays in the `ACTIVE` state. +** Verify the request is not re-queued and rlm_radius returns `fail`. === 3.3 SNDBUF exhaustion - single packet @@ -200,19 +407,18 @@ Platforms:: MacOS, Linux, FreeBSD Rationale:: Verify correct behaviour when the packet size exceeds the `SO_SNDBUF` value for the socket. `SO_SNDBUF` on the socket should be configured to be small enough that any -outbound requests immediately fail, hopefully with an error which allows us to distinguish -between a temporary buffer exhaustion condition, and a permanent error (packet too big). +outbound requests immediately fail, --- * set `.send_buff = 64` -* Send a <> - Verify that: -** An error is produced explaining why the packet can't be sent (`EMSGSIZE`). If a different +* Send a <> +** Verify an error is produced explaining why the packet can't be sent (`EMSGSIZE`). If a different error is produced, count this as a test fail and record the error. -** The request fails (Look for trunk state transition `PENDING -> FAILED`). -** The connection is *NOT* re-established. This is a temporary condition, not a fatal one. -** The request is not re-queued and the rlm_radius returns the `fail` return code. -* unset `.send_buff` +** Verify the request fails (Look for trunk request state transition `PENDING -> FAILED`). +** Verify the connection is *NOT* re-established and stays in the `ACTIVE` state + _(this is a temporary condition, not a fatal one)_. +** The request is not re-queued and rlm_radius returns `fail`. === 3.4 SNDBUF exhaustion - multiple packets @@ -225,16 +431,15 @@ fail. --- -* set `.send_buff` = 128` -* Send multiple <>s at a high rate - Verify that: -** The first request is sent successfully. -** A subsequent request fails. You'll likely see two failure conditions, one with an explicit +* set `.send_buff = 128` +* Send multiple <>s at a high rate +** Verify the first request is sent successfully. +** Verify a subsequent request fails. You'll likely see two failure conditions, one with an explicit error, and one where coalesced packets are silently re-queued. It depends on whether the - error occurs on the first packet being processed by sendmmsg or a subsequent one. + error occurs on the first packet being processed by `sendmmsg` or a subsequent one. ** Verify that for the explicit failure the error message is (`ENOBUFS`, `EWOULDBLOCK` or `EAGAIN`). ** The connection is *NOT* re-established. This is a temporary condition, not a fatal one. -** Requests do not explicitly fail (absence of `PENDING -> FAILED` transitions). -* unset `.send_buff` +** Requests do not explicitly fail (absence of trunk request `PENDING -> FAILED` transitions). === 3.5 EHOSTUNREACH - Unreachable host