From: Jim Jagielski Date: Fri, 5 Jun 2026 08:02:50 +0000 (+0000) Subject: mod_proxy_beacon: self-registering balancer membership over UDP X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=2897c3ad075d0516ba1950f7c6455285300dc249;p=thirdparty%2Fapache%2Fhttpd.git mod_proxy_beacon: self-registering balancer membership over UDP git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1934995 13f79535-47bb-0310-9956-ffa450edef68 --- diff --git a/CHANGES b/CHANGES index 9f9e2b9d06..1b7ada4999 100644 --- a/CHANGES +++ b/CHANGES @@ -1,6 +1,10 @@ -*- coding: utf-8 -*- Changes with Apache 2.5.1 + *) mod_proxy_beacon: Back-end reverse proxy servers can announce + themselves and be auto-added to their front-end proxy balancer. + [Jim Jagielski] + *) support: Add in Python versions of various Perl-based support scripts - apxs-ng, dbmmanage-ng, log_server_status-ng, logresolve.py, phf_abuse_log-ng.cgi and split-logfile-ng. diff --git a/docs/log-message-tags/next-number b/docs/log-message-tags/next-number index f6a15fb2f3..4e6347722a 100644 --- a/docs/log-message-tags/next-number +++ b/docs/log-message-tags/next-number @@ -1 +1 @@ -10567 +10589 diff --git a/docs/manual/mod/allmodules.xml b/docs/manual/mod/allmodules.xml index 9f114e808e..68f40302aa 100644 --- a/docs/manual/mod/allmodules.xml +++ b/docs/manual/mod/allmodules.xml @@ -97,6 +97,7 @@ mod_proxy_html.xml mod_proxy_http.xml mod_proxy_http2.xml + mod_proxy_beacon.xml mod_proxy_scgi.xml mod_proxy_uwsgi.xml mod_proxy_wstunnel.xml diff --git a/docs/manual/mod/mod_proxy_beacon.xml b/docs/manual/mod/mod_proxy_beacon.xml new file mode 100644 index 0000000000..c4ac7c70db --- /dev/null +++ b/docs/manual/mod/mod_proxy_beacon.xml @@ -0,0 +1,367 @@ + + + + + + + + + +mod_proxy_beacon +Dynamic Balancer membership where backends announce themselves +to the reverse proxy over unicast UDP datagrams +Extension +mod_proxy_beacon.c +proxy_beacon_module +Available in Apache 2.5 and later + + +

This module lets backend servers announce themselves to a + front-end reverse proxy, which then adds each announcing backend as a live + member (worker) of a mod_proxy_balancer balancer. When a + backend stops announcing, the proxy takes it out of rotation. This provides + self-registering, self-healing balancer membership without editing the proxy + configuration or driving the balancer-manager by hand.

+ +

Communication uses plain unicast UDP datagrams (not + multicast, which is filtered on most networks and does not traverse the + public Internet). The data flows from backend to proxy:

+ +
    +
  • The reverse proxy binds a UDP socket and receives on a + stable address (ProxyBeaconListen).
    • +
  • Each backend periodically sends a short announcement datagram + to the proxy (ProxyBeaconAddress), advertising + its own routable URL + (ProxyBeaconAdvertise).
    • +
+ +

Datagrams are fire-and-forget: a lost announcement is recovered by the + next periodic one, and reordering is rejected by a per-backend timestamp + check, so no connection, reconnect, or framing layer is needed.

+ +

On the proxy, ProxyBeaconBalancer names the balancer + that announced backends are added to. Membership changes are applied using + the same internal mechanism as the balancer-manager web + interface, so a backend added this way behaves exactly like a statically + configured or manually added + BalancerMember, and is visible and + editable in the balancer-manager.

+ +

This module requires the service of + mod_watchdog and mod_proxy_balancer. The + background work (listening, publishing, adding and evicting members) runs in + a single mod_watchdog child process, so it is not available + under the prefork MPM behaviour where that singleton cannot + run.

+ +Authentication +

Any host that can reach the proxy's receive port could otherwise announce + an arbitrary backend URL and cause the proxy to send client traffic to it + (and a UDP source address is trivially spoofable). Set + ProxyBeaconSecret to the same value on the proxy and on + every backend so that announcements are authenticated with a keyed + message-authentication code (MAC) and a timestamp. When a secret is + configured the proxy drops any announcement that is not validly signed and + recent. If no secret is configured the channel is unauthenticated + and the proxy logs a warning at startup.

+ + +Confidentiality +

Announcements are authenticated but not encrypted; the payload is + operational metadata (backend URLs), not secret data. Transport + confidentiality (e.g. DTLS) is not currently provided and would be a separate + future layer.

+ + + +mod_proxy +mod_proxy_balancer +mod_proxy_hcheck +mod_watchdog + +
+ Usage example + +

The following pair of configurations sets up a self-registering balancer. + The backends require no knowledge of each other and the proxy needs no + pre-declared BalancerMember + entries — only an empty balancer with room to grow.

+ +

On the reverse proxy:

+ +# Receive backend announcements on the cluster network interface (UDP). +ProxyBeaconListen 0.0.0.0:5555 +ProxyBeaconSecret "a-long-random-shared-cluster-secret" +ProxyBeaconBalancer cluster + +# A backend is dropped from rotation if it does not announce for 30 seconds. +ProxyBeaconTimeout 30 + +# An initially-empty balancer with spare slots for the dynamic members. +<Proxy balancer://cluster> + ProxySet growth=16 +</Proxy> +ProxyPass "/" "balancer://cluster/" +ProxyPassReverse "/" "balancer://cluster/" + + +

On each backend server:

+ +# Announce this backend's routable origin to the proxy every 10 seconds (UDP). +ProxyBeaconAddress proxy.example.com:5555 +ProxyBeaconAdvertise http://10.0.0.5:8080 +ProxyBeaconSecret "a-long-random-shared-cluster-secret" +ProxyBeaconInterval 10 + + +

When a backend starts it begins sending announcements. The proxy + verifies each announcement against the shared secret, adds + http://10.0.0.5:8080 as a member of + balancer://cluster, and enables it. If that backend later stops + announcing for longer than ProxyBeaconTimeout, the proxy + disables the member (taking it out of rotation); a subsequent announcement + re-enables it.

+ + +

A backend added at runtime occupies one of the balancer's growth slots + for the lifetime of the server process; it is disabled rather than removed + when it stops announcing, matching the behaviour of the + balancer-manager (which can add, but not remove, workers at + runtime). Size growth for the maximum number of backends you + expect to register.

+ + +
+ + +ProxyBeaconListen +Address on which the reverse proxy receives backend +beacons +ProxyBeaconListen [address][:port] +server configvirtual host + + + +

The ProxyBeaconListen directive marks a server as + the beacon receiver (the reverse proxy). It binds a UDP socket to + the given address, e.g. 0.0.0.0:5555 to receive on all + interfaces. A leading scheme (such as tcp://) is accepted and + ignored.

+ +

The address and port are both optional and, when omitted, are inherited + from this server's own address and port (its Listen/ServerName). With no argument at all, the beacon + listener binds the server's own address and port; given just an address it + inherits the port, and so on. Because UDP and TCP are independent port + spaces, binding the beacon socket to the server's port does not + collide with the server's TCP listener — letting the beacon channel + share the service endpoint, which also identifies the proxy to backends by + its real address. (The listener binds in an unprivileged child, so a + privileged port such as 80 or 443 cannot be shared this way; use the + server's port only when it is non-privileged.)

+ +

Backends send to this address via + ProxyBeaconAddress. The directive should be used + together with ProxyBeaconBalancer; without it, + announcements are received and logged but no members are added. + ProxyBeaconListen and + ProxyBeaconAddress are mutually exclusive on the same + server.

+ + + + +ProxyBeaconAddress +Address of the reverse proxy to which a backend sends its +announcements +ProxyBeaconAddress address:port +server configvirtual host + + + +

The ProxyBeaconAddress directive marks a server as an + announcement sender (a backend). It sends UDP datagrams to the + proxy's ProxyBeaconListen address given by + address:port, e.g. proxy.example.com:5555 (a leading + scheme such as tcp:// is accepted and ignored). Because UDP is + connectionless, a backend may be started before the proxy is available: + early datagrams are simply dropped and the next interval retries.

+ +

Use ProxyBeaconAdvertise to specify the routable URL + the backend announces. ProxyBeaconAddress and + ProxyBeaconListen are mutually exclusive on the same + server.

+ + + + +ProxyBeaconAdvertise +The routable URL a backend announces to the reverse proxy +ProxyBeaconAdvertise url +server configvirtual host + + + +

The ProxyBeaconAdvertise directive sets the backend's + own reachable origin (for example http://10.0.0.5:8080) that the + proxy will add as a BalancerMember. + It must be a full scheme://host[:port] URL that the proxy can + reach — not the local listen address — and is validated when the + configuration is parsed.

+ +

This directive is used on a backend, alongside + ProxyBeaconAddress. If it is omitted, the backend still + sends a heartbeat but advertises no URL, so the proxy logs the + announcement without adding a member.

+ + + + +ProxyBeaconBalancer +Name of the balancer that announced backends are added to +ProxyBeaconBalancer name +server configvirtual host + + + +

The ProxyBeaconBalancer directive names the balancer, + on the reverse proxy, into which announced backends are inserted as members. + Give the bare balancer name (for example cluster for + balancer://cluster); a leading balancer:// is + accepted and stripped.

+ +

The named balancer must exist and have spare capacity. Declare it with a + <Proxy> block and a + growth setting (or rely on + BalancerGrowth) so there are free + slots for the dynamically added members. This directive is used together + with ProxyBeaconListen.

+ + + + +ProxyBeaconInterval +How often a backend publishes its announcement +ProxyBeaconInterval interval +ProxyBeaconInterval 5 +server configvirtual host + + + +

The ProxyBeaconInterval directive sets how frequently + a backend (a ProxyBeaconAddress server) publishes its + announcement. It uses the + time-interval directive syntax and + defaults to seconds; the default is 5 seconds.

+ +

The interval must be meaningfully smaller than the proxy's + ProxyBeaconTimeout, so that the occasional lost or + delayed announcement does not cause a healthy backend to be evicted.

+ + + + +ProxyBeaconTimeout +How long the proxy waits, without an announcement, before a backend +is taken out of rotation +ProxyBeaconTimeout interval +ProxyBeaconTimeout 0 +server configvirtual host + + + +

The ProxyBeaconTimeout directive sets how long the + reverse proxy will wait for an announcement from a backend before disabling + that backend's balancer member (taking it out of rotation). A later + announcement from the same backend re-enables it. It uses the + time-interval directive syntax and + defaults to seconds.

+ +

The default, 0, disables eviction entirely: backends are + added when they announce but are never automatically removed. Set this to a + small multiple of the backends' ProxyBeaconInterval to + enable self-healing membership. This directive is used on the proxy.

+ + + + +ProxyBeaconSecret +Pre-shared secret used to authenticate announcements +ProxyBeaconSecret secret +server configvirtual host + + + +

The ProxyBeaconSecret directive sets a pre-shared + cluster secret. It must be configured with the same value on the + reverse proxy and on every backend. The backend (sender) signs each + announcement with a keyed message-authentication code (a SipHash MAC) derived + from the secret, together with a timestamp; the proxy (receiver) recomputes the MAC and + checks the timestamp, dropping any announcement that is forged, tampered + with, or replayed. Replayed messages are caught two ways: a freshness window + (ProxyBeaconMaxSkew) rejects old timestamps, and a + per-backend check rejects any announcement whose timestamp does not strictly + advance, so a captured-and-resent message (for example, one replayed to keep + a dead backend from being evicted) is dropped.

+ +

If ProxyBeaconSecret is set on the proxy, every + announcement must carry a valid, recent MAC or it is rejected. If the + secrets on the proxy and a backend differ, that backend's announcements are + silently rejected (and logged), which appears as the backend never joining + the balancer.

+ +

If no secret is configured the channel is unauthenticated and the proxy + emits a warning when it starts listening. Because the secret is stored in + the configuration file, restrict that file's permissions as you would for a + private key.

+ + Clock synchronisation +

The timestamp-based replay protection compares the announcement's time + against the proxy's clock, so the proxy and backends must have reasonably + synchronised clocks (for example via NTP). See + ProxyBeaconMaxSkew.

+ + + + + +ProxyBeaconMaxSkew +Maximum allowed age of a signed announcement +ProxyBeaconMaxSkew interval +server configvirtual host + + + +

The ProxyBeaconMaxSkew directive sets the anti-replay + window used when ProxyBeaconSecret is configured: the + proxy rejects any announcement whose signed timestamp differs from the + current time by more than this amount, in either direction. It uses the + time-interval directive syntax and + defaults to seconds.

+ +

If unset, the default is 30 seconds. A larger window tolerates greater + clock skew between hosts; a smaller window bounds the freshness check. Note + that the per-backend strictly-increasing-timestamp check (see + ProxyBeaconSecret) blocks replays regardless of this + window. This directive is used on the proxy.

+ + + + diff --git a/docs/manual/mod/mod_proxy_beacon.xml.meta b/docs/manual/mod/mod_proxy_beacon.xml.meta new file mode 100644 index 0000000000..3f37efddb0 --- /dev/null +++ b/docs/manual/mod/mod_proxy_beacon.xml.meta @@ -0,0 +1,12 @@ + + + + + mod_proxy_beacon + /mod/ + .. + + + en + + diff --git a/modules/proxy/README.beacon b/modules/proxy/README.beacon new file mode 100644 index 0000000000..143befd1b7 --- /dev/null +++ b/modules/proxy/README.beacon @@ -0,0 +1,127 @@ +mod_proxy_beacon + +Self-registering reverse-proxy balancer membership: backend servers announce +themselves to a front-end proxy over UDP datagrams, and the proxy adds, enables, +and evicts them as live balancer members. See +docs/manual/mod/mod_proxy_beacon.xml for the user-facing directive reference; +this file describes the architecture. + +The transport is plain APR UDP sockets -- small, periodic, fire-and-forget +announcements with no third-party dependency. + + +Dependencies: + Requires mod_watchdog and mod_proxy_balancer. No external library: the UDP + transport uses APR sockets and authentication uses SipHash from APR-util (no + OpenSSL dependency). + + +Transport (unicast UDP, not multicast): + Each backend sends a point-to-point UDP datagram (apr_socket_sendto) to the + proxy's configured address. This is deliberately UNICAST -- unlike + mod_heartbeat, which multicasts to a group. Multicast is filtered on most + networks and does not traverse the public Internet, so a cross-host control + plane must be unicast. The socket-handling pattern is adapted from + mod_heartmonitor's unicast receive path. + + Datagrams are fire-and-forget. The design tolerates UDP loss and reordering + without a reconnect or framing layer: a lost announcement just delays an update + by one interval (announcements are periodic and idempotent), and an out-of-order + datagram is rejected by the per-url monotonic timestamp (see Authentication). + + +Roles: + Backend = sender (ProxyBeaconAddress): sends to the proxy, advertising its + own routable URL (ProxyBeaconAdvertise) periodically. + Proxy = receiver (ProxyBeaconListen): binds a stable address, receives + announcements, and manages balancer:// (ProxyBeaconBalancer). + + The proxy is the fixed unicast rendezvous; each backend is configured with its + address. One proxy receiver serves many backend senders. A leading scheme in + an address (e.g. tcp://) is accepted and ignored. + + ProxyBeaconListen's host and port are both optional: an omitted host/port (or + no argument at all) is inherited from the server's own address/port. Since UDP + and TCP are independent port spaces, the beacon socket can share the server's + service port without colliding with its TCP listener -- so a backend can beacon + to the proxy at its real service endpoint. (The listener binds in an + unprivileged watchdog child, so privileged ports like 80/443 cannot be shared + this way.) The sender's ProxyBeaconAddress targets the proxy and must be given + in full. + + +Where it runs: + All background work runs in a single mod_watchdog SINGLETON callback -- exactly + one child process owns the socket (so only one process binds the receive port + and performs membership changes). The watchdog fires every ~100ms: + + STARTING open the UDP socket; receiver binds, sender resolves its dest. + RUNNING sender: send a throttled announcement (ProxyBeaconInterval). + receiver: drain the socket, then run a throttled (~1s) eviction + sweep. + STOPPING close the socket. + + Because this relies on a singleton watchdog child, it is inactive under the + prefork MPM. + + +Adding/removing members (reuses the balancer-manager path): + Runtime worker *removal* does not exist in httpd, so membership is managed + entirely as status-flag changes via mod_proxy_balancer's exported + balancer_manage() optional function -- the same code the balancer-manager web + UI calls. The receiver synthesizes a minimal request_rec (no real client + request exists in the watchdog thread; see beacon_make_fake_request) and calls: + + add b= b_nwrkr= b_wyes=1 (worker created, disabled) + enable b= w= w_status_D=0 (cleared -> serves traffic) + evict b= w= w_status_D=1 (disabled -> out of rotation) + + Adds bump the balancer's shm "updated" timestamp, so every other child picks + up the new worker through the normal ap_proxy_sync_balancer() path -- the + change made in the singleton propagates fleet-wide. The target balancer must + have spare slots (ProxySet growth / BalancerGrowth). + + Per-backend state (last-seen time, added/evicted flags) is kept in a hash in + the singleton's own pool -- no shared memory is needed, since the singleton is + the only process that adds or evicts. A re-announcement after eviction + re-enables the worker; the flags make these transitions idempotent. + + +Message format: + Plain ASCII, space-separated key=value tokens, one datagram per announcement: + + BEACON url=http://host:port host= pid= seq= ts= mac= + + url= is the routable backend origin the proxy adds as a BalancerMember. ts= + (microseconds since the epoch) and mac= are present only when a shared secret + is configured (see below). host=/pid=/seq= are informational. + + A UDP datagram is delivered whole, so the receiver reads each message into a + fixed stack buffer and NUL-terminates it (buf[len] = '\0') before parsing. + The sender transmits strlen(msg) bytes (no trailing NUL on the wire). + + +Authentication (ProxyBeaconSecret, optional but recommended): + Without a secret the channel is unauthenticated: anyone who can reach the + receiver port could announce a URL and hijack client traffic (the proxy logs a + warning at startup in this case). A UDP source address is trivially spoofable, + so authentication matters at least as much here as it would over a connection. + + With ProxyBeaconSecret set identically on the proxy and all backends, each + announcement is signed with a SipHash-2-4 MAC over the message prefix (the key + is derived from the passphrase via apr_md5, as in mod_session_crypto). The + receiver recomputes the MAC (constant-time compare) and rejects anything that + does not match -- before the URL is parsed or acted on. Replay protection has + two parts, both keyed on the signed ts= (microseconds): (1) a freshness window + (ProxyBeaconMaxSkew) rejects messages whose timestamp is too far from now, + which assumes proxy and backend clocks are roughly synchronized (NTP); (2) a + per-url high-water mark rejects any ts= that does not strictly exceed the last + one accepted for that url, so a byte-identical replay (same ts, same MAC) -- + e.g. replaying a dead backend's last announcement to keep it from being evicted + -- is dropped even within the freshness window. Microsecond granularity ensures + genuine sub-second announcements always advance. seq= is NOT used for replay + (it resets when a backend restarts); the wall-clock ts= does not. + + Announcements are authenticated, not encrypted -- the payload is operational + metadata, not secret. For transport confidentiality, DTLS would be a separate, + orthogonal future layer. diff --git a/modules/proxy/config.m4 b/modules/proxy/config.m4 index 9dc42e83f0..110c546556 100644 --- a/modules/proxy/config.m4 +++ b/modules/proxy/config.m4 @@ -64,6 +64,9 @@ APACHE_MODULE(serf, [Reverse proxy module using Serf], , , no, [ APACHE_MODULE(proxy_express, mass reverse-proxy module. Requires --enable-proxy., , , most, , proxy) APACHE_MODULE(proxy_hcheck, [reverse-proxy health-check module. Requires --enable-proxy and --enable-watchdog.], , , most, , [proxy,watchdog]) +proxy_beacon_objs="mod_proxy_beacon.lo" +APACHE_MODULE(proxy_beacon, [reverse-proxy UDP member announcement; backends self-register as balancer members. Requires --enable-proxy, --enable-proxy-balancer and --enable-watchdog.], $proxy_beacon_objs, , no, , [proxy,watchdog,proxy_balancer]) + APR_ADDTO(INCLUDES, [-I\$(top_srcdir)/$modpath_current -I\$(top_srcdir)/modules/http2]) APACHE_MODPATH_FINISH diff --git a/modules/proxy/mod_proxy_beacon-guide.md b/modules/proxy/mod_proxy_beacon-guide.md new file mode 100644 index 0000000000..930fb0000e --- /dev/null +++ b/modules/proxy/mod_proxy_beacon-guide.md @@ -0,0 +1,277 @@ +# A Practical Guide to `mod_proxy_beacon` + +*Self-registering, self-healing reverse-proxy balancer membership over UDP.* + +--- + +## 1. What it does + +Normally, adding a backend to an httpd reverse-proxy load balancer means editing +the proxy config (or clicking around in `balancer-manager`) every time the fleet +changes. `mod_proxy_beacon` flips that around: **each backend announces itself** +to the proxy, and the proxy automatically: + +- **adds and enables** the backend as a live `BalancerMember` when it first + announces, and +- **disables it** (takes it out of rotation) when it stops announcing — then + **re-enables it** when it comes back. + +The result is a balancer whose membership tracks the running fleet with no +operator action. A backend added this way is a normal balancer member: it shows +up in `balancer-manager` and behaves exactly like a statically configured +`BalancerMember`. + +``` + backend-1 ──┐ (periodic UDP "BEACON" datagrams, + backend-2 ──┤ each advertising its own URL) + backend-3 ──┘ │ + ▼ + ┌──────────────┐ + │ reverse proxy│ ProxyBeaconListen :5555 + │ balancer:// │ → add / enable / evict members + │ cluster │ + └──────────────┘ + │ + clients +``` + +--- + +## 2. How it works (the short version) + +- **Transport:** plain **unicast UDP** datagrams (not multicast — multicast is + filtered on most networks and never crosses the public Internet). Backends + `sendto` the proxy; the proxy binds one socket and receives. No external + library — APR sockets for transport, APR-util SipHash for auth. +- **Fire-and-forget:** a lost datagram is simply recovered by the next periodic + announcement; reordered/replayed datagrams are rejected by a per-backend + timestamp check. No connection, reconnect, or framing layer. +- **Where it runs:** all the work happens in a single `mod_watchdog` **singleton** + child process, which owns the socket and applies membership changes. The change + propagates fleet-wide through the balancer's shared-memory “updated” timestamp, + the same way `balancer-manager` edits do. + +--- + +## 3. Requirements + +| Need | Why | +|------|-----| +| `mod_proxy` + `mod_proxy_balancer` | the balancer the members are added to | +| `mod_watchdog` | runs the background send/receive/evict loop | +| A **non-prefork** MPM (`event`, `worker`, …) | the watchdog singleton can't run under `prefork`; the module is silently inactive there | +| `mod_proxy_http` (or your backend protocol module) | to actually proxy the traffic | + +Build it with `--enable-proxy-beacon` (or `--enable-mods-shared=all`). Confirm +it's present: + +```sh +httpd -M 2>&1 | grep proxy_beacon # shared build +# or +httpd -l | grep mod_proxy_beacon # static build +``` + +--- + +## 4. Quick start + +### On the reverse proxy + +```apache +# Receive backend announcements on the cluster-facing interface (UDP/5555). +ProxyBeaconListen 0.0.0.0:5555 +ProxyBeaconSecret "a-long-random-shared-cluster-secret" +ProxyBeaconBalancer cluster + +# Drop a backend from rotation if it goes silent for 30s; re-add on next beacon. +ProxyBeaconTimeout 30 + +# An initially EMPTY balancer with spare slots for the dynamic members. + + ProxySet growth=16 + + +ProxyPass "/" "balancer://cluster/" +ProxyPassReverse "/" "balancer://cluster/" +``` + +### On each backend + +```apache +# Announce this backend's routable origin to the proxy every 10s (UDP). +ProxyBeaconAddress proxy.example.com:5555 +ProxyBeaconAdvertise http://10.0.0.5:8080 +ProxyBeaconSecret "a-long-random-shared-cluster-secret" +ProxyBeaconInterval 10 +``` + +That's it. Start the backends; within one announcement interval each one appears +as an enabled member of `balancer://cluster` on the proxy. Stop one, and after +`ProxyBeaconTimeout` it drops out of rotation; start it again and it returns. + +> **`ProxyBeaconAdvertise` must be the URL the *proxy* can reach** — the +> backend's routable origin, not `127.0.0.1`. It's validated at config-parse +> time and must be a full `scheme://host[:port]`. + +--- + +## 5. Directive reference + +### Proxy side (the receiver) + +| Directive | Default | Meaning | +|-----------|---------|---------| +| `ProxyBeaconListen [addr][:port]` | — | **Marks this server as the receiver.** Binds a UDP socket. `addr`/`port` are optional and inherited from the server's own `Listen`/`ServerName` when omitted (see note below). | +| `ProxyBeaconBalancer name` | — | Balancer that announced backends are added to. Bare name (`cluster`); a leading `balancer://` is stripped. Must already exist with spare `growth`. | +| `ProxyBeaconTimeout interval` | `0` (no eviction) | Seconds of silence before a member is disabled. `0` = add-only, never auto-remove. Set to a small multiple of the backends' interval to get self-healing. | +| `ProxyBeaconSecret secret` | — (unauthenticated) | Shared cluster secret; **same value on proxy and all backends.** | +| `ProxyBeaconMaxSkew interval` | `30` | Anti-replay freshness window: reject announcements whose signed timestamp is more than this far from now (either direction). | + +**Inheriting the listen address:** because UDP and TCP are separate port spaces, +`ProxyBeaconListen` can share the server's own service port without colliding +with the TCP listener. With no argument it binds the server's own address:port; +given just an address it inherits the port; etc. (The socket binds in an +unprivileged watchdog child, so you **can't** share a privileged port like 80/443 +this way — use an explicit high port there.) + +### Backend side (the sender) + +| Directive | Default | Meaning | +|-----------|---------|---------| +| `ProxyBeaconAddress addr:port` | — | **Marks this server as a sender.** UDP target = the proxy's `ProxyBeaconListen` address. | +| `ProxyBeaconAdvertise url` | — | The routable `scheme://host[:port]` the proxy adds as a member. Omit it and the backend beacons but advertises nothing (logged, never added). | +| `ProxyBeaconInterval interval` | `5` | How often this backend announces. Must be **meaningfully smaller** than the proxy's `ProxyBeaconTimeout`. | +| `ProxyBeaconSecret secret` | — | Same shared secret as the proxy. | + +> `ProxyBeaconListen` and `ProxyBeaconAddress` are **mutually exclusive** on the +> same server — a server is either a receiver or a sender, not both. + +All directives accept `server config` and `virtual host` context. Interval-style +directives use the standard httpd +[time-interval syntax](../../docs/manual/mod/directive-dict.html#Syntax) and +default to seconds (e.g. `ProxyBeaconInterval 10` or `... 500ms`). + +--- + +## 6. Security — read this before production + +The control channel decides where the proxy sends client traffic, so an +unauthenticated channel is a route-hijack / SSRF risk: anyone who can reach the +receive port could announce an arbitrary backend URL, and **UDP source addresses +are trivially spoofable.** + +**Always set `ProxyBeaconSecret`** in production (identical on proxy and every +backend). With it: + +- Each announcement is signed with a **SipHash-2-4 MAC** plus a timestamp; the + proxy recomputes the MAC (constant-time compare) and drops anything forged or + tampered **before parsing the URL**. +- **Replay protection is two-layered:** (1) the `ProxyBeaconMaxSkew` freshness + window rejects stale timestamps, and (2) a per-backend high-water mark rejects + any timestamp that doesn't strictly advance — so capturing and re-sending a + dead backend's last datagram (to keep it from being evicted) is rejected even + inside the freshness window. + +Operational notes: + +- **Clocks must be roughly in sync** (NTP). The timestamp check compares the + announcement's time against the proxy's clock; widen `ProxyBeaconMaxSkew` if + your hosts drift. +- With **no secret**, the channel is unauthenticated and the proxy logs a + one-time `UNAUTHENTICATED` warning at startup. +- The secret lives in your config file — **restrict its permissions like a + private key.** +- Announcements are **authenticated, not encrypted.** The payload is operational + metadata (URLs), not secrets. There's no transport confidentiality (DTLS would + be a separate future layer). + +--- + +## 7. Tuning + +- **Interval vs. timeout.** Keep `ProxyBeaconInterval` well under + `ProxyBeaconTimeout` so an occasional dropped datagram doesn't evict a healthy + backend. A common ratio is timeout ≈ 3–6× interval (e.g. interval 10s, timeout + 30–60s). +- **Size `growth` for your peak fleet.** A runtime-added member occupies one + growth slot **for the life of the proxy process** — it's *disabled* on + eviction, not removed (httpd can't remove workers at runtime, mirroring + `balancer-manager`). So `growth` must cover the maximum number of distinct + backend URLs you ever expect, not just the count live at one moment. +- **One proxy, many backends.** A single receiver serves the whole fleet. For + multiple proxies, run `ProxyBeaconListen` on each and point backends at all of + them (one `ProxyBeaconAddress` server per proxy, or accept that only the + configured proxy learns about the backend). + +--- + +## 8. Verifying & troubleshooting + +**See members appear:** enable `balancer-manager` on the proxy and watch the +worker list: + +```apache + + SetHandler balancer-manager + Require host localhost + +``` + +**Watch the log.** Bump verbosity for the module and tail the error log: + +```apache +LogLevel proxy_beacon:info +``` + +Useful log signals (grep the error log): + +| Log fragment | Means | +|--------------|-------| +| `received: BEACON ... url=...` | proxy is receiving announcements | +| `added backend ... balancer://cluster` | member created + enabled | +| `evicted backend ...` | a backend went silent past the timeout | +| `re-enabled backend ...` | a previously-evicted backend came back | +| `dropped ... mac mismatch` | wrong/missing secret on a sender (or a forged datagram) | +| `replayed/reordered ts` | a stale/duplicate datagram was rejected | +| `UNAUTHENTICATED` (startup) | no `ProxyBeaconSecret` — channel is open | + +**Common gotchas:** + +- *Backend never joins.* Usually a **secret mismatch** (look for `mac mismatch`) + or a firewall blocking **UDP** on the beacon port (it's UDP, not TCP — separate + rule). Confirm the proxy is logging `received: BEACON` at all. +- *Nothing happens, no errors.* You're probably on the **prefork MPM** — the + watchdog singleton doesn't run there, so the module is inactive. Switch to + `event`/`worker`. +- *Traffic 503s to a backend that's “up”.* Check that `ProxyBeaconAdvertise` is + the URL the **proxy** can reach (not the backend's loopback) and that the + balancer has free `growth` slots. +- *Healthy backend keeps getting evicted.* Interval too close to timeout, or + clock skew larger than `ProxyBeaconMaxSkew`. Widen the timeout/skew or sync + clocks. + +--- + +## 9. The wire format (for the curious) + +One ASCII, space-separated, key=value datagram per announcement: + +``` +BEACON url=http://host:port host= pid= seq= ts= mac= +``` + +- `url=` — the routable origin added as a `BalancerMember`. +- `ts=` (microseconds since epoch) and `mac=` — present only when a secret is set; + `ts` drives replay protection, `mac` is the SipHash signature. +- `host=` / `pid=` / `seq=` — informational. (`seq` resets on backend restart, so + it's *not* used for replay detection — the wall-clock `ts` is.) + +--- + +## 10. See also + +- `mod_proxy`, `mod_proxy_balancer` — the balancer these members live in +- `mod_proxy_hcheck` — active health checking, complementary to beacon liveness +- `mod_watchdog` — the background runner this module rides on +- `modules/proxy/README.beacon` — the architecture/internals companion to this guide +- `docs/manual/mod/mod_proxy_beacon.xml` — the formal directive reference diff --git a/modules/proxy/mod_proxy_beacon.c b/modules/proxy/mod_proxy_beacon.c new file mode 100644 index 0000000000..a0c9567ca5 --- /dev/null +++ b/modules/proxy/mod_proxy_beacon.c @@ -0,0 +1,1241 @@ +/* Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +/* + * mod_proxy_beacon -- a UDP datagram announcement channel between backend + * servers and a front-end reverse proxy, by which backends self-register as + * balancer members. + * + * This module uses plain APR UDP sockets, adapting the socket-handling pattern + * of mod_heartmonitor. UDP datagrams are a natural fit for small, periodic, + * fire-and-forget announcements, and avoid a third-party dependency. + * + * UNICAST, not multicast: unlike mod_heartbeat (which multicasts to a group), + * each backend sends a point-to-point datagram to the proxy's configured address + * (apr_socket_sendto to a single resolved unicast sockaddr). Multicast is + * filtered on most networks and does not traverse the public Internet, so a + * cross-host control plane must be unicast. The proxy is a fixed unicast + * rendezvous; each backend is configured with its address. + * + * Direction: backends announce themselves to the proxy. The proxy binds a + * stable UDP port (the rendezvous) and receives; each backend sends to it: + * + * - Reverse proxy: ProxyBeaconListen 0.0.0.0:5555 (receiver, binds) + * - Backend server: ProxyBeaconAddress proxy-host:5555 (sender, sendto) + * + * A leading scheme (e.g. tcp://) in the address is accepted and ignored, so + * addresses written with a transport scheme keep working. Each backend sends + * a "BEACON" datagram every interval. Datagrams are fire-and-forget: a lost + * packet just delays an update by one interval, and reordering is rejected by + * the per-url monotonic timestamp (Phase 4), so the design tolerates UDP's lack + * of delivery/ordering guarantees without a reconnect or framing layer. + * + * Phase 1 (the channel): the proxy simply logged each announcement, proving the + * channel works between separate, remote servers. + * + * Phase 2: when the proxy receives an announcement that carries a backend + * URL, it ADDS that backend as a member (worker) of a configured balancer and + * ENABLES it -- the dynamic equivalent of "Add Worker" in the balancer-manager + * web UI, but driven by the backend announcing itself. We reuse + * mod_proxy_balancer's exported balancer_manage() optional function (the same + * core balancer_handler calls), so the add path is identical to the proven + * manager flow. The backend advertises its routable URL via ProxyBeaconAdvertise; + * the proxy names the target balancer via ProxyBeaconBalancer. + * + * Phase 3: the inverse -- when a backend STOPS announcing for longer than + * ProxyBeaconTimeout, the proxy takes it out of rotation by setting the worker's + * DISABLED flag (again via balancer_manage), and re-enables it if the backend + * starts announcing again. This gives the cluster automatic liveness without + * operator action. The timeout is a config directive (0 = disabled = pure + * Phase-2 behavior). All of this runs in the watchdog singleton that already + * owns the socket, so the per-URL last-seen table lives in this process's + * ctx -- no shared memory. + * + * Phase 4: authenticate the channel. Without this, anyone who can reach the + * proxy's bound port can announce an arbitrary url and the proxy will route real + * client traffic to it (SSRF / hijack) -- and a UDP source address is trivially + * spoofable, so authentication matters as much here as on any control plane. With + * a pre-shared cluster secret (ProxyBeaconSecret) on both ends, the sender appends + * a keyed MAC (SipHash-2-4 via APR-util -- the same primitive mod_session_crypto + * uses) plus a timestamp; the receiver recomputes the MAC and checks timestamp + * freshness (anti-replay), dropping any forged, tampered, or stale message + * before it is parsed/acted on. Authentication is opt-in: with no secret the + * channel behaves as before but logs a one-time "UNAUTHENTICATED" warning. We + * authenticate, not encrypt -- the payload (backend URLs) is not secret; for + * transport confidentiality DTLS would be a separate, orthogonal future layer. + * + * The background work runs on a mod_watchdog SINGLETON instance (exactly one + * child process owns the socket), mirroring mod_proxy_hcheck.c. That child has + * no request_rec, so the receiver synthesizes a minimal one for balancer_manage() + * (see beacon_make_fake_request -- adapted from mod_proxy_hcheck's + * create_request_rec, plus a fake conn_rec that ap_log_rerror requires). + */ + +#include "httpd.h" +#include "http_config.h" +#include "http_core.h" +#include "http_log.h" +#include "http_protocol.h" +#include "ap_provider.h" +#include "ap_listen.h" +#include "mod_watchdog.h" +#include "mod_proxy.h" + +#include "apr_strings.h" +#include "apr_time.h" +#include "apr_hash.h" +#include "apr_uri.h" +#include "apr_network_io.h" +#include "apr_siphash.h" +#include "apr_md5.h" + +#if APR_HAVE_UNISTD_H +#include /* for getpid() */ +#endif + +module AP_MODULE_DECLARE_DATA proxy_beacon_module; + +#define BEACON_WATCHDOG_NAME "_proxy_beacon_" +#define BEACON_DEFAULT_INTERVAL apr_time_from_sec(5) + +typedef enum { + BEACON_ROLE_NONE = 0, + BEACON_ROLE_SEND, /* backend: dials the proxy and announces itself */ + BEACON_ROLE_LISTEN /* reverse proxy: listens and receives announcements */ +} beacon_role_e; + +/* listener-side per-backend state, keyed by url in ctx->seen. */ +typedef struct { + apr_time_t last_seen; /* updated on every beacon for this url */ + apr_time_t last_attempt; /* last add attempt (throttles retry when full) */ + apr_int64_t last_ts; /* highest signed ts= accepted (replay guard) */ + int added; /* worker exists in the balancer (add dedup) */ + int evicted; /* WE set DISABLED on timeout (guards flip churn) */ +} beacon_member_t; + +typedef struct { + apr_pool_t *p; /* subpool for this context */ + server_rec *s; /* canonical server identity */ + char *pub_url; /* ProxyBeaconAddress (backend dial url) */ + char *sub_url; /* ProxyBeaconListen (proxy listen url) */ + char *advertise_url;/* ProxyBeaconAdvertise (sender: routable url) */ + char *balancer_name;/* ProxyBeaconBalancer (listener: bare bal name) */ + apr_interval_time_t interval; /* ProxyBeaconInterval (announce period) */ + apr_interval_time_t evict_timeout;/* ProxyBeaconTimeout (0 = no eviction) */ + beacon_role_e role; + apr_socket_t *sock; /* live UDP socket; valid only when opened */ + apr_sockaddr_t *addr; /* listener: bind addr; sender: destination addr */ + int opened; /* guard double-open / stop-without-start */ + apr_time_t last_publish; /* sender throttle */ + apr_time_t last_sweep; /* listener eviction-scan throttle */ + apr_uint64_t seq; /* announcement counter */ + apr_hash_t *seen; /* listener: url -> beacon_member_t* */ + APR_OPTIONAL_FN_TYPE(balancer_manage) *manage_fn; /* listener: cached, lazy */ + /* Phase 4: channel authentication. */ + int has_secret; /* ProxyBeaconSecret was set */ + unsigned char mac_key[APR_MD5_DIGESTSIZE]; /* siphash key (16 bytes) */ + apr_int64_t max_skew; /* ProxyBeaconMaxSkew, seconds (replay window) */ + apr_uint64_t reject_count; /* listener: dropped-since-last-log counter */ + apr_time_t last_reject_log; /* listener: reject-log rate limiter */ + int warned_insecure; /* listener: emitted the one-time warning */ + int open_failed; /* STARTING: socket open/listen/dial failed permanently */ +} beacon_ctx_t; + +/* SipHash-2-4 output is 8 bytes -> 16 hex chars (+ NUL). */ +#define BEACON_MAC_HEXLEN (APR_SIPHASH_DSIZE * 2) +#define BEACON_MAXSKEW_DEFAULT 30 /* seconds */ + +/* Max announcement datagram we accept. Our messages are a few hundred bytes; + * anything larger is not one of ours and is truncated on receive. */ +#define BEACON_MAX_MSG_LEN 2048 + +/* When an add fails (e.g. the balancer is full), don't retry on every + * announcement -- back off this long between attempts for a given url. */ +#define BEACON_RETRY_BACKOFF apr_time_from_sec(60) + +/* Cap on tracked backend urls, to bound memory against an unauthenticated + * channel announcing unboundedly many distinct urls. Once reached, unknown + * urls are dropped (rate-limited log) rather than tracked/added. */ +#define BEACON_MAX_MEMBERS 256 + +/* Process-wide watchdog handle, like mod_proxy_hcheck's static watchdog. */ +static ap_watchdog_t *beacon_watchdog; + +/* + * Per-server config. No socket calls here -- the role may not even be known yet, + * and this runs in the parent at config-parse time (possibly more than once). + */ +static void *beacon_create_server_config(apr_pool_t *p, server_rec *s) +{ + beacon_ctx_t *ctx = apr_pcalloc(p, sizeof(beacon_ctx_t)); + + ctx->s = s; + apr_pool_create(&ctx->p, p); + apr_pool_tag(ctx->p, "proxy_beacon"); + ctx->interval = BEACON_DEFAULT_INTERVAL; + ctx->role = BEACON_ROLE_NONE; + ctx->seen = apr_hash_make(ctx->p); + + return ctx; +} + +static const char *beacon_set_sub_url(cmd_parms *cmd, void *dummy, + int argc, char *const argv[]) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + if (argc > 1) { + return "ProxyBeaconListen takes at most one argument: [host][:port]"; + } + if (ctx->pub_url) { + return "ProxyBeaconListen and ProxyBeaconAddress are mutually exclusive " + "on the same server"; + } + /* No argument -> inherit both host and port from this server; one argument + * -> "[host][:port]", an omitted host or port being inherited from the + * server too. Store "" for the no-argument case (resolved against the + * server's own address at startup). */ + ctx->sub_url = (argc == 1) ? apr_pstrdup(cmd->pool, argv[0]) : ""; + ctx->role = BEACON_ROLE_LISTEN; + return NULL; +} + +static const char *beacon_set_pub_url(cmd_parms *cmd, void *dummy, const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + if (ctx->sub_url) { + return "ProxyBeaconListen and ProxyBeaconAddress are mutually exclusive " + "on the same server"; + } + ctx->pub_url = apr_pstrdup(cmd->pool, arg); + ctx->role = BEACON_ROLE_SEND; + return NULL; +} + +static const char *beacon_set_advertise(cmd_parms *cmd, void *dummy, + const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + apr_uri_t uri; + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + /* Must be a routable backend URL (scheme://host[:port]); it becomes a + * BalancerMember on the proxy. */ + if (apr_uri_parse(cmd->pool, arg, &uri) != APR_SUCCESS + || !uri.scheme || !uri.hostname) { + return apr_psprintf(cmd->pool, + "ProxyBeaconAdvertise: '%s' is not a valid " + "scheme://host[:port] URL", arg); + } + ctx->advertise_url = apr_pstrdup(cmd->pool, arg); + return NULL; +} + +static const char *beacon_set_balancer(cmd_parms *cmd, void *dummy, + const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + /* Store the bare name; balancer_manage() prepends BALANCER_PREFIX itself. + * Be forgiving if the admin wrote the full "balancer://name". */ + if (!strncasecmp(arg, BALANCER_PREFIX, sizeof(BALANCER_PREFIX) - 1)) { + arg += sizeof(BALANCER_PREFIX) - 1; + } + if (!*arg) { + return "ProxyBeaconBalancer: empty balancer name"; + } + ctx->balancer_name = apr_pstrdup(cmd->pool, arg); + return NULL; +} + +static const char *beacon_set_interval(cmd_parms *cmd, void *dummy, + const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + apr_interval_time_t iv; + apr_status_t rv; + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + rv = ap_timeout_parameter_parse(arg, &iv, "s"); + if (rv != APR_SUCCESS) { + return "Unparse-able ProxyBeaconInterval setting"; + } + if (iv < AP_WD_TM_SLICE) { + return apr_psprintf(cmd->pool, + "ProxyBeaconInterval must be greater than %" + APR_TIME_T_FMT "ms", + apr_time_as_msec(AP_WD_TM_SLICE)); + } + ctx->interval = iv; + return NULL; +} + +static const char *beacon_set_timeout(cmd_parms *cmd, void *dummy, + const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + apr_interval_time_t iv; + apr_status_t rv; + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + rv = ap_timeout_parameter_parse(arg, &iv, "s"); + if (rv != APR_SUCCESS || iv < 0) { + return "ProxyBeaconTimeout must be a non-negative time (0 disables " + "eviction)"; + } + /* 0 = eviction disabled (pure announce-and-add behavior). */ + ctx->evict_timeout = iv; + return NULL; +} + +static const char *beacon_set_secret(cmd_parms *cmd, void *dummy, const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + if (!*arg) { + return "ProxyBeaconSecret: empty secret"; + } + /* Derive a 16-byte SipHash key from the passphrase. This is key + * derivation (spreading the shared secret to the key size both ends use), + * not message hashing, so MD5 is fine here -- same approach as + * mod_session_crypto.c's compute_auth(). */ + apr_md5(ctx->mac_key, arg, strlen(arg)); + ctx->has_secret = 1; + return NULL; +} + +static const char *beacon_set_maxskew(cmd_parms *cmd, void *dummy, const char *arg) +{ + beacon_ctx_t *ctx = ap_get_module_config(cmd->server->module_config, + &proxy_beacon_module); + apr_interval_time_t iv; + apr_status_t rv; + const char *err = ap_check_cmd_context(cmd, NOT_IN_HTACCESS); + if (err) { + return err; + } + rv = ap_timeout_parameter_parse(arg, &iv, "s"); + if (rv != APR_SUCCESS || iv <= 0) { + return "ProxyBeaconMaxSkew must be a positive time (replay window)"; + } + ctx->max_skew = apr_time_sec(iv); + return NULL; +} + +static const command_rec beacon_cmds[] = { + AP_INIT_TAKE_ARGV("ProxyBeaconListen", beacon_set_sub_url, NULL, RSRC_CONF, + "reverse proxy: UDP address [host][:port] to receive beacons " + "on. An omitted host or port (or no argument at all) is " + "inherited from this server's own address and port, so the " + "beacon channel can share the service endpoint. " + "e.g. 0.0.0.0:5555"), + AP_INIT_TAKE1("ProxyBeaconAddress", beacon_set_pub_url, NULL, RSRC_CONF, + "backend: UDP address host:port of the proxy to send beacons " + "to, e.g. proxy-host:5555"), + AP_INIT_TAKE1("ProxyBeaconInterval", beacon_set_interval, NULL, RSRC_CONF, + "backend announcement interval in seconds (default 5)"), + AP_INIT_TAKE1("ProxyBeaconAdvertise", beacon_set_advertise, NULL, RSRC_CONF, + "backend: routable URL to advertise to the proxy, " + "e.g. http://10.0.0.5:8080 (added as a BalancerMember)"), + AP_INIT_TAKE1("ProxyBeaconBalancer", beacon_set_balancer, NULL, RSRC_CONF, + "proxy: name of the balancer that announced backends are " + "added to, e.g. mycluster (for balancer://mycluster)"), + AP_INIT_TAKE1("ProxyBeaconTimeout", beacon_set_timeout, NULL, RSRC_CONF, + "proxy: seconds without an announcement after which a backend " + "is disabled (taken out of rotation); 0 disables eviction"), + AP_INIT_TAKE1("ProxyBeaconSecret", beacon_set_secret, NULL, RSRC_CONF, + "pre-shared cluster secret; set on both proxy and backends to " + "authenticate announcements (SipHash MAC). Keep the conf file " + "readable only by the server user."), + AP_INIT_TAKE1("ProxyBeaconMaxSkew", beacon_set_maxskew, NULL, RSRC_CONF, + "proxy: max allowed seconds between an announcement's timestamp " + "and now (anti-replay window; requires NTP-synced clocks). " + "Default 30 seconds."), + { NULL } +}; + +/* + * Resolve a "[scheme://][host][:port]" address string to a UDP sockaddr. Any + * leading scheme (e.g. tcp://) is accepted and ignored, so addresses written + * with a transport scheme keep working. A NULL/empty url, or an omitted host + * or port, is filled from the caller-supplied defaults: + * - default_host NULL -> bind/resolve the wildcard address (all interfaces); + * - default_port 0 -> the port is then required (APR_EINVAL if missing). + * Letting the host/port be inherited lets the beacon channel share the server's + * own address:port; UDP and TCP are independent port spaces, so binding UDP:N + * does not collide with the server's TCP listener. *addr_out is from p. + */ +static apr_status_t beacon_resolve(apr_pool_t *p, const char *url, + const char *default_host, + apr_port_t default_port, + apr_sockaddr_t **addr_out) +{ + char *host = NULL, *scope; + apr_port_t port = 0; + apr_status_t rv; + + if (url && *url) { + const char *hostport = url, *sep; + if ((sep = strstr(url, "://")) != NULL) { + hostport = sep + 3; + } + rv = apr_parse_addr_port(&host, &scope, &port, hostport, p); + if (rv != APR_SUCCESS) { + return rv; + } + } + if (!host || !*host) { + host = (char *)default_host; /* NULL -> wildcard (any address) */ + } + if (!port) { + port = default_port; + } + if (!port) { + return APR_EINVAL; /* no port given and none to inherit */ + } + return apr_sockaddr_info_get(addr_out, host, APR_UNSPEC, port, 0, p); +} + +/* + * The listener may inherit its bind host/port from this server (so the beacon + * channel can share the server's own endpoint). Derive a concrete address/port + * from the server's actual bound listeners (ap_listeners) -- this reflects the + * real Listen socket (e.g. 0.0.0.0), which is what the beacon should match, and + * avoids leaving the host NULL (which resolves to the IPv6 wildcard and could + * miss IPv4 senders on a v6-only-strict host). The port is the server's own + * (ServerName/Listen) port; the host is taken from the listener on that port, + * else the first listener. Either output may remain unset (host NULL / port 0) + * if no listeners are configured. + */ +static void beacon_server_default_addr(server_rec *s, const char **host_out, + apr_port_t *port_out) +{ + const char *host = NULL; + apr_port_t port = s->port; + ap_listen_rec *lr, *match = NULL; + + for (lr = ap_listeners; lr; lr = lr->next) { + if (!lr->bind_addr) { + continue; + } + if (!match) { + match = lr; /* first, as a fallback */ + } + if (port && lr->bind_addr->port == port) { + match = lr; /* prefer the server's own port */ + break; + } + } + if (match && match->bind_addr) { + char *ip = NULL; + if (!port) { + port = match->bind_addr->port; + } + if (apr_sockaddr_ip_get(&ip, match->bind_addr) == APR_SUCCESS) { + host = ip; + } + } + + *host_out = host; + *port_out = port; +} + +/* STARTING: open the UDP socket; listener binds to receive, sender resolves + * its destination. */ +static void beacon_cb_starting(beacon_ctx_t *ctx) +{ + server_rec *s = ctx->s; + apr_status_t rv; + + /* A prior open/bind failure is treated as permanent: the admin must fix the + * configuration (bad address, port conflict) and restart. Without this + * guard the watchdog would retry every ~100ms and fill the error log. */ + if (ctx->opened || ctx->open_failed) { + return; + } + + if (ctx->role == BEACON_ROLE_LISTEN) { + /* An omitted host and/or port in ProxyBeaconListen is inherited from + * this server's own address/port (ServerName/Listen), so the beacon + * channel can share the server's service endpoint. UDP and TCP are + * independent, so binding UDP: does not collide with the server's + * TCP listener. (A privileged port still cannot be bound here -- this + * callback runs in an unprivileged watchdog child -- so endpoint sharing + * is for non-privileged service ports.) */ + const char *defhost; + apr_port_t defport; + beacon_server_default_addr(s, &defhost, &defport); + if ((rv = beacon_resolve(ctx->p, ctx->sub_url, defhost, defport, + &ctx->addr)) != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, + APLOGNO(10567) "mod_proxy_beacon: cannot resolve " + "ProxyBeaconListen address '%s' (an omitted port " + "requires a known server port)", + *ctx->sub_url ? ctx->sub_url : "(inherit)"); + ctx->open_failed = 1; + return; + } + if ((rv = apr_socket_create(&ctx->sock, ctx->addr->family, SOCK_DGRAM, + APR_PROTO_UDP, ctx->p)) != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, + APLOGNO(10568) "mod_proxy_beacon: apr_socket_create failed"); + ctx->open_failed = 1; + return; + } + apr_socket_opt_set(ctx->sock, APR_SO_REUSEADDR, 1); + apr_socket_opt_set(ctx->sock, APR_SO_NONBLOCK, 1); + if ((rv = apr_socket_bind(ctx->sock, ctx->addr)) != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, + APLOGNO(10569) "mod_proxy_beacon: bind to %pI failed", + ctx->addr); + apr_socket_close(ctx->sock); + ctx->open_failed = 1; + return; + } + ctx->opened = 1; + ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, s, + APLOGNO(10570) "mod_proxy_beacon: listening for beacons on %pI", + ctx->addr); + /* Warn if no balancer is configured: beacons will be logged but no + * backend workers will be added (Phase-1 log-only behavior). */ + if (!ctx->balancer_name) { + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, + APLOGNO(10571) "mod_proxy_beacon: listener on %pI has no " + "ProxyBeaconBalancer; beacons will be logged only. " + "Set ProxyBeaconBalancer to add announced backends.", + ctx->addr); + } + /* Phase 4: warn once if we'll add members from an unauthenticated + * channel (anyone who can reach this port could announce a backend). */ + if (ctx->balancer_name && !ctx->has_secret && !ctx->warned_insecure) { + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, + APLOGNO(10572) "mod_proxy_beacon: beacon channel on %pI is " + "UNAUTHENTICATED; set ProxyBeaconSecret on the proxy and " + "all backends to require signed beacons", ctx->addr); + ctx->warned_insecure = 1; + } + } + else if (ctx->role == BEACON_ROLE_SEND) { + /* Sender: the destination host:port targets the proxy and cannot be + * inferred from this backend's own address, so both must be given + * explicitly (no defaults). */ + if ((rv = beacon_resolve(ctx->p, ctx->pub_url, NULL, 0, &ctx->addr)) + != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, + APLOGNO(10573) "mod_proxy_beacon: cannot resolve " + "ProxyBeaconAddress '%s' (must be host:port)", + ctx->pub_url); + ctx->open_failed = 1; + return; + } + if ((rv = apr_socket_create(&ctx->sock, ctx->addr->family, SOCK_DGRAM, + APR_PROTO_UDP, ctx->p)) != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_ERR, rv, s, + APLOGNO(10574) "mod_proxy_beacon: apr_socket_create failed"); + ctx->open_failed = 1; + return; + } + /* UDP is connectionless: no dial, no reconnect. Datagrams are sent to + * ctx->addr each interval; if the proxy isn't up yet they are simply + * dropped and the next interval tries again. */ + apr_socket_opt_set(ctx->sock, APR_SO_NONBLOCK, 1); + ctx->opened = 1; + ctx->last_publish = 0; /* beacon on the first running tick */ + ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, s, + APLOGNO(10575) "mod_proxy_beacon: beaconing to %pI", ctx->addr); + } +} + +/* + * Synthesize a minimal request_rec for balancer_manage(), which runs here in + * the watchdog thread with no real client request. Adapted from + * mod_proxy_hcheck.c's create_request_rec (same watchdog context), but that one + * attaches a real backend conn_rec before logging; we have none, so we fabricate + * a minimal conn_rec too. + * + * balancer_manage()/balancer_process_balancer_worker() touch only r->pool, + * r->server, and ap_log_rerror(). ap_log_rerror is the trap: log_error_core() + * asserts r->connection != NULL and do_errorlog_default() unconditionally reads + * r->connection->outgoing, while add_log_id()->core_generate_log_id() reads + * c->current_thread. So we (a) give r a non-NULL conn_rec with ->outgoing set, + * and (b) pre-set r->log_id / c->log_id so the log-id generation path is skipped + * entirely. + */ +static request_rec *beacon_make_fake_request(apr_pool_t *p, server_rec *s) +{ + conn_rec *c = apr_pcalloc(p, sizeof(*c)); + request_rec *r = apr_pcalloc(p, sizeof(*r)); + + c->pool = p; + c->base_server = s; + c->log = &s->log; + c->log_id = "beacon"; /* non-NULL -> skip add_log_id/log-id gen */ + c->outgoing = 1; /* read by do_errorlog_default() */ + c->client_ip = "-"; + c->notes = apr_table_make(p, 1); + + r->pool = p; + r->server = s; + r->connection = c; + r->log = &s->log; + r->log_id = "beacon"; + r->per_dir_config = s->lookup_defaults; + r->request_config = ap_create_request_config(p); + r->notes = apr_table_make(p, 4); + r->subprocess_env = apr_table_make(p, 4); + r->headers_in = apr_table_make(p, 1); + r->headers_out = apr_table_make(p, 1); + r->err_headers_out = apr_table_make(p, 1); + r->useragent_ip = "-"; + r->useragent_addr = NULL; + r->proxyreq = PROXYREQ_PROXY; + r->status = HTTP_OK; + + return r; +} + +/* + * Resolve mod_proxy_balancer's balancer_manage() optional fn (lazily, cached). + * Registered at hook-registration time, so available by the time we run. + */ +static int beacon_have_manage_fn(beacon_ctx_t *ctx) +{ + if (!ctx->manage_fn) { + ctx->manage_fn = APR_RETRIEVE_OPTIONAL_FN(balancer_manage); + if (!ctx->manage_fn) { + ap_log_error(APLOG_MARK, APLOG_ERR, 0, ctx->s, + APLOGNO(10576) "mod_proxy_beacon: balancer_manage unavailable " + "(is mod_proxy_balancer loaded?)"); + return 0; + } + } + return 1; +} + +/* + * Set (or clear) a worker's DISABLED flag via balancer_manage -- the exact path + * the balancer-manager UI uses (params { b, w, w_status_D }). This is how both + * "enable" (disabled=0) and "evict" (disabled=1) are performed. Returns + * APR_SUCCESS on success. + */ +static apr_status_t beacon_set_worker_disabled(beacon_ctx_t *ctx, apr_pool_t *pool, + const char *url, int disabled) +{ + apr_pool_t *subp; + request_rec *r; + apr_table_t *params; + apr_status_t rv; + + apr_pool_create(&subp, pool); + r = beacon_make_fake_request(subp, ctx->s); + + params = apr_table_make(subp, 4); + apr_table_setn(params, "b", ctx->balancer_name); + apr_table_setn(params, "w", url); + apr_table_setn(params, "w_status_D", disabled ? "1" : "0"); + rv = ctx->manage_fn(r, params); + + apr_pool_destroy(subp); + return rv; +} + +/* + * Emit a rate-limited (~1/s) message about dropped/failed announcements, so a + * full balancer or a flood of bad urls can't flood the error log. Shares the + * counter/timer with the Phase-4 rejection logging. + */ +static void beacon_log_throttled(beacon_ctx_t *ctx, apr_time_t now, const char *what) +{ + ctx->reject_count++; + if (ctx->last_reject_log == 0 + || now - ctx->last_reject_log >= apr_time_from_sec(1)) { + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, ctx->s, + APLOGNO(10577) "mod_proxy_beacon: dropped %" APR_UINT64_T_FMT + " announcement(s) (last: %s)", ctx->reject_count, what); + ctx->reject_count = 0; + ctx->last_reject_log = now; + } +} + +/* + * Attempt to add (and enable) url as a member of the configured balancer, + * reusing balancer_manage(). Two calls: add (the worker is created DISABLED by + * default), then clear the DISABLED flag so it serves traffic. Updates the + * caller-owned member entry m: m->added is set on success. Returns APR_SUCCESS + * on success. A failure here (e.g. the balancer is full) is expected and + * handled by the caller via backoff, not retried every announcement. + */ +static apr_status_t beacon_try_add(beacon_ctx_t *ctx, apr_pool_t *pool, + const char *url, beacon_member_t *m, + apr_time_t now) +{ + server_rec *s = ctx->s; + apr_pool_t *subp; + request_rec *r; + apr_table_t *params; + apr_status_t rv; + + m->last_attempt = now; + + if (!beacon_have_manage_fn(ctx)) { + return APR_EGENERAL; + } + + apr_pool_create(&subp, pool); + r = beacon_make_fake_request(subp, s); + + /* Step 1: add the worker (created DISABLED). */ + params = apr_table_make(subp, 4); + apr_table_setn(params, "b", ctx->balancer_name); + apr_table_setn(params, "b_nwrkr", url); + apr_table_setn(params, "b_wyes", "1"); + rv = ctx->manage_fn(r, params); + apr_pool_destroy(subp); + if (rv != APR_SUCCESS) { + /* Most likely the balancer has no free slots (grow it via ProxySet + * growth / BalancerGrowth). Throttled log; retried only after the + * backoff window, not on every announcement. */ + beacon_log_throttled(ctx, now, "balancer add failed (full?)"); + return rv; + } + + /* Step 2: enable it (clear the DISABLED flag) so it serves traffic. + * beacon_set_worker_disabled is self-contained -- it creates its own subpool + * from `pool` -- so it does not rely on `subp`, which we destroyed above. */ + rv = beacon_set_worker_disabled(ctx, pool, url, 0); + if (rv != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_WARNING, 0, s, + APLOGNO(10578) "mod_proxy_beacon: added but failed to enable worker %s " + "in balancer://%s", url, ctx->balancer_name); + /* fall through: still a member, just disabled */ + } + + m->added = 1; + m->evicted = 0; + ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, s, + APLOGNO(10579) "mod_proxy_beacon: added backend %s to balancer://%s", + url, ctx->balancer_name); + return APR_SUCCESS; +} + +/* + * Handle one announced url. Tracks every url in ctx->seen (capped) so that: + * - a successful member is deduplicated and its last_seen refreshed; + * - a previously-evicted member is re-enabled when it announces again; + * - an add that failed (e.g. balancer full) is retried only after a backoff, + * not on every announcement. + * msg_ts is the signed timestamp (microseconds) from the verified message, or 0 + * when the channel is unauthenticated. + */ +static void beacon_handle_announce(beacon_ctx_t *ctx, apr_pool_t *pool, + const char *url, apr_time_t now, + apr_int64_t msg_ts) +{ + beacon_member_t *m = apr_hash_get(ctx->seen, url, APR_HASH_KEY_STRING); + + if (!m) { + /* Bound memory: don't track unboundedly many distinct urls (a concern + * on an unauthenticated channel). */ + if (apr_hash_count(ctx->seen) >= BEACON_MAX_MEMBERS) { + beacon_log_throttled(ctx, now, "member table full"); + return; + } + m = apr_pcalloc(ctx->p, sizeof(*m)); + m->last_seen = now; + m->last_ts = msg_ts; + apr_hash_set(ctx->seen, apr_pstrdup(ctx->p, url), APR_HASH_KEY_STRING, + m); + beacon_try_add(ctx, pool, url, m, now); + return; + } + + /* Anti-replay (2): on an authenticated channel, each url's signed ts must + * strictly increase. A replayed (byte-identical) announcement carries a ts + * we've already accepted, so reject it -- this closes the in-window replay + * the freshness check alone allows (e.g. replaying a dead backend's last + * announcement to keep it from being evicted). */ + if (ctx->has_secret && msg_ts <= m->last_ts) { + beacon_log_throttled(ctx, now, "replayed/reordered ts"); + return; + } + m->last_ts = msg_ts; + m->last_seen = now; + + if (!m->added) { + /* A prior add failed; retry, but only after the backoff window. */ + if (now - m->last_attempt >= BEACON_RETRY_BACKOFF) { + beacon_try_add(ctx, pool, url, m, now); + } + return; + } + + if (m->evicted) { + if (beacon_set_worker_disabled(ctx, pool, url, 0) == APR_SUCCESS) { + m->evicted = 0; + ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, ctx->s, + APLOGNO(10580) "mod_proxy_beacon: re-enabled backend %s in " + "balancer://%s (announcing again)", + url, ctx->balancer_name); + } + } +} + +/* + * Eviction sweep: disable any member that has not announced within + * evict_timeout. No-op when eviction is disabled (timeout 0). Runs in the + * watchdog singleton, throttled by the caller. + */ +static void beacon_sweep(beacon_ctx_t *ctx, apr_pool_t *pool, apr_time_t now) +{ + apr_hash_index_t *hi; + + if (ctx->evict_timeout <= 0 || !beacon_have_manage_fn(ctx)) { + return; + } + + for (hi = apr_hash_first(pool, ctx->seen); hi; hi = apr_hash_next(hi)) { + const void *key; + void *val; + beacon_member_t *m; + + apr_hash_this(hi, &key, NULL, &val); + m = val; + if (!m->added || m->evicted) { + continue; + } + if (now - m->last_seen > ctx->evict_timeout) { + const char *url = key; + if (beacon_set_worker_disabled(ctx, pool, url, 1) == APR_SUCCESS) { + m->evicted = 1; + ap_log_error(APLOG_MARK, APLOG_NOTICE, 0, ctx->s, + APLOGNO(10581) "mod_proxy_beacon: evicted backend %s from " + "balancer://%s (no announcement for %" + APR_TIME_T_FMT "s)", + url, ctx->balancer_name, + apr_time_sec(now - m->last_seen)); + } + } + } +} + +/* + * Extract the value of the "url=" token from a beacon message into buf. + * Returns 1 on success. The payload is untrusted network input, so parse + * defensively: the token is "url=" followed by non-space characters. + */ +static int beacon_parse_url(const char *msg, char *buf, apr_size_t buflen) +{ + const char *p = msg; + + while (p && *p) { + if (!strncmp(p, "url=", 4)) { + apr_size_t i = 0; + p += 4; + while (*p && *p != ' ' && i + 1 < buflen) { + buf[i++] = *p++; + } + buf[i] = '\0'; + return (i > 0); + } + /* advance to the next space-separated token */ + p = strchr(p, ' '); + if (p) { + p++; + } + } + return 0; +} + +/* ----- Phase 4: channel authentication (SipHash MAC + timestamp) ----- */ + +/* Constant-time buffer compare (time depends only on size, not contents) -- + * avoids a timing oracle on the MAC. apr_crypto_equals() would do this but is + * gated behind APU_HAVE_CRYPTO, which isn't always built; this is the same + * branchless idiom. */ +static int beacon_const_time_eq(const void *a, const void *b, apr_size_t n) +{ + const unsigned char *p1 = a, *p2 = b; + volatile unsigned char diff = 0; + apr_size_t i; + for (i = 0; i < n; i++) { + diff |= p1[i] ^ p2[i]; + } + return diff == 0; +} + +/* Hex-encode the SipHash-2-4 MAC of [base, base+len) into out (BEACON_MAC_HEXLEN+1 + * bytes). */ +static void beacon_mac_hex(const beacon_ctx_t *ctx, const char *base, + apr_size_t len, char *out) +{ + unsigned char mac[APR_SIPHASH_DSIZE]; + apr_siphash24_auth(mac, base, len, ctx->mac_key); + ap_bin2hex(mac, sizeof(mac), out); /* writes BEACON_MAC_HEXLEN + NUL */ +} + +/* sender: return " mac=" (signed) when a secret is set, else base. */ +static const char *beacon_sign(beacon_ctx_t *ctx, apr_pool_t *pool, const char *base) +{ + char hex[BEACON_MAC_HEXLEN + 1]; + + if (!ctx->has_secret) { + return base; + } + beacon_mac_hex(ctx, base, strlen(base), hex); + return apr_psprintf(pool, "%s mac=%s", base, hex); +} + +/* + * listener: verify a received, NUL-terminated message. Returns 1 if the MAC matches + * the pre-shared key AND the ts= timestamp is within ctx->max_skew of now. + * Only called when ctx->has_secret. Constant-time MAC compare. On success the + * parsed ts= (microseconds since the epoch) is written to *ts_out for the + * caller's per-url monotonic replay check. reason (out) is set on failure. + */ +static int beacon_verify(beacon_ctx_t *ctx, const char *msg, apr_time_t now, + apr_int64_t *ts_out, const char **reason) +{ + const char *macp = NULL, *p, *tsp; + char expected[BEACON_MAC_HEXLEN + 1]; + apr_size_t prefix_len; + apr_int64_t ts, skew, delta; + + /* Locate the last " mac=" -- the MAC covers everything before it. */ + for (p = msg; (p = strstr(p, " mac=")) != NULL; p += 5) { + macp = p; + } + if (!macp) { + *reason = "no mac"; + return 0; + } + /* mac= must be the final field: exactly BEACON_MAC_HEXLEN chars followed by + * end-of-string. strlen is NUL-safe here -- the caller hands us a + * NUL-terminated apr_pstrmemdup() copy -- so a short or garbled payload + * cannot make the constant-time compare below over-read past the buffer. + * The compare itself validates the hex content; we deliberately do not + * pre-scan for hex digits, which would leak MAC bytes through an early + * exit (timing oracle). */ + if (strlen(macp + 5) != BEACON_MAC_HEXLEN) { + *reason = "bad mac length"; + return 0; + } + + prefix_len = (apr_size_t)(macp - msg); + beacon_mac_hex(ctx, msg, prefix_len, expected); + if (!beacon_const_time_eq(expected, macp + 5, BEACON_MAC_HEXLEN)) { + *reason = "mac mismatch"; + return 0; + } + + /* Anti-replay (1): ts= must be present and within the freshness window. + * ts is in microseconds (raw apr_time_now()) so that announcements made + * less than a second apart still have strictly-increasing timestamps for + * the caller's per-url check. */ + tsp = NULL; + for (p = msg; (p = strstr(p, "ts=")) != NULL && p < macp; p += 3) { + /* token-start: beginning of message or preceded by a space */ + if (p == msg || *(p - 1) == ' ') { + tsp = p; + } + } + if (!tsp) { + *reason = "no ts"; + return 0; + } + ts = apr_atoi64(tsp + 3); + skew = ctx->max_skew > 0 ? ctx->max_skew : BEACON_MAXSKEW_DEFAULT; + delta = (apr_int64_t)now - ts; + if (delta < 0) { + delta = -delta; + } + if (delta > apr_time_from_sec(skew)) { + *reason = "stale timestamp"; + return 0; + } + + *ts_out = ts; + return 1; +} + +/* RUNNING: sender sends a throttled beacon; listener drains its receive socket. */ +static void beacon_cb_running(beacon_ctx_t *ctx, apr_pool_t *pool) +{ + server_rec *s = ctx->s; + apr_status_t rv; + + if (!ctx->opened) { + return; + } + + if (ctx->role == BEACON_ROLE_SEND) { + apr_time_t now = apr_time_now(); + if (now >= ctx->last_publish + ctx->interval) { + const char *host = ctx->s->server_hostname + ? ctx->s->server_hostname : "(unknown)"; + const char *msg; + char *base; + apr_size_t len; + /* Carry the routable URL so the proxy can add us as a member. + * Without ProxyBeaconAdvertise, fall back to the Phase-1 payload + * (channel proof only; the listener just logs it). ts= is the signing + * timestamp in microseconds (Phase 4 anti-replay): the proxy checks + * both a freshness window and that it strictly increases per url, so + * a byte-identical replay (same ts) is rejected. Harmless when + * unsigned. */ + if (ctx->advertise_url) { + base = apr_psprintf(pool, + "BEACON url=%s host=%s pid=%" APR_PID_T_FMT + " seq=%" APR_UINT64_T_FMT " ts=%" APR_INT64_T_FMT, + ctx->advertise_url, host, getpid(), ctx->seq++, + (apr_int64_t)now); + } + else { + base = apr_psprintf(pool, + "BEACON host=%s pid=%" APR_PID_T_FMT + " seq=%" APR_UINT64_T_FMT " ts=%" APR_INT64_T_FMT, + host, getpid(), ctx->seq++, + (apr_int64_t)now); + } + /* Phase 4: append a keyed MAC when a shared secret is configured. */ + msg = beacon_sign(ctx, pool, base); + /* Send strlen(msg) bytes (no trailing NUL on the wire) as a single + * UDP datagram to the proxy. Fire-and-forget; a failure here just + * means this announcement is lost and the next interval retries. */ + len = strlen(msg); + if ((rv = apr_socket_sendto(ctx->sock, ctx->addr, 0, msg, &len)) + != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_WARNING, rv, s, + APLOGNO(10582) "mod_proxy_beacon: sendto %s failed", + ctx->pub_url); + } + ctx->last_publish = now; + } + } + else if (ctx->role == BEACON_ROLE_LISTEN) { + apr_time_t now; + + /* Drain the socket every tick (re-enable latency matters). */ + for (;;) { + char buf[BEACON_MAX_MSG_LEN + 1]; + apr_sockaddr_t from; + apr_size_t sz = BEACON_MAX_MSG_LEN; + apr_time_t msg_now; + const char *msg_str, *safe; + char url[512]; + apr_int64_t msg_ts = 0; + + from.pool = pool; + rv = apr_socket_recvfrom(&from, ctx->sock, 0, buf, &sz); + if (APR_STATUS_IS_EAGAIN(rv)) { + break; /* socket drained */ + } + if (rv != APR_SUCCESS) { + ap_log_error(APLOG_MARK, APLOG_WARNING, rv, s, + APLOGNO(10583) "mod_proxy_beacon: recvfrom failed"); + break; + } + if (sz == 0) { + continue; /* empty datagram */ + } + + /* Capture a single wall-clock snapshot for this message so that + * MAC verification, throttle logging, and last-seen recording all + * share the same time base (no TOCTOU drift across calls). */ + msg_now = apr_time_now(); + + /* recvfrom wrote into our own stack buffer; NUL-terminate it (a UDP + * datagram is delivered whole, so sz is the full message length). */ + buf[sz] = '\0'; + msg_str = buf; + + /* Escape the untrusted payload before it can reach the log: a + * publisher -- or, on an unauthenticated channel, anyone who can + * reach the listen port -- could embed newline/control/ANSI + * sequences to forge or corrupt error-log lines. */ + safe = ap_escape_logitem(pool, msg_str); + + /* Phase 4: when a secret is set, verify the MAC and freshness + * BEFORE logging or acting on the payload, so forged/tampered + * content is dropped (throttled) and never reaches the INFO log. + * msg_ts (microseconds) is the signed timestamp, used below for the + * per-url monotonic replay check. Verification only matters in the + * active (balancer) path; with no balancer we take no action and + * the escaped payload is safe to log for diagnostics. */ + if (ctx->balancer_name && ctx->has_secret) { + const char *reason = "?"; + if (!beacon_verify(ctx, msg_str, msg_now, &msg_ts, &reason)) { + beacon_log_throttled(ctx, msg_now, reason); + continue; + } + } + + ap_log_error(APLOG_MARK, APLOG_INFO, 0, s, + APLOGNO(10584) "mod_proxy_beacon: received: %s", safe); + + if (!ctx->balancer_name) { + continue; + } + + /* Phase 2/3: carries a routable url= -> add the backend (or + * refresh last-seen / re-enable if previously evicted). */ + if (beacon_parse_url(msg_str, url, sizeof(url))) { + beacon_handle_announce(ctx, pool, url, msg_now, msg_ts); + } + } + + /* Phase 3: evict backends that stopped announcing. Throttle the scan + * to ~1s (the recv drain above already runs every tick). */ + now = apr_time_now(); + if (ctx->balancer_name && ctx->evict_timeout > 0 + && now - ctx->last_sweep >= apr_time_from_sec(1)) { + beacon_sweep(ctx, pool, now); + ctx->last_sweep = now; + } + } +} + +/* STOPPING: close the socket if we opened it. */ +static void beacon_cb_stopping(beacon_ctx_t *ctx) +{ + if (ctx->opened) { + apr_socket_close(ctx->sock); + ctx->opened = 0; + ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, ctx->s, + APLOGNO(10585) "mod_proxy_beacon: socket closed"); + } +} + +/* + * Watchdog callback. Always returns APR_SUCCESS: a non-success return would + * unregister the callback and permanently stop the channel, which we never + * want for a transient socket error. + */ +static apr_status_t beacon_watchdog_callback(int state, void *data, + apr_pool_t *pool) +{ + beacon_ctx_t *ctx = (beacon_ctx_t *)data; + + switch (state) { + case AP_WATCHDOG_STATE_STARTING: + beacon_cb_starting(ctx); + break; + case AP_WATCHDOG_STATE_RUNNING: + beacon_cb_running(ctx, pool); + break; + case AP_WATCHDOG_STATE_STOPPING: + beacon_cb_stopping(ctx); + break; + } + + return APR_SUCCESS; +} + +static int beacon_post_config(apr_pool_t *pconf, apr_pool_t *plog, + apr_pool_t *ptemp, server_rec *main_s) +{ + apr_status_t rv; + server_rec *s; + APR_OPTIONAL_FN_TYPE(ap_watchdog_get_instance) *beacon_get_instance; + APR_OPTIONAL_FN_TYPE(ap_watchdog_register_callback) *beacon_register_callback; + + /* post_config runs twice; skip the initial config-test pass. */ + if (ap_state_query(AP_SQ_MAIN_STATE) == AP_SQ_MS_CREATE_PRE_CONFIG) { + return OK; + } + + beacon_get_instance = APR_RETRIEVE_OPTIONAL_FN(ap_watchdog_get_instance); + beacon_register_callback = + APR_RETRIEVE_OPTIONAL_FN(ap_watchdog_register_callback); + if (!beacon_get_instance || !beacon_register_callback) { + ap_log_error(APLOG_MARK, APLOG_CRIT, 0, main_s, + APLOGNO(10586) "mod_proxy_beacon: mod_watchdog is required"); + return !OK; + } + + rv = beacon_get_instance(&beacon_watchdog, BEACON_WATCHDOG_NAME, 0, 1, pconf); + if (rv) { + ap_log_error(APLOG_MARK, APLOG_CRIT, rv, main_s, + APLOGNO(10587) "mod_proxy_beacon: failed to create watchdog instance (%s)", + BEACON_WATCHDOG_NAME); + return !OK; + } + + for (s = main_s; s; s = s->next) { + beacon_ctx_t *ctx = ap_get_module_config(s->module_config, + &proxy_beacon_module); + /* Skip non-canonical server contexts (shared across vhosts). */ + if (s != ctx->s) { + continue; + } + /* Only servers with a directive participate. */ + if (ctx->role == BEACON_ROLE_NONE) { + continue; + } + rv = beacon_register_callback(beacon_watchdog, AP_WD_TM_SLICE, ctx, + beacon_watchdog_callback); + if (rv) { + ap_log_error(APLOG_MARK, APLOG_CRIT, rv, s, + APLOGNO(10588) "mod_proxy_beacon: failed to register watchdog callback"); + return !OK; + } + } + + return OK; +} + +static void beacon_register_hooks(apr_pool_t *p) +{ + static const char *const aszSucc[] = { "mod_watchdog.c", + "mod_proxy_balancer.c", NULL }; + ap_hook_post_config(beacon_post_config, NULL, aszSucc, APR_HOOK_LAST); +} + +AP_DECLARE_MODULE(proxy_beacon) = +{ + STANDARD20_MODULE_STUFF, + NULL, /* create per-dir config */ + NULL, /* merge per-dir config */ + beacon_create_server_config, /* create per-server config */ + NULL, /* merge per-server config */ + beacon_cmds, /* command table */ + beacon_register_hooks /* register hooks */ +}; diff --git a/test/pytest_suite/t/conf/proxy_beacon.conf.in b/test/pytest_suite/t/conf/proxy_beacon.conf.in new file mode 100644 index 0000000000..dc4bea0fc5 --- /dev/null +++ b/test/pytest_suite/t/conf/proxy_beacon.conf.in @@ -0,0 +1,115 @@ +#t/TEST -proxy_beacon +# +# Config for mod_proxy_beacon (UDP backend announcement). +# +# Backends announce themselves to the reverse proxy over UDP datagrams: +# +# - reverse proxy (main server): ProxyBeaconListen -- binds, receives +# - backend (vhost proxy_beacon): ProxyBeaconAddress -- sends to the proxy +# +# Phase 1 proved the channel (the receiver logs each BEACON). Phase 2 adds the +# announcing backend as a member of balancer://beacon and enables it, so a +# request to /beacon on the proxy routes to the dynamically-added backend. +# Phase 3 evicts a backend that stops announcing. Phase 4 authenticates the +# channel with a pre-shared secret (ProxyBeaconSecret): the proxy drops any +# announcement that isn't validly signed and fresh. +# +# Everything runs in this single test instance over loopback UDP, driven by the +# mod_watchdog singleton thread. The backend advertises the MAIN server's +# origin (@SERVERNAME@:@PORT@), which serves the framework's index.html +# ("welcome to :"); routing /beacon/index.html through the +# proxy must return that body, proving the added member serves traffic. +# +# Two backends send to the SAME proxy port: one with the matching secret (must be +# added + serve), and a decoy with a WRONG secret advertising an unroutable URL +# (must be rejected and never added) -- proving authentication works. + + + + + # Received-beacon + balancer add/route logging in the error_log. + LogLevel proxy_beacon:info proxy_balancer:debug + + # Reverse proxy == receiver. ProxyBeaconListen with NO argument inherits the + # server's own address and port (here the main server's @PORT@), so the UDP + # beacon socket shares the server's TCP service port -- UDP and TCP are + # independent port spaces, so this does not collide with the TCP listener. + # Backends therefore beacon to the proxy at its real service endpoint. + ProxyBeaconListen + ProxyBeaconBalancer beacon + + # Phase 4: require announcements to be signed with this shared secret. + # Unsigned / wrong-secret / stale messages are dropped before being acted on. + ProxyBeaconSecret beacon-test-shared-secret + + # Phase 3: disable a backend that hasn't announced in 1s. Combined with the + # backend's 4s announce interval below, this guarantees an eviction gap + # (announce -> stale after ~1s -> evicted -> next announce re-enables). + ProxyBeaconTimeout 1 + + # An initially-empty balancer with spare slots for runtime members. + + ProxySet growth=10 + + ProxyPass /beacon balancer://beacon + + # Legit backend == sender: send to the proxy, advertise its routable origin + # (the main server here), sign with the MATCHING secret, announce every 4s. + + ProxyBeaconAddress 127.0.0.1:@PORT@ + ProxyBeaconAdvertise http://@SERVERNAME@:@PORT@ + ProxyBeaconSecret beacon-test-shared-secret + # Announce every 4s. With the proxy's 1s ProxyBeaconTimeout this leaves + # a ~3s window each cycle where the member is stale and gets evicted, then + # the next announcement re-enables it -- exercising the full Phase-3 + # add -> evict -> re-enable lifecycle in a single-instance test. + ProxyBeaconInterval 4 + + + # Phase 4 negative case: a rogue backend sending to the SAME proxy port with a + # WRONG secret, advertising an unroutable decoy URL. Its announcements must + # be rejected (bad MAC) and the decoy must NEVER become a balancer member. + + ProxyBeaconAddress 127.0.0.1:@PORT@ + ProxyBeaconAdvertise http://127.0.0.1:1 + ProxyBeaconSecret wrong-secret + ProxyBeaconInterval 1 + + + # + # Capacity / slot-exhaustion scenario (separate, second receiver + balancer). + # balancer://cap has room for exactly ONE runtime member (growth=1). Two + # backends announce distinct URLs to it: one wins the slot, the other can + # never be added. The failed add must be retried on a long backoff, NOT on + # every announcement -- so there must be no per-interval add-failure storm. + # + Define BEACON_CAP_PORT @NextAvailablePort@ + + # Second receiver lives on its own vhost so it has a distinct bind port. + + ProxyBeaconListen 127.0.0.1:${BEACON_CAP_PORT} + ProxyBeaconBalancer cap + ProxyBeaconSecret beacon-test-shared-secret + ProxyBeaconTimeout 0 + + ProxySet growth=1 + + ProxyPass /cap balancer://cap + + + # Two backends competing for the single cap slot (announce every 1s). + + ProxyBeaconAddress 127.0.0.1:${BEACON_CAP_PORT} + ProxyBeaconAdvertise http://127.0.0.2:18081 + ProxyBeaconSecret beacon-test-shared-secret + ProxyBeaconInterval 1 + + + ProxyBeaconAddress 127.0.0.1:${BEACON_CAP_PORT} + ProxyBeaconAdvertise http://127.0.0.2:18082 + ProxyBeaconSecret beacon-test-shared-secret + ProxyBeaconInterval 1 + + + + diff --git a/test/pytest_suite/tests/t/modules/test_proxy_beacon.py b/test/pytest_suite/tests/t/modules/test_proxy_beacon.py new file mode 100644 index 0000000000..2fa951c3d6 --- /dev/null +++ b/test/pytest_suite/tests/t/modules/test_proxy_beacon.py @@ -0,0 +1,126 @@ +r"""Test for mod_proxy_beacon -- UDP backend announce -> balancer membership. + +The reverse proxy (main server) binds a UDP socket on a loopback port to receive +announcements; a backend vhost sends BEACON datagrams to it carrying a routable +URL (see t/conf/proxy_beacon.conf.in). Both halves run in the same instance, +driven by the mod_watchdog singleton thread. + + Phase 1: the receiver logs each "BEACON" it receives (channel proof). + Phase 2: on an announcement carrying url=, the proxy adds that backend as a + member of balancer://beacon (reusing mod_proxy_balancer's + balancer_manage) and enables it, so /announce routes to the backend. + Phase 3: when a backend stops announcing for longer than ProxyBeaconTimeout, the + proxy disables it (out of rotation); a later announcement re-enables + it. + Phase 4: announcements are authenticated with a pre-shared secret + (ProxyBeaconSecret); the proxy drops any that aren't validly signed and + fresh. A second backend sends to the same port with the WRONG secret + and a decoy URL -- it must be rejected and never added. + +The config sets ProxyBeaconTimeout=1s on the proxy and ProxyBeaconInterval=4s on the +legit backend, so each cycle is: announce (add/enable) -> stale after ~1s +(evict) -> next announce (re-enable). We assert that whole lifecycle from the +error_log (the reliable signal; cf. t/modules/heartbeat.t's log-scan approach), +prove the member serves traffic while enabled, and assert the wrong-secret +backend is rejected. + +mod_watchdog runs its singleton in a child process, so (like heartbeat.t) this +is skipped under the prefork MPM. +""" + +import time +from pathlib import Path + +import pytest + +from apache_pytest import need_module, t_cmp + +# Backend announces every 4s; cover at least one full add/evict/re-enable cycle. +NB_SECONDS = 11 + + +@need_module("proxy_beacon", "watchdog", "proxy_balancer", "proxy_http") +def test_proxy_beacon(http): + if http.have_module("mpm_prefork"): + pytest.skip("proxy_beacon not run under the prefork MPM") + + error_log = Path(http.vars("t_logs")) / "error_log" + start = error_log.stat().st_size if error_log.exists() else 0 + + # Prove the member serves while enabled: poll /announce right after startup + # (the first announce enables it within ~1 announce interval). + vars_ = http.vars() + expected_body = f"welcome to {vars_['servername']}:{vars_['port']}" + served = False + deadline = time.time() + 6 + while time.time() < deadline: + r = http.GET("/beacon/index.html") + if r.status_code == 200 and expected_body in r.text: + served = True + break + time.sleep(0.5) + assert served, "request through balancer://beacon never reached the backend" + + # Let the add -> evict -> re-enable lifecycle play out. + time.sleep(NB_SECONDS) + + with error_log.open("r", errors="replace") as fh: + fh.seek(start) + loglines = fh.read().splitlines() + + # Announcements are received and carry a routable url=. + received = [ln for ln in loglines if "received: BEACON" in ln] + assert received, "no announcements received by the SUB" + assert any("url=" in ln for ln in received), ( + "announcements should carry a url= token") + + # Phase 2: the backend was added exactly once (dedup), no add-failure spam. + # Qualify by balancer://beacon so the capacity-test balancer (below) doesn't + # perturb these counts. + added = [ln for ln in loglines + if "added backend" in ln and "balancer://beacon" in ln] + assert len(added) == 1, ( + f"backend should be added exactly once; saw {len(added)}: {added}") + assert not [ln for ln in loglines if "failed to add worker" in ln + and "balancer://beacon:" in ln], ( + "unexpected add-failure log lines for balancer://beacon") + + # Phase 3: the member was evicted (stopped announcing within the timeout) + # and later re-enabled (started announcing again). + evicted = [ln for ln in loglines if "evicted backend" in ln] + reenabled = [ln for ln in loglines if "re-enabled backend" in ln] + assert evicted, "backend should have been evicted after the announce gap" + assert reenabled, "backend should have been re-enabled on the next announce" + + # Phase 4: the rogue backend (wrong secret, decoy URL) must be rejected -- + # its announcements dropped and its url NEVER added as a balancer member. + assert not [ln for ln in loglines if "http://127.0.0.1:1" in ln + and "added backend" in ln], ( + "decoy backend with wrong secret must never be added") + assert [ln for ln in loglines + if "dropped" in ln and "mac mismatch" in ln], ( + "proxy should log dropping the wrong-secret announcements") + + # Slot exhaustion: balancer://cap has room for one member but two backends + # announce to it. Exactly one must be added; the other can never fit. + cap_added = [ln for ln in loglines + if "added backend" in ln and "balancer://cap" in ln] + assert len(cap_added) == 1, ( + f"exactly one backend should fit balancer://cap; saw: {cap_added}") + + # The key regression guard: the un-addable backend must NOT trigger an + # add attempt (and its log) on every announcement. Both backends announce + # once per second over ~16s, so a per-interval retry storm would be >10 + # failure logs; the backoff must keep it tiny. + cap_fail = [ln for ln in loglines if "balancer add failed" in ln] + assert len(cap_fail) <= 3, ( + f"failed add must back off, not retry every announcement; " + f"saw {len(cap_fail)} failure logs: {cap_fail}") + + # Anti-replay: the per-url monotonic ts check must NOT false-reject the + # legitimate backends, which announce ~once/second. Because ts is in + # microseconds it strictly increases between announcements, so no genuine + # announcement should be logged as a replay. (A real replay can't be + # injected from this harness; this guards against over-rejection.) + assert not [ln for ln in loglines if "replayed/reordered ts" in ln], ( + "legitimate sub-second announcements must not be seen as replays")