From: Andrew Tridgell <andrew@tridgell.net>
Date: Wed, 3 Jun 2026 23:35:33 +0000 (+1000)
Subject: testsuite: add fleettest.py fleet CI harness
X-Git-Url: http://git.ipfire.org/gitweb/index.cgi?a=commitdiff_plain;h=09656e19c167230f4814c01545d82f4977751b2d;p=thirdparty%2Frsync.git

testsuite: add fleettest.py fleet CI harness

fleettest.py builds the committed HEAD of a checkout on a fleet of remote machines over ssh and runs the test suite under both the stdio-pipe and --use-tcp transports in parallel, reporting only the unexpected results. Each target mirrors a .github/workflows/*.yml job: its configure flags, and the RSYNC_EXPECT_SKIPPED list parsed from the workflow.

The fleet is described by a JSON file (testsuite/fleettest.json, git-ignored); fleettest.json.example is a worked template. Use --fleet to point at another config and --repo to build a tree other than the current directory.

A target with nonroot:true reruns, as the unprivileged ssh user, the tests that declare a module-level fleet_nonroot=True (here ownership-depth and daemon). The set lives in the test files, so new privilege-sensitive tests join the non-root pass with no fleet-config change.

Also rename testsuite/README.testsuite to README.md and rewrite it as markdown documenting the current testsuite: runtests.py, the make check/check29/check30/installcheck/coverage targets, the result/exit-code conventions, and fleettest.py.
---

diff --git a/.gitignore b/.gitignore
index a1f912b77..59a6d09eb 100644
--- a/.gitignore
+++ b/.gitignore
@@ -52,6 +52,7 @@ aclocal.m4
 /testsuite/chown-fake.test
 /testsuite/devices-fake.test
 /testsuite/xattrs-hlink.test
+/testsuite/fleettest.json
 /patches
 /patches.gen
 /build
diff --git a/testsuite/README.md b/testsuite/README.md
new file mode 100644
index 000000000..d3f87273a
--- /dev/null
+++ b/testsuite/README.md
@@ -0,0 +1,160 @@
+# rsync testsuite
+
+This directory holds rsync's automated regression tests. Ideally every code
+change or bug fix comes with a test that would have caught the problem.
+
+The tests are Python scripts named `testsuite/*_test.py`, driven by the
+`runtests.py` harness at the top of the tree (the old shell-based `runtests.sh`
+is gone). Shared helpers live in `testsuite/rsyncfns.py`. A handful of C helper
+programs (`tls`, `getgroups`, `trimslash`, …) are built alongside `rsync` and
+used by some tests. Coverage notes are in [COVERAGE.md](COVERAGE.md).
+
+## Running the tests
+
+### Via make
+
+Run from the build directory:
+
+- **`make check`** — build the helper programs and run the whole suite in
+  parallel (`CHECK_J`, default 8) against the just-built `./rsync`. You do **not**
+  need `make install` first; indeed you generally should not install before
+  testing. Use `make check CHECK_J=1` to run serially.
+- **`make check29`** / **`make check30`** — the same, forcing protocol version 29
+  or 30.
+- **`make installcheck`** — run the suite against the *installed* binary (e.g.
+  `/usr/local/bin/rsync`). Per the GNU standards this does not search `$PATH`.
+  Handy for testing a distribution build.
+- **`make check-progs`** — (re)build just the C helper programs the tests need,
+  without running anything.
+- **`make coverage`** / **`coverage-tcp`** / **`coverage-all`** — generate an HTML
+  coverage report (needs `./configure --enable-coverage` and `gcovr`);
+  `coverage-all` merges runs across protocol versions and the tcp transport.
+
+### Via runtests.py directly
+
+`make check` just drives `runtests.py`; run it directly for finer control. It
+defaults `--rsync-bin` to `./rsync`, so run it from the build directory (or pass
+`--rsync-bin` / `--tooldir`):
+
+```sh
+./runtests.py                 # all tests
+./runtests.py chmod-temp-dir  # a single test by name
+./runtests.py 'xattr*'        # a glob of test names
+```
+
+Useful options:
+
+- `-j N`, `--parallel N` — run up to N tests at once
+- `--use-tcp` — run daemon tests against a real `rsyncd` on `127.0.0.1` (the
+  default runs them over a stdio pipe). **Read the security warning below before
+  using this on a shared machine.**
+- `--protocol VER` — force a protocol version
+- `--preserve-scratch` — keep each test's scratch dir afterwards
+- `--log-level N`, `--always-log` — more verbose output / show logs for passing tests too
+- `--stop-on-fail` — stop after the first failure
+- `--timeout SECS` — per-test timeout (default 300)
+- `--valgrind`, `--valgrind-opts OPTS` — run rsync under valgrind
+- `--rsync-bin PATH`, `--tooldir DIR`, `--srcdir DIR` — locate the binary / build / source dirs
+- `--expect-skipped LIST` — see skip enforcement below
+
+### Security warning: `--use-tcp`
+
+> **⚠️ Do not use `--use-tcp` on a machine with untrusted local users.**
+>
+> `--use-tcp` starts a real `rsync` daemon listening on a loopback TCP port
+> (`127.0.0.1` / `::1`) and **deliberately configures insecure test scenarios**
+> (daemon modules without authentication, unsafe options enabled, etc.). Loopback
+> addresses are reachable by *every* local user, so for as long as the tests run,
+> any other user on the machine can connect to that daemon and exploit those
+> deliberately-insecure modules — potentially reading or writing files with the
+> privileges of the user running the tests (which is **root** if you run the suite
+> as root).
+>
+> Only run `--use-tcp` where there are **no possible local users who might try to
+> exploit it** — a single-user workstation or a dedicated, isolated CI machine.
+> The default stdio-pipe transport carries no such risk: it talks to the daemon
+> over a private pipe with nothing listening on the network, so prefer it on any
+> shared or multi-user host.
+
+### Results and exit codes
+
+Each test prints one result line — `PASS`, `FAIL`, `ERROR`, `SKIP` (with a
+reason), or `XFAIL` (an expected failure) — and the run ends with a
+`passed / failed / skipped` summary. Per-test exit-code convention:
+
+| code | meaning |
+|------|---------|
+| 0    | pass    |
+| 1    | fail    |
+| 2    | error   |
+| 77   | skip    |
+| 78   | xfail   |
+
+`runtests.py` exits non-zero if any test fails. Some tests need root or another
+precondition and otherwise `SKIP` — read the individual test scripts for details.
+
+**Skip enforcement:** on a full run, set `RSYNC_EXPECT_SKIPPED=a,b,c` (or
+`--expect-skipped a,b,c`) and the run fails if the set of skipped tests does not
+match. This is how the CI workflows pin each platform's expected skip set.
+
+### Scratch dirs and debugging
+
+Each test runs in `testtmp/<name>/`. On failure the scratch directory is left in
+place (also `--preserve-scratch`); including its logs in a bug report is helpful.
+
+### Preconditions
+
+You need `python3`, `/bin/sh`, and the normal build toolchain. The ACL/xattr
+tests need the `acl` and `attr` tools (`getfacl`/`setfacl`,
+`getfattr`/`setfattr`) and skip if they are absent. Some tests need root.
+
+These tests also run in CI via GitHub Actions (see `.github/workflows/`).
+
+## Fleet testing (fleettest.py)
+
+`testsuite/fleettest.py` builds the committed HEAD of an rsync checkout on a
+fleet of remote machines over ssh and runs the suite under both transports
+(stdio-pipe and `--use-tcp`) in parallel, reporting only the *unexpected*
+results. It is a fast local pre-flight for the GitHub CI matrix: each target
+mirrors a `.github/workflows/*.yml` job — its configure flags, and the
+`RSYNC_EXPECT_SKIPPED` list parsed straight from the workflow.
+
+Because every run includes a `--use-tcp` pass, the fleet stands up the insecure
+loopback test daemon on each target — so only point it at machines with **no
+untrusted local users** (see the [security warning](#security-warning---use-tcp)
+above).
+
+The fleet — which machines, and how to reach and build on each — is described in
+a JSON file. Copy the bundled example (it is git-ignored) and edit it for your
+hosts:
+
+```sh
+cp testsuite/fleettest.json.example testsuite/fleettest.json   # then edit
+# (or symlink it, or point elsewhere with --fleet PATH)
+```
+
+Each entry names an ssh host (`null` to run locally), the workflow it mirrors,
+and its configure flags, plus optional per-target settings (`make`, `privilege`,
+`env_prefix`, …). See the comments in `fleettest.json.example`.
+
+A target with `"nonroot": true` does an extra pass, after the main (root) run,
+that reruns the privilege-sensitive tests as the unprivileged ssh user. Which
+tests those are is **not** listed in the fleet config — a test opts in by
+setting a module-level `fleet_nonroot = True`, so the set is maintained in the
+test files and new privilege-sensitive tests join automatically with no
+fleet-config change.
+
+Run it from inside a checkout (it builds the current directory's HEAD; use
+`--repo PATH` for another tree):
+
+```sh
+python3 testsuite/fleettest.py                       # whole fleet, both transports
+python3 testsuite/fleettest.py --list                # list configured targets
+python3 testsuite/fleettest.py --targets NAME[,NAME] --clean
+python3 testsuite/fleettest.py --fleet other.json --transport pipe
+```
+
+Each target must be provisioned with the build toolchain its workflow installs
+(autoconf, automake, a C compiler, perl, a python3 markdown module such as
+cmarkgfm or commonmark unless the flags pass `--disable-md2man`, and the dev
+libraries its configure flags enable). A missing piece shows up as `BUILD-FAIL`.
diff --git a/testsuite/README.testsuite b/testsuite/README.testsuite
deleted file mode 100644
index 782cb1ca6..000000000
--- a/testsuite/README.testsuite
+++ /dev/null
@@ -1,28 +0,0 @@
-automatic testsuite for rsync			-*- text -*-
-
-We're trying to develop some more substantial tests to prevent rsync
-regressions.  Ideally, all code changes or bug reports would come with
-an appropriate test suite.
-
-You can run these tests by typing "make check" in the build directory.
-The tests will run using the rsync binary in the build directory, so
-you do not need to do "make install" first.  Indeed, you probably
-should not install rsync before running the tests.
-
-If you instead type "make installcheck" then the suite will test the
-rsync binary from its installed location (e.g. /usr/local/bin/rsync).
-You can use this to test a distribution build, or perhaps to run a new
-test suite against an old version of rsync.  Note that in accordance
-with the GNU Standards, installcheck does not look for rsync on the
-path.
-
-If the tests pass, you should see a report to that effect.  Some tests
-require being root or some other precondition, and so will normally not
-be checked -- look at the test scripts for more information.
-
-If the tests fail, you will see rather more output.  The scratch
-directory will remain in the build directory.  It would be useful if
-you could include the log messages when reporting a failure.
-
-These tests also run automatically on the build farm, and you can see
-the results on http://build.samba.org/.
diff --git a/testsuite/daemon_test.py b/testsuite/daemon_test.py
index 8d9ab4b50..525c7c70a 100644
--- a/testsuite/daemon_test.py
+++ b/testsuite/daemon_test.py
@@ -6,6 +6,10 @@
 # atimes-format variant. We avoid actually starting a listening server
 # by using RSYNC_CONNECT_PROG to spawn the daemon as a child of rsync.
 
+# Rerun under the fleet harness's non-root pass (testsuite/fleettest.py): a
+# non-root rsyncd emits different uid/gid config, so exercise that path too.
+fleet_nonroot = True
+
 import os
 import subprocess
 
diff --git a/testsuite/fleettest.json.example b/testsuite/fleettest.json.example
new file mode 100644
index 000000000..5ecfa7dee
--- /dev/null
+++ b/testsuite/fleettest.json.example
@@ -0,0 +1,100 @@
+{
+  "_comment": [
+    "Example fleet definition for testsuite/fleettest.py -- this is one",
+    "maintainer's setup. Copy (or symlink) this file to testsuite/fleettest.json",
+    "and edit it for your own machines, or point at another file with --fleet PATH.",
+    "fleettest.json is git-ignored; this .example is the committed template.",
+    "",
+    "Each object under \"targets\" maps to fields of the Target dataclass in",
+    "fleettest.py. Required: name, ssh_host (null = run locally), workflow",
+    "(a file under .github/workflows, whose configure flags and RSYNC_EXPECT_SKIPPED",
+    "this target mirrors), configure_flags. Optional (with defaults): make (\"make\"),",
+    "python (\"python3\"), rsync_bin (\"rsync\"; \"rsync.exe\" on Cygwin), privilege",
+    "(\"root\" | \"sudo\" | \"user\"), pipe_jobs/tcp_jobs (8), builddir (\"rsync-citest\",",
+    "relative to the remote $HOME), env_prefix, configure_pre, nonroot.",
+    "",
+    "nonroot: true reruns -- as the non-root ssh user, after the sudo runs -- the",
+    "tests that declare `fleet_nonroot = True` at module level (so the set is",
+    "maintained in the test files, not here). Keys starting with \"_\" are comments.",
+    "See testsuite/README.md."
+  ],
+  "targets": [
+    {
+      "name": "freebsd",
+      "ssh_host": "root@freebsd",
+      "workflow": "freebsd-build.yml",
+      "make": "gmake",
+      "configure_flags": ["--with-rrsync", "--disable-zstd", "--disable-md2man",
+                          "--disable-xxhash", "--disable-lz4"]
+    },
+    {
+      "name": "solaris",
+      "ssh_host": "root@solaris",
+      "workflow": "solaris-build.yml",
+      "make": "gmake",
+      "configure_flags": ["--with-rrsync", "--disable-zstd", "--disable-md2man",
+                          "--disable-xxhash", "--disable-lz4"]
+    },
+    {
+      "name": "openbsd",
+      "ssh_host": "root@openbsd",
+      "workflow": "openbsd-build.yml",
+      "make": "gmake",
+      "configure_pre": "export AUTOCONF_VERSION=2.71 AUTOMAKE_VERSION=1.16;",
+      "tcp_jobs": 2,
+      "configure_flags": ["--with-rrsync", "--disable-zstd", "--disable-md2man",
+                          "--disable-xxhash", "--disable-lz4"]
+    },
+    {
+      "name": "netbsd",
+      "ssh_host": "root@netbsd",
+      "workflow": "netbsd-build.yml",
+      "make": "gmake",
+      "configure_flags": ["--with-rrsync", "--disable-zstd", "--disable-md2man",
+                          "--disable-xxhash", "--disable-lz4"]
+    },
+    {
+      "_comment": "Ubuntu 20.04 older-LTS backport coverage on a root@ box; no 20.04 runner image exists so it mirrors the 22.04 workflow.",
+      "name": "ubuntu-2004",
+      "ssh_host": "root@ubuntu-2004",
+      "workflow": "ubuntu-22.04-build.yml",
+      "configure_flags": ["--with-rrsync"]
+    },
+    {
+      "_comment": "Builds unprivileged (like a CI runner) and runs the suite via sudo; the nonroot pass reruns the privilege-sensitive tests as the ssh user.",
+      "name": "ubuntu-2204",
+      "ssh_host": "runner@ubuntu-2204",
+      "workflow": "ubuntu-22.04-build.yml",
+      "privilege": "sudo",
+      "nonroot": true,
+      "configure_flags": ["--with-rrsync"]
+    },
+    {
+      "name": "ubuntu-2604",
+      "ssh_host": "runner@ubuntu-2604",
+      "workflow": "ubuntu-build.yml",
+      "privilege": "sudo",
+      "nonroot": true,
+      "configure_flags": ["--with-rrsync"]
+    },
+    {
+      "_comment": "macOS: brew is not on the non-interactive ssh PATH, so put it on PATH for the whole build and pass brew include/lib dirs to configure.",
+      "name": "mac2",
+      "ssh_host": "runner@mac2",
+      "workflow": "macos-build.yml",
+      "privilege": "sudo",
+      "env_prefix": "export PATH=/opt/homebrew/bin:/usr/local/bin:$PATH",
+      "configure_pre": "CPPFLAGS=\"-I$(brew --prefix)/include -I$(brew --prefix openssl)/include\"; LDFLAGS=\"-L$(brew --prefix)/lib -L$(brew --prefix openssl)/lib\"; export CPPFLAGS LDFLAGS;",
+      "configure_flags": ["--with-rrsync"]
+    },
+    {
+      "_comment": "Cygwin: non-root plain user (no sudo), binary is rsync.exe.",
+      "name": "cygwin",
+      "ssh_host": "win11",
+      "workflow": "cygwin-build.yml",
+      "rsync_bin": "rsync.exe",
+      "privilege": "user",
+      "configure_flags": ["--with-rrsync"]
+    }
+  ]
+}
diff --git a/testsuite/fleettest.py b/testsuite/fleettest.py
new file mode 100755
index 000000000..9478c1045
--- /dev/null
+++ b/testsuite/fleettest.py
@@ -0,0 +1,648 @@
+#!/usr/bin/env python3
+"""Fleet CI harness for rsync.
+
+Builds the committed HEAD of an rsync checkout on a fleet of remote machines
+(over ssh), runs the test suite under both transports (default stdio-pipe and
+--use-tcp) in parallel, and prints one report of only the UNEXPECTED results --
+a fast local pre-flight for the GitHub CI matrix.
+
+Each target maps 1:1 to a .github/workflows/*.yml job: the per-target configure
+flags mirror that workflow, and the pipe-run RSYNC_EXPECT_SKIPPED list is PARSED
+from the workflow (not hardcoded). The --use-tcp run never sets an expected-skip
+list (matching the workflows), so only test FAILs matter there.
+
+The fleet -- which machines, how to reach and build each -- is read from a JSON
+config: fleettest.json next to this script, or --fleet PATH. Copy the bundled
+fleettest.json.example to fleettest.json (or symlink it) and edit for your own
+hosts; see testsuite/README.md and the comments in fleettest.json.example.
+
+Source = `git archive HEAD` of the rsync tree (the current directory, or --repo
+PATH) -- source-only, no .o/binaries are ever pushed. Build is incremental by
+default (each target's tree is kept in sync; native objects are preserved and
+only changed files rebuild). Use --clean for a from-scratch build (recommended
+on a target's first run).
+
+PROVISIONING: each target must have the build toolchain its workflow's prepare
+step installs -- the target regenerates its own configure/proto.h/man pages, so
+it needs autoconf+automake, perl, a python3 markdown lib (cmarkgfm or commonmark)
+unless its flags pass --disable-md2man, and the dev libraries for whatever its
+configure flags enable (e.g. --with-rrsync needs openssl/xxhash/zstd/lz4 headers).
+A missing piece shows up as BUILD-FAIL with configure's own "you need X" hint.
+
+Per-target "privilege" (set in the JSON) controls how the suite runs: "root"
+(already root -- run directly), "sudo" (build unprivileged, run the suite via
+sudo to match a CI runner), or "user" (run directly as a plain non-root user). A
+target with "nonroot": true additionally reruns -- as the (non-root) ssh user,
+after the sudo runs -- every test that declares `fleet_nonroot = True` at module
+level, so privilege-sensitive tests opt in from the test file itself with no
+fleet-config edit when new ones are added.
+
+Usage (run from inside an rsync checkout, or pass --repo):
+    python3 testsuite/fleettest.py                 # whole fleet, both transports
+    python3 testsuite/fleettest.py --targets cygwin,freebsd
+    python3 testsuite/fleettest.py --transport pipe --clean
+    python3 testsuite/fleettest.py --no-push       # reuse synced trees
+    python3 testsuite/fleettest.py --fleet my-fleet.json --list
+
+Exit 0 iff every selected (target x transport) cell is OK.
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import dataclasses
+import json
+import os
+import re
+import subprocess
+import sys
+import tempfile
+import threading
+import time
+from pathlib import Path
+
+# Set from --repo in main() (default: cwd). The harness builds whatever rsync
+# source tree these point at, so it must be run from inside an rsync checkout
+# or given --repo PATH.
+REPO = Path.cwd()
+WORKFLOWS = REPO / ".github" / "workflows"
+
+# Fleet config: fleettest.json next to this script, overridable with --fleet.
+DEFAULT_CONFIG = Path(__file__).resolve().parent / "fleettest.json"
+EXAMPLE_CONFIG = DEFAULT_CONFIG.with_name(DEFAULT_CONFIG.name + ".example")
+
+# The pushed tree is source-only (git archive). Each target regenerates its own
+# build files, so --delete must NOT prune them: we exclude everything `make`
+# produces (autotools output, proto.h, man pages, config.h/Makefile, *.o, the
+# binaries) plus test artifacts a prior sudo run left root-owned (testtmp,
+# __pycache__, *.pyc -- which a non-root --delete can't unlink). Excluded paths
+# are protected from --delete, so each target keeps its native build state for
+# incremental rebuilds. `configure` itself is committed, so it is NOT excluded.
+PUSH_EXCLUDES = [
+    ".git", "config.h", "config.status", "config.log", "Makefile", "shconfig",
+    "configure.sh", "config.h.in", "aclocal.m4", "proto.h", "git-version.h",
+    "/rsync.1", "/rsync-ssl.1", "/rsyncd.conf.5", "/rrsync.1",
+    "*.o", "*.exe", "__pycache__", "*.pyc", "/testtmp",
+    "/rsync", "/tls", "/getgroups", "/getfsdev", "/trimslash", "/wildtest",
+    "/testrun", "/simdtest", "/t_unsafe", "/t_chmod_secure", "/t_rename_secure",
+    "/t_symlink_secure", "/t_secure_relpath",
+]
+
+
+@dataclasses.dataclass
+class Target:
+    name: str
+    ssh_host: str | None          # null in JSON => run locally
+    workflow: str                 # filename under .github/workflows
+    configure_flags: list[str]
+    make: str = "make"            # e.g. "gmake" on the BSDs/Solaris
+    env_prefix: str = ""          # exported before configure AND make (e.g. PATH)
+    configure_pre: str = ""       # shell run before ./configure (env exports, brew)
+    python: str = "python3"
+    rsync_bin: str = "rsync"      # "rsync.exe" on Cygwin
+    privilege: str = "root"       # "root" (already root) | "sudo" | "user" (plain, no sudo)
+    pipe_jobs: int = 8
+    tcp_jobs: int = 8
+    builddir: str = "rsync-citest"   # relative to remote $HOME; absolute for local
+    # When true, after the sudo runs, additionally run -- as the (non-root) ssh
+    # user -- every test that declares `fleet_nonroot = True` (see
+    # discover_nonroot_tests). Mirrors a workflow's non-root check step.
+    nonroot: bool = False
+
+
+def load_fleet(path: Path) -> list[Target]:
+    """Load the fleet from a JSON file of the shape {"targets": [ {...}, ... ]}.
+
+    Each entry's keys are Target fields; keys starting with "_" are treated as
+    comments and ignored (both at top level and per target). Validation errors
+    name the offending target so a typo is easy to find."""
+    try:
+        data = json.loads(path.read_text())
+    except OSError as e:
+        sys.exit(f"cannot read fleet config {path}: {e}")
+    except json.JSONDecodeError as e:
+        sys.exit(f"invalid JSON in {path}: {e}")
+    if not isinstance(data, dict) or not isinstance(data.get("targets"), list):
+        sys.exit(f'{path}: expected a JSON object with a "targets" array')
+    fields = {f.name for f in dataclasses.fields(Target)}
+    fleet: list[Target] = []
+    for i, entry in enumerate(data["targets"]):
+        if not isinstance(entry, dict):
+            sys.exit(f"{path}: targets[{i}] is not an object")
+        entry = {k: v for k, v in entry.items() if not k.startswith("_")}
+        who = entry.get("name", f"targets[{i}]")
+        bad = set(entry) - fields
+        if bad:
+            sys.exit(f"{path}: target {who!r} has unknown key(s): "
+                     f"{', '.join(sorted(bad))}")
+        try:
+            fleet.append(Target(**entry))
+        except TypeError as e:
+            sys.exit(f"{path}: target {who!r}: {e}")
+    if not fleet:
+        sys.exit(f"{path}: no targets defined")
+    return fleet
+
+
+# ---------------------------------------------------------------------------
+# command execution (ssh for remote, local shell when ssh_host is null)
+# ---------------------------------------------------------------------------
+
+
+@dataclasses.dataclass
+class CmdResult:
+    rc: int
+    out: str          # combined stdout + stderr
+    timed_out: bool = False
+
+
+def run_on(target: Target, script: str, timeout: int) -> CmdResult:
+    """Run a /bin/sh script on the target. Remote via ssh, else local."""
+    if target.ssh_host:
+        argv = ["ssh", "-o", "BatchMode=yes", "-o", "ConnectTimeout=15",
+                target.ssh_host, script]
+    else:
+        argv = ["/bin/sh", "-c", script]
+    try:
+        p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout)
+        return CmdResult(p.returncode, (p.stdout or "") + (p.stderr or ""))
+    except subprocess.TimeoutExpired as e:
+        out = (e.stdout or b"") + (e.stderr or b"")
+        if isinstance(out, bytes):
+            out = out.decode(errors="replace")
+        return CmdResult(124, out, timed_out=True)
+    except FileNotFoundError as e:
+        return CmdResult(127, str(e))
+
+
+def push_argv(target: Target, staging: str, clean: bool) -> list[str]:
+    # -rlpgoD = -a without -t: do NOT preserve mtimes. The host clock can be
+    # hours AHEAD of a target, so preserved (commit-time) mtimes land "in the
+    # future" there and rsync's `Makefile: Makefile.in config.status` rule
+    # triggers a config.status/autoconf regeneration storm. Letting files take
+    # the target's own clock avoids that. --checksum keeps the transfer
+    # incremental despite the unstable mtimes (decide by content, not size+time).
+    args = ["rsync", "-rlpgoD", "--checksum", "--delete"]
+    for ex in PUSH_EXCLUDES:
+        args.append(f"--exclude={ex}")
+    dst = f"{target.ssh_host}:{target.builddir}/" if target.ssh_host \
+        else f"{target.builddir}/"
+    args += [f"{staging}/", dst]
+    return args
+
+
+# ---------------------------------------------------------------------------
+# workflow skip-list parsing
+# ---------------------------------------------------------------------------
+
+# The trailing '? tolerates a `bash -c '... make check'` wrapper (e.g. Cygwin).
+_SKIP_RE = re.compile(r"RSYNC_EXPECT_SKIPPED=(\S+)\s+make\s+check'?\s*$", re.M)
+
+
+def parse_workflow_skip(workflow: str) -> str | None:
+    """Return the literal RSYNC_EXPECT_SKIPPED csv for the `make check` step, or
+    None if the workflow leaves it unset."""
+    path = WORKFLOWS / workflow
+    try:
+        text = path.read_text()
+    except OSError:
+        return None
+    m = _SKIP_RE.search(text)
+    return m.group(1) if m else None
+
+
+# ---------------------------------------------------------------------------
+# non-root test discovery
+# ---------------------------------------------------------------------------
+
+# A test opts into the fleet's extra non-root pass by setting a module-level
+# `fleet_nonroot = True`. We read it with a text scan rather than importing the
+# module (test files execute their body on import), so a new privilege-sensitive
+# test joins the pass just by carrying the marker -- no fleet-config edit needed.
+_NONROOT_RE = re.compile(r"^[ \t]*fleet_nonroot[ \t]*=[ \t]*True\b", re.M)
+
+
+def discover_nonroot_tests(testsuite_dir: Path) -> list[str]:
+    """Return the names (without the _test.py suffix) of the tests under
+    testsuite_dir that declare `fleet_nonroot = True`."""
+    names = []
+    for p in sorted(testsuite_dir.glob("*_test.py")):
+        try:
+            if _NONROOT_RE.search(p.read_text(errors="replace")):
+                names.append(p.name[: -len("_test.py")])
+        except OSError:
+            continue
+    return names
+
+
+# ---------------------------------------------------------------------------
+# remote script builders
+# ---------------------------------------------------------------------------
+
+
+def build_script(t: Target) -> str:
+    flags = " ".join(t.configure_flags)
+    # configure only when not yet configured (keeps incremental builds fast);
+    # --clean wipes the builddir beforehand so Makefile is absent -> reconfigure.
+    pre = f'{t.env_prefix}\n' if t.env_prefix else ''
+    return (
+        f'cd {t.builddir} || exit 3\n'
+        f'{pre}'
+        f'if [ ! -f Makefile ]; then {t.configure_pre} ./configure {flags} || exit 4; fi\n'
+        f'{t.make} -j{t.pipe_jobs} check-progs || exit 5\n'
+    )
+
+
+def test_script(t: Target, transport: str, skip_csv: str | None, jobs: int) -> str:
+    rb = f'--rsync-bin="$PWD/{t.rsync_bin}"'
+    tcp = " --use-tcp" if transport == "tcp" else ""
+    # PYTHONDONTWRITEBYTECODE: don't drop root-owned __pycache__/*.pyc into the
+    # tree (a sudo run would, breaking the next non-root push --delete).
+    env = "PYTHONDONTWRITEBYTECODE=1 "
+    if skip_csv:
+        env += f"RSYNC_EXPECT_SKIPPED={skip_csv} "
+    runtests = f'{t.python} runtests.py {rb}{tcp} -j {jobs}'
+    # env_prefix (e.g. a brew PATH) must reach the test too: some tests build a
+    # helper binary on the fly (a test may invoke `make`, which needs gawk etc.),
+    # so the build tools must be on PATH at test time.
+    pre = f'{t.env_prefix}; ' if t.env_prefix else ''
+    if t.privilege == "sudo":
+        # -n: never prompt (capture_output has no TTY -- a prompt would hang
+        # the whole timeout). Targets need passwordless sudo or a fresh
+        # `sudo -v`. env keeps the vars (and PATH) across the sudo boundary.
+        path_pass = 'PATH="$PATH" ' if t.env_prefix else ''
+        cmd = f"{pre}sudo -n env {path_pass}{env}{runtests}"
+    else:
+        cmd = pre + env + runtests
+    return f'cd {t.builddir} || exit 3\n{cmd}\n'
+
+
+def nonroot_test_script(t: Target, names: list[str]) -> str:
+    """Run the given tests as the (non-root) ssh user -- the fleet analogue of a
+    workflow's non-root check step. Explicit test names make runtests.py
+    full_run False, so no RSYNC_EXPECT_SKIPPED is involved; only FAILs matter.
+    The prior sudo pipe/tcp runs left testtmp root-owned, so clear it (via sudo)
+    before the non-root run recreates it."""
+    pre = f'{t.env_prefix}; ' if t.env_prefix else ''
+    runtests = (f'PYTHONDONTWRITEBYTECODE=1 {t.python} runtests.py '
+                f'--rsync-bin="$PWD/{t.rsync_bin}" {" ".join(names)}')
+    return (f'cd {t.builddir} || exit 3\n'
+            f'sudo -n rm -rf testtmp\n'
+            f'{pre}{runtests}\n')
+
+
+# ---------------------------------------------------------------------------
+# runtests.py output parsing
+# ---------------------------------------------------------------------------
+
+RE_RESULT = re.compile(r"^(PASS|FAIL|ERROR|XFAIL|SKIP)\s+(\S+)", re.M)
+RE_COUNT = re.compile(r"^\s+(\d+)\s+(passed|failed|xfailed|skipped)\b", re.M)
+RE_SKIP_HDR = re.compile(r"^----- skipped results:", re.M)
+RE_SKIP_EXP = re.compile(r"^\s+expected:\s*(.*)$", re.M)
+RE_SKIP_GOT = re.compile(r"^\s+got:\s*(.*)$", re.M)
+
+
+def _csv_set(s: str) -> set[str]:
+    return {x for x in s.strip().split(",") if x}
+
+
+@dataclasses.dataclass
+class TransportResult:
+    transport: str
+    exit_code: int
+    timed_out: bool
+    counts: dict[str, int]
+    failed: list[str]
+    skip_checked: bool
+    skip_expected: set[str]
+    skip_got: set[str]
+    raw: str
+
+    @property
+    def skip_mismatch(self) -> bool:
+        return self.skip_checked and self.skip_expected != self.skip_got
+
+    @property
+    def ok(self) -> bool:
+        return (not self.timed_out and self.exit_code == 0
+                and not self.failed and not self.skip_mismatch)
+
+
+def parse_transport(transport: str, r: CmdResult, skip_checked: bool) -> TransportResult:
+    counts = {"passed": 0, "failed": 0, "xfailed": 0, "skipped": 0}
+    for m in RE_COUNT.finditer(r.out):
+        counts[m.group(2)] = int(m.group(1))
+    failed = [m.group(2) for m in RE_RESULT.finditer(r.out)
+              if m.group(1) in ("FAIL", "ERROR")]
+    exp = got = set()
+    if skip_checked and RE_SKIP_HDR.search(r.out):
+        em = RE_SKIP_EXP.search(r.out)
+        gm = RE_SKIP_GOT.search(r.out)
+        exp = _csv_set(em.group(1)) if em else set()
+        got = _csv_set(gm.group(1)) if gm else set()
+    return TransportResult(transport, r.rc, r.timed_out, counts, failed,
+                           skip_checked, exp, got, r.out)
+
+
+@dataclasses.dataclass
+class TargetResult:
+    target: str
+    reachable: bool = True
+    pushed: bool = True
+    build_ok: bool = True
+    error: str = ""
+    build_log: str = ""
+    transports: dict[str, TransportResult] = dataclasses.field(default_factory=dict)
+
+
+# ---------------------------------------------------------------------------
+# per-target worker
+# ---------------------------------------------------------------------------
+
+_print_lock = threading.Lock()
+
+
+def log(msg: str) -> None:
+    with _print_lock:
+        print(msg, flush=True)
+
+
+def run_target(t: Target, args, staging: str) -> TargetResult:
+    res = TargetResult(t.name)
+    log(f"[{t.name}] start")
+
+    if t.ssh_host:
+        ping = run_on(t, "echo ok", timeout=25)
+        if ping.rc != 0:
+            res.reachable = False
+            res.error = f"ssh unreachable (rc={ping.rc}): {ping.out.strip()[:200]}"
+            log(f"[{t.name}] UNREACHABLE")
+            return res
+
+    if not args.no_push:
+        if args.clean:
+            bd = t.builddir
+            if bd and bd not in ("/", "~", os.path.expanduser("~")):
+                run_on(t, f'rm -rf {bd}', timeout=120)
+        push = subprocess.run(push_argv(t, staging, args.clean),
+                              capture_output=True, text=True, timeout=600)
+        if push.returncode != 0:
+            res.pushed = False
+            res.error = f"push failed (rc={push.returncode}): {push.stderr.strip()[:300]}"
+            log(f"[{t.name}] PUSH-FAIL")
+            return res
+
+    b = run_on(t, build_script(t), timeout=1200)
+    res.build_ok = b.rc == 0
+    res.build_log = b.out
+    if not res.build_ok:
+        log(f"[{t.name}] BUILD-FAIL")
+        return res
+
+    for transport in args.transports:
+        skip_csv = parse_workflow_skip(t.workflow) if transport == "pipe" else None
+        jobs = (args.jobs if args.jobs else
+                (t.tcp_jobs if transport == "tcp" else t.pipe_jobs))
+        cmd = test_script(t, transport, skip_csv, jobs)
+        r = run_on(t, cmd, timeout=2400)
+        res.transports[transport] = parse_transport(transport, r, skip_csv is not None)
+        log(f"[{t.name}] {transport} done "
+            f"({'ok' if res.transports[transport].ok else 'ISSUE'})")
+
+    # Extra non-root pass (after the sudo runs) for targets that opt in, running
+    # the tests that declare `fleet_nonroot = True` (discovered in main()).
+    if t.nonroot and args.nonroot_tests:
+        r = run_on(t, nonroot_test_script(t, args.nonroot_tests), timeout=2400)
+        res.transports["nonroot"] = parse_transport("nonroot", r, skip_checked=False)
+        log(f"[{t.name}] nonroot done "
+            f"({'ok' if res.transports['nonroot'].ok else 'ISSUE'})")
+    return res
+
+
+# ---------------------------------------------------------------------------
+# reporting
+# ---------------------------------------------------------------------------
+
+
+def cell_status(res: TargetResult, transport: str) -> str:
+    if not res.reachable:
+        return "UNREACHABLE"
+    if not res.pushed:
+        return "PUSH-FAIL"
+    if not res.build_ok:
+        return "BUILD-FAIL"
+    tr = res.transports.get(transport)
+    if tr is None:
+        return "-"
+    if tr.timed_out:
+        return "TIMEOUT"
+    if tr.failed:
+        return f"FAIL({len(tr.failed)})"
+    if tr.skip_mismatch:
+        return "SKIP-MISMATCH"
+    if tr.exit_code != 0:
+        return f"EXIT({tr.exit_code})"
+    return "OK"
+
+
+def print_report(results: list[TargetResult], args, fleet: list[Target]) -> bool:
+    by_name = {t.name: t for t in fleet}
+    order = {t.name: i for i, t in enumerate(fleet)}
+    results.sort(key=lambda r: order.get(r.target, 99))
+    # The 'nonroot' column appears only when some target ran a non-root pass;
+    # targets without one show "-" there (a neutral N/A, not a failure).
+    transports = list(args.transports)
+    if any("nonroot" in r.transports for r in results):
+        transports.append("nonroot")
+    ts = time.strftime("%Y-%m-%d %H:%M")
+    print("\n" + "=" * 64)
+    print(f"rsync fleet CI — branch {current_branch()} — {ts}")
+    print(f"source: HEAD   build: {'clean' if args.clean else 'incremental'}   "
+          f"transports: {','.join(args.transports)}")
+    print("(A target's pipe skip-set is only enforced when its workflow sets "
+          "RSYNC_EXPECT_SKIPPED; otherwise only FAILs matter. The 'nonroot' "
+          "column runs the privilege-sensitive tests as the unprivileged user; "
+          "'-' = N/A.)")
+    print("=" * 64)
+
+    width = max(len(t) for t in order) + 2
+    header = "TARGET".ljust(width) + "".join(tr.upper().ljust(16) for tr in transports)
+    print(header)
+    all_ok = True
+    for res in results:
+        row = res.target.ljust(width)
+        for transport in transports:
+            st = cell_status(res, transport)
+            if st not in ("OK", "-"):    # "-" = N/A (e.g. no nonroot pass)
+                all_ok = False
+            row += st.ljust(16)
+        # data-driven row notes: local target, or a target with a distinct tcp -j
+        t = by_name.get(res.target)
+        notes = []
+        if t is not None:
+            if t.ssh_host is None:
+                notes.append("(local)")
+            if "tcp" in transports and t.tcp_jobs != t.pipe_jobs:
+                notes.append(f"(tcp -j{t.tcp_jobs})")
+        print(row + " ".join(notes))
+
+    # detail section: only the unexpected cells
+    details: list[str] = []
+    for res in results:
+        if not res.reachable:
+            details.append(f"{res.target} — UNREACHABLE: {res.error}")
+            continue
+        if not res.pushed:
+            details.append(f"{res.target} — PUSH-FAIL: {res.error}")
+            continue
+        if not res.build_ok:
+            tail = "\n    ".join(res.build_log.strip().splitlines()[-20:])
+            details.append(f"{res.target} — BUILD-FAIL:\n    {tail}")
+            continue
+        for transport in transports:
+            tr = res.transports.get(transport)
+            if tr is None or tr.ok:
+                continue
+            if tr.timed_out:
+                details.append(f"{res.target} / {transport} — TIMEOUT")
+            if tr.failed:
+                details.append(f"{res.target} / {transport} — {len(tr.failed)} failed:\n    "
+                               + " ".join(tr.failed))
+            if tr.skip_mismatch:
+                extra = tr.skip_got - tr.skip_expected
+                missing = tr.skip_expected - tr.skip_got
+                diff = []
+                if extra:
+                    diff.append(f"unexpected skips: {','.join(sorted(extra))}")
+                if missing:
+                    diff.append(f"expected-but-ran: {','.join(sorted(missing))}")
+                details.append(f"{res.target} / {transport} — skip mismatch ("
+                               + "; ".join(diff) + ")\n"
+                               f"    expected: {','.join(sorted(tr.skip_expected))}\n"
+                               f"    got:      {','.join(sorted(tr.skip_got))}")
+            elif not tr.failed and not tr.timed_out and tr.exit_code != 0:
+                details.append(f"{res.target} / {transport} — runtests exit {tr.exit_code}")
+
+    # Exclude N/A ("-") cells (e.g. the nonroot column for targets that don't
+    # run a non-root pass) from the OK/not-OK tally.
+    statuses = [cell_status(res, transport)
+                for res in results for transport in transports]
+    cells = sum(1 for s in statuses if s != "-")
+    ok_cells = sum(1 for s in statuses if s == "OK")
+    print("=" * 64)
+    if details:
+        print("==== UNEXPECTED RESULTS ====")
+        for d in details:
+            print(d)
+        print("=" * 64)
+    print(f"{len(results)} targets x {len(transports)} transports = {cells} cells: "
+          f"{ok_cells} OK, {cells - ok_cells} not OK")
+    return all_ok
+
+
+def current_branch() -> str:
+    try:
+        return subprocess.run(["git", "-C", str(REPO), "rev-parse",
+                               "--abbrev-ref", "HEAD"],
+                              capture_output=True, text=True).stdout.strip() or "?"
+    except Exception:
+        return "?"
+
+
+# ---------------------------------------------------------------------------
+# main
+# ---------------------------------------------------------------------------
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Fleet CI harness for rsync.")
+    ap.add_argument("--targets", help="comma-separated subset (default: all)")
+    ap.add_argument("--transport", choices=["pipe", "tcp", "both"], default="both")
+    ap.add_argument("--no-push", action="store_true",
+                    help="reuse the already-synced tree on each target")
+    ap.add_argument("--clean", action="store_true",
+                    help="wipe each builddir and reconfigure (recommended first run)")
+    ap.add_argument("--jobs", type=int, help="override -j for both transports")
+    ap.add_argument("--repo", help="rsync source tree to build (default: cwd)")
+    ap.add_argument("--fleet", help="fleet config JSON "
+                    "(default: fleettest.json next to this script)")
+    ap.add_argument("--list", action="store_true", help="list targets and exit")
+    args = ap.parse_args()
+
+    global REPO, WORKFLOWS
+    REPO = Path(args.repo).resolve() if args.repo else Path.cwd()
+    WORKFLOWS = REPO / ".github" / "workflows"
+    if not (REPO / "runtests.py").is_file():
+        print(f"{REPO} is not an rsync source tree (no runtests.py); "
+              f"run from inside a checkout or pass --repo", file=sys.stderr)
+        return 2
+
+    config_path = Path(args.fleet).resolve() if args.fleet else DEFAULT_CONFIG
+    if not config_path.exists():
+        print(f"no fleet config at {config_path}\n"
+              f"copy {EXAMPLE_CONFIG} to {DEFAULT_CONFIG} (or pass --fleet PATH)",
+              file=sys.stderr)
+        return 2
+    fleet = load_fleet(config_path)
+
+    if args.list:
+        for t in fleet:
+            host = t.ssh_host or "(local)"
+            skip = parse_workflow_skip(t.workflow)
+            print(f"{t.name:12} {host:18} {t.make:6} "
+                  f"pipe-skip={'set' if skip else 'unset'}")
+        return 0
+
+    args.transports = ["pipe", "tcp"] if args.transport == "both" else [args.transport]
+
+    chosen = fleet
+    if args.targets:
+        want = [s.strip() for s in args.targets.split(",") if s.strip()]
+        by_name = {t.name: t for t in fleet}
+        bad = [w for w in want if w not in by_name]
+        if bad:
+            print(f"unknown target(s): {', '.join(bad)}", file=sys.stderr)
+            print(f"known: {', '.join(by_name)}", file=sys.stderr)
+            return 2
+        chosen = [by_name[w] for w in want]
+
+    # Stage committed HEAD (source-only). Each target regenerates its own
+    # build files with its own toolchain -- exactly like the CI jobs, which
+    # install autotools / python-markdown / dev-libs in their prepare step.
+    # (Pushing locally-generated files instead fights rsync's Makefile
+    # maintainer rules: a target with a different autoconf version sees
+    # "configure.sh has CHANGED" and errors.) So each target must be
+    # provisioned like its workflow -- see the module docstring.
+    staging = tempfile.mkdtemp(prefix="rsync-fleettest-stage.")
+    try:
+        ar = subprocess.run(f"git -C {REPO} archive HEAD | tar -x -C {staging}",
+                            shell=True, capture_output=True, text=True)
+        if ar.returncode != 0:
+            print(f"git archive failed: {ar.stderr}", file=sys.stderr)
+            return 2
+
+        # Tests that opt into the non-root pass (same for every target).
+        args.nonroot_tests = discover_nonroot_tests(Path(staging) / "testsuite")
+
+        results: list[TargetResult] = []
+        with concurrent.futures.ThreadPoolExecutor(max_workers=len(chosen)) as ex:
+            futs = {ex.submit(run_target, t, args, staging): t for t in chosen}
+            for fut in concurrent.futures.as_completed(futs):
+                t = futs[fut]
+                try:
+                    results.append(fut.result())
+                except Exception as e:  # never let one target kill the run
+                    r = TargetResult(t.name)
+                    r.reachable = False
+                    r.error = f"harness exception: {e!r}"
+                    results.append(r)
+    finally:
+        subprocess.run(["rm", "-rf", staging])
+
+    all_ok = print_report(results, args, fleet)
+    return 0 if all_ok else 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/testsuite/ownership-depth_test.py b/testsuite/ownership-depth_test.py
index 6facae257..9a5d5ed80 100644
--- a/testsuite/ownership-depth_test.py
+++ b/testsuite/ownership-depth_test.py
@@ -7,6 +7,10 @@ covered too. As a normal user we can still remap the group to a secondary group
 we belong to; the uid side then needs root and is skipped.
 """
 
+# Rerun under the fleet harness's non-root pass (testsuite/fleettest.py): the uid
+# remap only runs as root, so a non-root run exercises the group-only path too.
+fleet_nonroot = True
+
 import os
 
 from rsyncfns import (