Cross-version testing with named configurations

In #3381 (and #3385), we committed a backward-incompatible change to
BIND 9.19.5, 9.18.7, and 9.16.33, explicitly requiring "inline-signing"
for every "dnssec-policy".

We did this backward-incompatible change deliberately, knowing the
consequences for users and their configurations. But if we didn't, say,
we were unaware this is a backward-incompatible change and fixed failing
systems test by "tweaking a knob to make the CI pass", we would not have
a second look before the change hits user configurations.

"cross-version-config-tests" CI job is such a second look. It will run
system tests from the latest release tag specific to the particular
branch (e.g., v9.19.12 for the "main" branch) with BIND 9 binaries from
the current "HEAD" (the future v9.19.13). This Frankenstein build gets
conceived by altering the "TOP_BUILDDIR" variable in
"bin/tests/system/conf.sh".

Caveats:
- Only system test configurations are tested; no actual test code is
  run.
- Problems with namedN.conf configurations are not identified.

When backward-incompatible change is introduced, the CI job is expected
to fail. If the change is deliberate, the job will keep failing until
the version with the backward-incompatible change is tagged, and the
minor version in configure.ac is bumped.

(cherry picked from commit cc54211baa0a5d42ce45d471af8ef11f6b851b07)

diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 8fded88571b677bce07d0e29bac177c4a38830f3..eba4fafe0a8cd59c79c4df23cd63b7e30242a820 100644 (file)
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -347,6 +347,52 @@ stages:
     - export SLOT=$(sh -x bin/tests/prepare-softhsm2.sh)
     - test -n "${SLOT}" && test "${SLOT}" -gt 0
 
+cross-version-config-tests:
+  stage: system
+  <<: *base_image
+  <<: *default_triggering_rules
+  variables:
+    CC: gcc
+    CFLAGS: "${CFLAGS_COMMON}"
+    # Disable option checking to prevent problems with new default options in
+    # the &configure anchor.
+    EXTRA_CONFIGURE: "--disable-option-checking"
+  script:
+    # Exclude the dyndb test from the system test as the sample library can't
+    # locate the libdns library from the BIND 9 baseline version.
+    - sed -i '/^dyndb \\$/d' bin/tests/system/conf.sh.common
+    - *configure
+    - *setup_interfaces
+    - make -j${BUILD_PARALLEL_JOBS:-1}
+    - export BIND_BRANCH=16
+    # When testing a .0 release, compare it against the previous development
+    # release (e.g., 9.19.0 and 9.18.0 should both be compared against 9.17.22).
+    - if [ "$(sed -n -E "s|^m4_define\(\[bind_VERSION_PATCH\], ([0-9]+)\)dnl$|\1|p" configure.ac)" = "0" ]; then export BIND_BRANCH=$((BIND_BRANCH - 1 - (BIND_BRANCH % 2))); fi
+    - BASELINE="$(curl -s "https://gitlab.isc.org/api/v4/projects/1/repository/tags?search=^v9.${BIND_BRANCH}&order_by=version" | jq -r ".[0].name")"
+    - git clone --branch "${BASELINE}" --depth 1 https://gitlab.isc.org/isc-projects/bind9.git "bind-${BASELINE}"
+    - cd "bind-${BASELINE}"
+    - autoreconf -fi
+    - *configure
+    - make -j${BUILD_PARALLEL_JOBS:-1}
+    - cd bin/tests/system
+    # Neutralize shell and pytests; in effect, "nsX" servers are just started
+    # and stopped, thus configuration checked.
+    - truncate --size=0 */tests{.sh,*.py}
+    # Run the setup phase of all system tests in the most recently tagged BIND 9
+    # release using the binaries built for the current BIND 9 version.  This
+    # intends to detect obvious backward compatibility issues with the latter.
+    - sed -i -E "s|(export TOP)=.*|\1=${CI_PROJECT_DIR}|" conf.sh
+    - make -j${TEST_PARALLEL_JOBS:-1} -k check V=1
+  needs:
+    - job: autoreconf
+      artifacts: true
+  artifacts:
+    paths:
+      - bind-*
+    untracked: true
+    expire_in: "1 day"
+    when: on_failure
+
 .system_test_common: &system_test_common
   <<: *default_triggering_rules
   stage: system