test/README.testsuite

   1 The extended testsuite only works with UID=0. It consists of the subdirectories
   2 named "test/TEST-??-*", each of which contains a description of an OS image and
   3 a test which consists of systemd units and scripts to execute in this image.
   4 The same image is used for execution under `systemd-nspawn` and `qemu`.
   5
   6 To run the extended testsuite do the following:
   7
   8 $ ninja -C build  # Avoid building anything as root later
   9 $ sudo test/run-integration-tests.sh
  10 ninja: Entering directory `/home/zbyszek/src/systemd/build'
  11 ninja: no work to do.
  12 --x-- Running TEST-01-BASIC --x--
  13 + make -C TEST-01-BASIC clean setup run
  14 make: Entering directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
  15 TEST-01-BASIC CLEANUP: Basic systemd setup
  16 TEST-01-BASIC SETUP: Basic systemd setup
  17 ...
  18 TEST-01-BASIC RUN: Basic systemd setup [OK]
  19 make: Leaving directory '/home/zbyszek/src/systemd/test/TEST-01-BASIC'
  20 --x-- Result of TEST-01-BASIC: 0 --x--
  21 --x-- Running TEST-02-CRYPTSETUP --x--
  22 + make -C TEST-02-CRYPTSETUP clean setup run
  23
  24 If one of the tests fails, then $subdir/test.log contains the log file of
  25 the test.
  26
  27 To run just one of the cases:
  28
  29 $ sudo make -C test/TEST-01-BASIC clean setup run
  30
  31 Specifying the build directory
  32 ==============================
  33
  34 If the build directory is not detected automatically, it can be specified
  35 with BUILD_DIR=:
  36
  37 $ sudo BUILD_DIR=some-other-build/ test/run-integration-tests
  38
  39 or
  40
  41 $ sudo make -C test/TEST-01-BASIC BUILD_DIR=../../some-other-build/ ...
  42
  43 Note that in the second case, the path is relative to the test case directory.
  44 An absolute path may also be used in both cases.
  45
  46 Testing installed binaries instead of built
  47 ===========================================
  48
  49 To run the extended testsuite using the systemd installed on the system instead
  50 of the systemd from a build, use the NO_BUILD=1:
  51
  52 $ sudo NO_BUILD=1 test/run-integration-tests
  53
  54 Configuration variables
  55 =======================
  56
  57 TEST_NO_QEMU=1
  58     Don't run tests under qemu
  59
  60 TEST_QEMU_ONLY=1
  61     Run only tests that require qemu
  62
  63 TEST_NO_NSPAWN=1
  64     Don't run tests under systemd-nspawn
  65
  66 TEST_PREFER_NSPAWN=1
  67     Run all tests that do not require qemu under systemd-nspawn
  68
  69 TEST_NO_KVM=1
  70     Disable qemu KVM auto-detection (may be necessary when you're trying to run the
  71     *vanilla* qemu and have both qemu and qemu-kvm installed)
  72
  73 TEST_NESTED_KVM=1
  74     Allow tests to run with nested KVM. By default, the testsuite disables
  75     nested KVM if the host machine already runs under KVM. Setting this
  76     variable disables such checks
  77
  78 QEMU_MEM=512M
  79     Configure amount of memory for qemu VMs (defaults to 512M)
  80
  81 QEMU_SMP=1
  82     Configure number of CPUs for qemu VMs (defaults to 1)
  83
  84 KERNEL_APPEND='...'
  85     Append additional parameters to the kernel command line
  86
  87 NSPAWN_ARGUMENTS='...'
  88     Specify additional arguments for systemd-nspawn
  89
  90 QEMU_TIMEOUT=infinity
  91     Set a timeout for tests under qemu (defaults to 1800 sec)
  92
  93 NSPAWN_TIMEOUT=infinity
  94     Set a timeout for tests under systemd-nspawn (defaults to 1800 sec)
  95
  96 INTERACTIVE_DEBUG=1
  97     Configure the machine to be more *user-friendly* for interactive debuggung
  98     (e.g. by setting a usable default terminal, suppressing the shutdown after
  99     the test, etc.)
 100
 101 The kernel and initrd can be specified with $KERNEL_BIN and $INITRD. (Fedora's
 102 or Debian's default kernel path and initrd are used by default)
 103
 104 A script will try to find your qemu binary. If you want to specify a different
 105 one with $QEMU_BIN.
 106
 107 Debugging the qemu image
 108 ========================
 109
 110 If you want to log in the testsuite virtual machine, you can specify additional
 111 kernel command line parameter with $KERNEL_APPEND and then log in as root.
 112
 113 $ sudo make -C test/TEST-01-BASIC KERNEL_APPEND="systemd.unit=multi-user.target" run
 114
 115 Root password is empty.
 116
 117 Ubuntu CI
 118 =========
 119
 120 New PR submitted to the project are run through regression tests, and one set
 121 of those is the 'autopkgtest' runs for several different architectures, called
 122 'Ubuntu CI'.  Part of that testing is to run all these tests.  Sometimes these
 123 tests are temporarily deny-listed from running in the 'autopkgtest' tests while
 124 debugging a flaky test; that is done by creating a file in the test directory
 125 named 'deny-list-ubuntu-ci', for example to prevent the TEST-01-BASIC test from
 126 running in the 'autopkgtest' runs, create the file
 127 'TEST-01-BASIC/deny-list-ubuntu-ci'.
 128
 129 The tests may be disabled only for specific archs, by creating a deny-list file
 130 with the arch name at the end, e.g.
 131 'TEST-01-BASIC/deny-list-ubuntu-ci-arm64' to disable the TEST-01-BASIC test
 132 only on test runs for the 'arm64' architecture.
 133
 134 Note the arch naming is not from 'uname -m', it is Debian arch names:
 135 https://wiki.debian.org/ArchitectureSpecificsMemo
 136
 137 For PRs that fix a currently deny-listed test, the PR should include removal
 138 of the deny-list file.
 139
 140 In case a test fails, the full set of artifacts, including the journal of the
 141 failed run, can be downloaded from the artifacts.tar.gz archive which will be
 142 reachable in the same URL parent directory as the logs.gz that gets linked on
 143 the Github CI status.
 144
 145 To add new dependencies or new binaries to the packages used during the tests,
 146 a merge request can be sent to: https://salsa.debian.org/systemd-team/systemd
 147 targeting the 'upstream-ci' branch.
 148
 149 The cloud-side infrastructure, that is hooked into the Github interface, is
 150 located at:
 151
 152 https://git.launchpad.net/autopkgtest-cloud/
 153
 154 In case of infrastructure issues with this CI, things might go wrong in two
 155 places:
 156
 157 - starting a job: this is done via a Github webhook, so check if the HTTP POST
 158   are failing on https://github.com/systemd/systemd/settings/hooks
 159 - running a job: all currently running jobs are listed at
 160   https://autopkgtest.ubuntu.com/running#pkg-systemd-upstream in case the PR
 161   does not show the status for some reason
 162 - reporting the job result: this is done on Canonical's cloud infrastructure,
 163   if jobs are started and running but no status is visible on the PR, then it is
 164   likely that reporting back is not working
 165
 166 For infrastructure help, reaching out to Canonical via the #ubuntu-devel channel
 167 on libera.chat is an effective way to receive support in general.
 168
 169 Manually running a part of the Ubuntu CI test suite
 170 ===================================================
 171
 172 In some situations one may want/need to run one of the tests run by Ubuntu CI
 173 locally for debugging purposes. For this, you need a machine (or a VM) with
 174 the same Ubuntu release as is used by Ubuntu CI (Focal ATTOW).
 175
 176 First of all, clone the Debian systemd repository and sync it with the code of
 177 the PR (set by the $UPSTREAM_PULL_REQUEST env variable) you'd like to debug:
 178
 179 # git clone https://salsa.debian.org/systemd-team/systemd.git
 180 # cd systemd
 181 # git checkout upstream-ci
 182 # TEST_UPSTREAM=1 UPSTREAM_PULL_REQUEST=12345 ./debian/extra/checkout-upstream
 183
 184 Now install necessary build & test dependencies:
 185
 186 ## PPA with some newer Ubuntu packages required by upstream systemd
 187 # add-apt-repository -y ppa:upstream-systemd-ci/systemd-ci
 188 # apt build-dep -y systemd
 189 # apt install -y autopkgtest debhelper genisoimage git qemu-system-x86 \
 190                  libzstd-dev libfdisk-dev libtss2-dev libfido2-dev libssl-dev \
 191                  python3-jinja2 zstd
 192
 193 Build systemd deb packages with debug info:
 194
 195 # DEB_BUILD_OPTIONS="nocheck nostrip" dpkg-buildpackage -us -uc
 196 # cd ..
 197
 198 Prepare a testbed image for autopkgtest (tweak the release as necessary):
 199
 200 # autopkgtest-buildvm-ubuntu-cloud -v -a amd64 -r focal
 201
 202 And finally run the autopkgtest itself:
 203
 204 # autopkgtest -o logs *.deb systemd/ \
 205               --timeout-factor=3 \
 206               --test-name=boot-and-services \
 207               --shell-fail \
 208               -- autopkgtest-virt-qemu autopkgtest-focal-amd64.img
 209
 210 where --test-name= is the name of the test you want to run/debug. The
 211 --shell-fail option will pause the execution in case the test fails and shows
 212 you the information how to connect to the testbed for further debugging.
 213
 214 Manually running CodeQL analysis
 215 =====================================
 216
 217 This is mostly useful for debugging various CodeQL quirks.
 218
 219 Download the CodeQL Bundle from https://github.com/github/codeql-action/releases
 220 and unpack it somewhere. From now the 'tutorial' assumes you have the `codeql`
 221 binary from the unpacked archive in $PATH for brevity.
 222
 223 Switch to the systemd repository if not already:
 224
 225 $ cd <systemd-repo>
 226
 227 Create an initial CodeQL database:
 228
 229 $ CCACHE_DISABLE=1 codeql database create codeqldb --language=cpp -vvv
 230
 231 Disabling ccache is important, otherwise you might see CodeQL complaining:
 232
 233 No source code was seen and extracted to /home/mrc0mmand/repos/@ci-incubator/systemd/codeqldb.
 234 This can occur if the specified build commands failed to compile or process any code.
 235  - Confirm that there is some source code for the specified language in the project.
 236  - For codebases written in Go, JavaScript, TypeScript, and Python, do not specify
 237    an explicit --command.
 238  - For other languages, the --command must specify a "clean" build which compiles
 239    all the source code files without reusing existing build artefacts.
 240
 241 If you want to run all queries systemd uses in CodeQL, run:
 242
 243 $ codeql database analyze codeqldb/ --format csv --output results.csv .github/codeql-custom.qls .github/codeql-queries/*.ql -vvv
 244
 245 Note: this will take a while.
 246
 247 If you're interested in a specific check, the easiest way (without hunting down
 248 the specific CodeQL query file) is to create a custom query suite. For example:
 249
 250 $ cat >test.qls <<EOF
 251 - queries: .
 252   from: codeql/cpp-queries
 253 - include:
 254     id:
 255         - cpp/missing-return
 256 EOF
 257
 258 And then execute it in the same way as above:
 259
 260 $ codeql database analyze codeqldb/ --format csv --output results.csv test.qls -vvv
 261
 262 More about query suites here: https://codeql.github.com/docs/codeql-cli/creating-codeql-query-suites/
 263
 264 The results are then located in the `results.csv` file as a comma separated
 265 values list (obviously), which is the most human-friendly output format the
 266 CodeQL utility provides (so far).