Merge branch 'logger-wrong-typo' of https://github.com/chentooerl/util-linux

[thirdparty/util-linux.git] / Documentation / howto-tests.txt

 util-linux regression tests
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~

 It's expected that for each invasive change or important bugfix you will
 include a test to your patch.

 Compile binaries, libs, extra test programs and run the basic tests:

	$ make check

 Compile extra test programs only:

	$ make check-programs

 Note that the configure option --disable-static disables many of libmount and
 libblkid unit tests.

 Run all tests including tests that require root permissions:

	# cd tests
	# ./run.sh [options, see --help]

	Alternatively using sudo and make:

	$ make check-programs
	$ sudo -E make check TS_OPTS="--parallel=1"

 note that as root you have to manually remove output and diff directories

	# rm -rf output diff

 or run 'make clean' as root.


 Run subset of tests:

	$ cd tests
	$ ./run.sh <test_directory-name>

        for example:

              $ ./run.sh blkid
              $ ./run.sh libmount

        or individual test script, for example:

	      $ ./ts/cal/year

 The tests is possible to exclude by ./run.sh --exclude=<list>  where the
 <list> is blank separated test names in format "testdir/testname", for example:

	$ ./run.sh --exclude="mount/move"

 The --exclude is evaluated by the ./run.sh script only. See below

	TS_OPT_testdir_[testscript_]fake=

 environment variable which provides more powerful functionality to skip tests.


 *** WARNING for root users ***

 The tests touch your /etc/fstab, initialize loop devices or scsi_debug devices
 if executed with root permissions.

 Please, be careful and use these tests only for development and never on
 production system.

fuzz targets
------------

The fuzz targets can be built and run along with the other tests (after installing
clang):

	$ ./tools/config-gen fuzz
	$ make check

environment variables
---------------------

TS_COMMAND

    Evaluated by "make check" to override the default command (run.sh).
    Example:
      - build all test dependencies, but skip the actual test
        $ make check TS_COMMAND="true"

TS_OPTS

    Evaluated by "make check" to pass options.to run.sh (see ./run.sh --help).
    Examples:
      - run utmp tests only
        $ make check TS_OPTS="--parallel=1 utmp"

TS_OPT_testdir_[testscript_]fake="<yes|no>"

    Evaluated by any test script to skip certain tests.
    Examples:
      - skip all the tests within "fdisk" test-directory:
        $ make check TS_OPT_fdisk_fake="yes"

      - skip only "fdisk/bsd" test:
        $ make check TS_OPT_fdisk_bsd_fake="yes"

      - skip all "fdisk" tests except fdisk/bsd:
        $ make check TS_OPT_fdisk_fake="yes" TS_OPT_fdisk_bsd_fake="no"

TS_OPT_testdir_[testscript_]known_fail="<yes|no>"

    Similar usage like TS_OPT_*_fake above. "known_fail" means that the given
    test will run but (negative) results will be ignored. The build log and test
    diffs will still remind you about the issue.

TS_OPT_testdir_[testscript_]verbose="<yes|no>"

    Set verbosity for certain tests. Similar usage like TS_OPT_*_fake above.

TS_OPT_testdir_[testscript_]memcheck="<yes|no>"

    Run certain tests with valgrind. Similar usage like TS_OPT_*_fake above.


External services
-----------------

Coveralls coverage reports

    URL: https://coveralls.io/github/util-linux/util-linux

lgtm CI - automatically executed security code analysis

    URL: https://lgtm.com/projects/g/util-linux/util-linux/

Coverity Scan

    URL: https://scan.coverity.com/projects/karelzak-util-linux

Fossies codespell report

    URL: https://fossies.org/linux/test/util-linux-master.tar.gz/codespell.html

OSS-Fuzz

    URL: https://google.github.io/oss-fuzz/
    URL: https://oss-fuzz-build-logs.storage.googleapis.com/index.html#util-linux
    URL: https://oss-fuzz.com/coverage-report/job/libfuzzer_asan_util-linux/latest

CIFuzz

    URL: https://google.github.io/oss-fuzz/getting-started/continuous-integration/
    URL: https://github.com/util-linux/util-linux/actions?query=workflow%3ACIFuzz