[thirdparty/kmod.git] / testsuite / README

testsuite

OVERVIEW
========

Kmod's testsuite was designed to automate the process of running tests with
different scenarios, configurations and architectures. The idea is that once we
received a bug report, we reproduce it using the testsuite so we avoid
recurring on the same bug in future.


FEATURES
========

- Isolate each test by running them in separate processes;
- Exec a binary, so we can test the tools and not only the lib API
- Fake accesses to filesystem so we can provide a test rootfs with all the
  configuration, indexes, etc that test needs to be executed.
- Fake calls to init_module(), delete_module() and uname(), so we don't have to
  run tests as root and figure out how to deal with different architectures.

HOW TO ADD A TEST
=================

The simplest way to add a test is to copy and paste one already there. Most of
the options you can have are covered by another tests. This is what you need to
pay attention when writing a test:

1 - Look at testsuite.h, struct test, to see all the options available.

2 - Use TESTSUITE_MAIN and DEFINE_TEST to add new tests. Don't forget to fill
    its description.

3 - If you want testsuite to compare the stdout/stderr of your tests in order
    to check if it worked or not, fill in output.{stderr,stdout} the file with
    the expected output. Bear in mind the same file is used for all
    architectures, so don't print arch-dependent content if you are comparing
    the output.

4 - Fill in the config vector. Setting any of these configuration will make
    testsuite to export LD_PRELOAD with the necessary override libs before
    executing the test. If you are not exec'ing an external binary, you need to
    pass "need_spawn = true" below, otherwise it will not work (LD_PRELOAD is
    only applied when exec'ing a binary). See each config description in
    testsuite.h

5 - need_spawn: if testsuite will exec itself before running a test

6 - expected_fail: if that test is expected to fail, i.e. the return code is
    expected not to be 0.

7 - The rootfs is populated by copying the entire contents of rootfs-pristine
    then running populate-modules.sh to copy generated modules from
    module-playground. Update the latter script to include any modules your
    test need.

8 - Tests can be run individually, outside of 'make check'. strace and gdb work
    too, as long as you tell them to operate on child process.

9 - Make sure test passes when using "default" build flags, i.e. by running
    bootstrap-configure instead of simpler bootstrap/autogen.sh
Commit	Line	Data
23e354bf LDM	1	testsuite
	2
	3	OVERVIEW
	4	========
	5
	6	Kmod's testsuite was designed to automate the process of running tests with
	7	different scenarios, configurations and architectures. The idea is that once we
	8	received a bug report, we reproduce it using the testsuite so we avoid
	9	recurring on the same bug in future.
	10
	11
	12	FEATURES
	13	========
	14
	15	- Isolate each test by running them in separate processes;
	16	- Exec a binary, so we can test the tools and not only the lib API
	17	- Fake accesses to filesystem so we can provide a test rootfs with all the
	18	configuration, indexes, etc that test needs to be executed.
	19	- Fake calls to init_module(), delete_module() and uname(), so we don't have to
	20	run tests as root and figure out how to deal with different architectures.
	21
	22	HOW TO ADD A TEST
	23	=================
	24
	25	The simplest way to add a test is to copy and paste one already there. Most of
	26	the options you can have are covered by another tests. This is what you need to
	27	pay attention when writing a test:
	28
	29	1 - Look at testsuite.h, struct test, to see all the options available.
	30
4083b099 LDM	31	2 - Use TESTSUITE_MAIN and DEFINE_TEST to add new tests. Don't forget to fill
4083b099 LDM	32	its description.
23e354bf LDM	33
23e354bf LDM	34	3 - If you want testsuite to compare the stdout/stderr of your tests in order
4083b099	35	to check if it worked or not, fill in output.{stderr,stdout} the file with
f4f408f9	36	the expected output. Bear in mind the same file is used for all
4083b099 LDM	37	architectures, so don't print arch-dependent content if you are comparing
4083b099 LDM	38	the output.
23e354bf LDM	39
23e354bf LDM	40	4 - Fill in the config vector. Setting any of these configuration will make
4083b099 LDM	41	testsuite to export LD_PRELOAD with the necessary override libs before
	42	executing the test. If you are not exec'ing an external binary, you need to
	43	pass "need_spawn = true" below, otherwise it will not work (LD_PRELOAD is
	44	only applied when exec'ing a binary). See each config description in
	45	testsuite.h
23e354bf LDM	46
	47	5 - need_spawn: if testsuite will exec itself before running a test
	48
	49	6 - expected_fail: if that test is expected to fail, i.e. the return code is
4083b099	50	expected not to be 0.
23e354bf	51
f4f408f9 CMOF	52	7 - The rootfs is populated by copying the entire contents of rootfs-pristine
	53	then running populate-modules.sh to copy generated modules from
	54	module-playground. Update the latter script to include any modules your
	55	test need.
23e354bf LDM	56
23e354bf LDM	57	8 - Tests can be run individually, outside of 'make check'. strace and gdb work
4083b099	58	too, as long as you tell them to operate on child process.
54c43dff LDM	59
54c43dff LDM	60	9 - Make sure test passes when using "default" build flags, i.e. by running
4083b099	61	bootstrap-configure instead of simpler bootstrap/autogen.sh