From: Stefan Eissing Date: Mon, 8 Nov 2021 14:40:44 +0000 (+0000) Subject: * test HTTP/2: also run core tests and worker mpm X-Git-Tag: 2.5.0-alpha2-ci-test-only~703 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=9c9fb22508d50b781928b4d2c47033da40859858;p=thirdparty%2Fapache%2Fhttpd.git * test HTTP/2: also run core tests and worker mpm * test/README.pytest on how to use pytest suite and add test cases. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894835 13f79535-47bb-0310-9956-ffa450edef68 --- diff --git a/.travis.yml b/.travis.yml index 0d65dc85a74..a8d23e976e7 100644 --- a/.travis.yml +++ b/.travis.yml @@ -266,12 +266,12 @@ jobs: CLEAR_CACHE=1 # ------------------------------------------------------------------------- - if: *condition_not_24x - name: Linux Ubuntu, event MPM, HTTP/2 test suite + name: Linux Ubuntu, MPMs [event, worker], core + HTTP/2 test suite dist: focal env: APR_VERSION=1.7.0 APU_VERSION=1.6.1 APU_CONFIG="--with-crypto" CONFIG="--enable-mods-shared=reallyall --with-mpm=event --enable-mpms-shared=event" - NO_TEST_FRAMEWORK=1 TEST_INSTALL=1 TEST_H2=1 + NO_TEST_FRAMEWORK=1 TEST_INSTALL=1 TEST_H2=1 TEST_CORE=1 addons: apt: sources: diff --git a/test/README.pytest b/test/README.pytest new file mode 100644 index 00000000000..01fd2890582 --- /dev/null +++ b/test/README.pytest @@ -0,0 +1,127 @@ +Apache httpd pytest suite +========================= +Using pytest () and a Python >= 3.8 +for a more flexible testing of Apache httpd. + +Usage +----- +In your httpd source checkout, do: + +> make install +> pytest + +and all tests defined run on the installed executable and modules. +> pytest test/modules/core +runs all the core tests. You can run tests also by name selection: +> pytest -k test_core +runs all test cases starting with 'test_core'. Similar: +> pytest -k test_core_001_04 +runs the test cases starting with that. + +Test output gets more verbose, by adding one or several '-v'. This +also raises the error log level of the tested modules. +> pytest -vvv -k test_h2_004_01 +run the specific test with mod_http2 at log level TRACE2. + +By default, test cases will configure httpd with mpm_event. You +can change that with the invocation: +> MPM=worker pytest test/modules/http2 + +Some tests rely on additional tools installed in your environment +and will 'skip' if those are not present. In a non-verbose run, +these will appear as 's' in the output. If you run pytest more +verbose, the skipped test cases will mention a reason for why +they are disabled. + +For example, most tests in test/modules/md require an installation +of 'pebble', an ACME test server, and look for it in $PATH. + + +Workings +-------- +All tests start httpd on their own and try hard to shut it down +afterwards. You can abort tests with ^C and they clean up. + +httpd is started/stopped repeatedly in testing as configurations +for individual test cases are changed. This is a main difference to +the Perl test framework which starts the server with all possible +combinations configured that are needed by tests. + +In test/gen/apache a server root is created with config, logs and htdocs +directories. test/gen/apache/logs/error_log will be the log. +Configs start in test/gen/apache/conf/httpd.conf. modules.conf is +dynamically created for the list of modules that a test suite needs. + +Test cases write their specific setup in test.conf and reload/restart +the httpd process. This makes for a small configuration in a test case. + + +Development +----------- + +Adding a test in an existing file is done by adding a method. Its name +must start with 'test_' and the common practise is to have the name +of the test suite there as well. All http2 tests start with 'test_h2_'. + +Following this can be any characters. If you make test cases of a +certain feature with a common prefix, it is easier to invoke just +them using the '-k' selector on the command line. + +You can also add just a new file to a test suite, if you do a new +range of test cases that do not fit anywhere else. A very simple +one is found in test/modules/http2/test_001_httpd_alive.py. + +There is a python class defined with 2 methods. One is the test +method itself and the other one ' is + + @pytest.fixture(autouse=True, scope='class') + def _class_scope(self, env): + code + +is marked as a pytest 'fixture'. This is some pytest magic. +'autouse=True' means that this fixture is run, even though +no test case uses it directly. scope='class' means that it +is run once for all test cases in this class. + +As you see, this fixture gets a parameter named 'env' and +that is the name of another pytest fixture, defined in the +file 'conftest.py' in the same directory. + + @pytest.fixture(scope="package") + def env(pytestconfig) -> H2TestEnv: + code + +This one runs one time per 'package', meaning for all test +cases in this directory. And it gets the 'pytestconfig' +as parameter which is a standard pytest fixture. + +So, when you run 'pytest -k test_h2_004', pytest will look +at _all_ test cases defined and collect those with that +prefix. For each directory with test cases found, it will +process the 'conftest.py', boot-strapping the 'env' fixture, +and the process the files with active test cases. + +As a result, if you invoke just a single test case, only +the fixtures needed for that test case are created. This +gives good turn around times when debugging a test case. + +If you want to add a new test suite, create a new directory. +Add the files '__init__.py', 'conftest.py' and a first +'test_suitename_something.py'. test/modules/core is the +simplest example. 'test/modules/http2' shows how you load +other modules. 'test/modules/md' checks and starts external +processes (an ACME test server). + + +Infrastructure +-------------- +The test cases rely on the classes provided in 'test/pyhttpd' +for common code in managing a httpd test instance and do +provide some base setups: +- a small set of virtual hosts with some files +- access to paths and definitions (env fixture) +- start/stop httpd and inspect the error log +- run clients like curl and nghttp +- create certificates for your hosts and make curl use + the root certs (so no --insecure calls by curl). + diff --git a/test/travis_run_linux.sh b/test/travis_run_linux.sh index 9a7a3aec7c3..58a249d4b07 100755 --- a/test/travis_run_linux.sh +++ b/test/travis_run_linux.sh @@ -159,10 +159,20 @@ if ! test -v SKIP_TESTING; then grep -v ':\(debug\|trace[12345678]\)\]' test/perl-framework/t/logs/error_log fi + if test -v TEST_CORE -a $RV -eq 0; then + # Run HTTP/2 tests. + MPM=event py.test-3 test/modules/core + RV=$? + fi + if test -v TEST_H2 -a $RV -eq 0; then # Run HTTP/2 tests. - py.test-3 test/modules/http2 + MPM=event py.test-3 test/modules/http2 RV=$? + if test $RV -eq 0; then + MPM=worker py.test-3 test/modules/http2 + RV=$? + fi fi if test -v TEST_MD -a $RV -eq 0; then