From: Iker Pedrosa Date: Tue, 26 Aug 2025 08:48:15 +0000 (+0200) Subject: doc/contributions/tests.md: add Python system tests X-Git-Tag: 4.19.0-rc1~61 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=899e7048f294c5ef9062c518057fb4b0c161f536;p=thirdparty%2Fshadow.git doc/contributions/tests.md: add Python system tests Document the new Python system tests: - Benefits - Contribution guidance - How to setup the testing environment - Test configuration and execution - Advanced testing features - Development patterns - Debugging information - Troubleshooting & FAQs Signed-off-by: Iker Pedrosa --- diff --git a/doc/contributions/tests.md b/doc/contributions/tests.md index efe89baef..4ecf2b683 100644 --- a/doc/contributions/tests.md +++ b/doc/contributions/tests.md @@ -1,6 +1,8 @@ # Tests -Currently, shadow provides unit and system tests. +Currently, shadow provides unit tests and system tests. +System tests are split in two: +Bash system tests and Python system tests. ## Unit tests @@ -20,15 +22,243 @@ make check ## System tests -These type of tests are written in shell. Unfortunately, the testing framework -is tightly coupled to the Ubuntu distribution and it can only be run in this -distribution. Besides, if anything fails during the execution the system can -be left in an unstable state. Taking that into account you shouldn't run this -workflow in your host machine, we recommend to use a disposable system like a -VM or a container instead. +System tests verify the behavior of shadow utilities +in a complete system environment. +There are two types of system tests available. -You can execute system tests by running: +### Bash system tests + +These tests are written in shell +and are tightly coupled to the Ubuntu distribution. +They can only be run on Ubuntu, +and if anything fails during execution, +the system can be left in an unstable state. +You shouldn't run this workflow on your host machine—we recommend using a disposable system +like a VM or a container instead. + +You can execute Bash system tests by running: ``` cd tests && ./run_all ``` + +### Python system tests + +The new system tests use Python, +[pytest](https://pytest.org/) +and [pytest-mh](https://pytest-mh.readthedocs.io/). +The Python framework provides several advantages over Bash tests: + +• **Cross-distribution compatibility**: works on all distributions in CI + (Alpine, Debian, Fedora and openSUSE) + unlike Bash tests which only run on Ubuntu. +• **Proper environment management**: ensures clean setup and teardown + with environment restoration even when tests fail. +• **Improved test maintainability**: provides a rich, high-level API + that reduces complex test code + and makes tests easier to understand. +• **Automated artifact collection**: automatically collects logs and artifacts + when tests finish (even if they fail). +• **Flexible infrastructure**: supports both VMs and containers for testing + with local execution capabilities. +• **Future extensibility**: enables potential filtering by ticket + or importance and other advanced features. + +[Read more about the benefits and rationale](https://github.com/shadow-maint/shadow/issues/835). + +**For all new contributions, +we recommend using the Python system test framework.** + +#### Contribution guidance + +The framework is under active development +and welcomes contributions for missing features. +When contributing: + +1. **Framework improvements**: enhance the infrastructure in `tests/system/framework/`. +2. **Proposing new tests**: place them in `tests/system/tests/`. + Review existing examples in `tests/system/tests/` + to understand the established patterns and conventions. +3. **Missing features**: if you're comfortable implementing missing pytest-mh features, + contributions are encouraged. + +#### Running tests + +##### Environment setup + +A testing environment is required to run the system test framework. +While the same system can be used, +a disposable environment (container or VM) is recommended. + +The default configuration relies on a pre-built container named `builder` +that must be available in your container environment. +This container contains: + +- A complete shadow-utils installation +- All required system dependencies + +**Setup steps:** + +1. **Build the container environment**: follow the instructions in [build_install.md](build_install.md) + to create the required `builder` container. +2. **Verify container availability**: ensure the `builder` container is accessible via Docker. + +The default `mhc.yaml` configuration expects this container +to be available as `builder`. + +To ensure dependency isolation and system protection, +run the tests within a Python virtual environment: + +```bash +cd tests/system +python -m venv .venv +source .venv/bin/activate +pip install -r ./requirements.txt +``` + +##### Configuration + +The default configuration located in `tests/system` should be enough +if you are using containers. +If you run another environment you'll need to tune `tests/system/mhc.yaml`. +Instructions are available in the +[pytest-mh mhc.yaml documentation](https://pytest-mh.readthedocs.io/en/latest/articles/mhc-yaml.html). + +##### Execution + +Run all Python system tests: + +```bash +pytest --mh-config=mhc.yaml --mh-lazy-ssh -v +``` + +Run tests in a file: + +```bash +pytest --mh-config=mhc.yaml --mh-lazy-ssh -v tests/test_useradd.py +``` + +Run a single test case: + +```bash +pytest --mh-config=mhc.yaml --mh-lazy-ssh -v -k test_useradd__add_user +``` + +**Command options explained:** +- `--mh-config=mhc.yaml`: Specifies the multihost configuration file +- `--mh-lazy-ssh`: Enables lazy SSH connections for better performance +- `-v`: Verbose output showing individual test results +- `-k`: Filters tests by name pattern + +For additional running options, +see the [pytest-mh running tests documentation](https://pytest-mh.readthedocs.io/en/latest/articles/running-tests.html). + +#### Advanced testing features + +The Python testing framework includes several advanced features +for comprehensive testing: + +**Test markers:** +- `@pytest.mark.topology(KnownTopology.Shadow)`: specifies the test topology. +- `@pytest.mark.builtwith("gshadow")`: skips tests when specific features are unavailable. +- `@pytest.mark.parametrize()`: extensive parameterized testing + for edge cases and different inputs. + +**Rich command parsing:** +All shadow command outputs are parsed into structured Python objects +using the `jc` library: +- `PasswdEntry`, `ShadowEntry`, `GroupEntry`, `GShadowEntry` for file entries +- `ID` command parsing with group membership verification +- Structured error handling and validation + +#### Test development patterns + +When writing new Python tests, +follow these established patterns: + +**Basic test structure:** +```python +@pytest.mark.topology(KnownTopology.Shadow) +def test_command__specific_behavior(shadow: Shadow): + """ + :title: Descriptive test title + :setup: + 1. Setup steps + :steps: + 1. Test steps + :expectedresults: + 1. Expected outcomes + :customerscenario: False + """ + # Arrange + shadow.useradd("testuser") + + # Act + result = shadow.tools.getent.passwd("testuser") + + # Assert + assert result is not None, "User should exist" + assert result.name == "testuser", "Username should match" +``` + +**Distribution-specific testing:** +```python +if "Debian" in shadow.host.distro_name: + assert result.shell == "/bin/sh" +else: + assert result.shell == "/bin/bash" +``` + +**Parameterized testing:** +```python +@pytest.mark.parametrize( + "input_date, expected_epoch", + [ + ("1970-01-01", 0), + ("2025-01-01", 20089), + ("0", 0), + ], +) +def test_date_handling(shadow: Shadow, input_date: str, expected_epoch: int): + # Test implementation +``` + +#### Debugging information + +**Test artifacts:** +- **Location**: `tests/system/artifacts/` directory. +- **Content**: automatically collected logs and system state information. +- **Collection**: happens automatically when tests finish, + regardless of success or failure. + +**System state management:** +The framework automatically manages shadow system files: +- **Backup**: complete backup of `/etc/passwd`, `/etc/shadow`, `/etc/group`, `/etc/gshadow`, `/home` + before tests. +- **Restore**: automatic restoration after each test. +- **Verification**: file mismatch detection + to ensure unmodified files remain unchanged. + +**Common debugging steps:** +1. **Check container status**: ensure the `builder` container is running and accessible. +2. **Verify artifacts**: examine collected logs in the `artifacts/` directory. +3. **Manual testing**: access the container directly + to reproduce issues manually. +4. **File state**: check if unexpected file modifications caused test failures. + +#### Troubleshooting & FAQs + +**Common pitfalls:** +- **Missing virtual environment**: always activate the Python virtual environment + before running tests. +- **Mismatched environment**: ensure your `mhc.yaml` configuration matches your testing environment. +- **Container vs host confusion**: remember that tests run inside containers by default; + adjust expectations accordingly. + +**Tips for disposable environments:** +- Use containers or VMs to avoid system state pollution. +- Ensure proper cleanup in the framework between test runs. +- Verify container/VM has necessary privileges for user/group operations. + +For additional information about the testing framework, +see the [pytest-mh documentation](https://pytest-mh.readthedocs.io).