SQLALCHEMY UNIT TESTS
=====================
-**NOTE:** SQLAlchemy as of 0.9.4 now standardizes on `pytest <http://pytest.org/>`_
-for test running! However, the existing support for Nose **still remains**!
-That is, you can now run the tests via pytest or nose. We hope to keep the
-suite nose-compatible indefinitely however this might change at some point.
+Updated for 1.1, 1.2
-SQLAlchemy unit tests by default run using Python's built-in sqlite3
-module. If running on a Python installation that doesn't include this
-module, then pysqlite or compatible must be installed.
+Basic Test Running
+==================
-Unit tests can be run with pytest or nose:
+A test target exists within the setup.py script. For basic test runs::
- py.test: http://pytest.org/
+ python setup.py test
- nose: https://pypi.python.org/pypi/nose/
-The suite includes enhanced support when running with pytest.
+Running with Tox
+================
-SQLAlchemy implements plugins for both pytest and nose that must be
-present when tests are run. In the case of pytest, this plugin is automatically
-used when pytest is run against the SQLAlchemy source tree. However,
-for Nose support, a special test runner script must be used.
+For more elaborate CI-style test running, the tox script provided will
+run against various Python / database targets. For a basic run against
+Python 2.7 using an in-memory SQLite database::
+ tox -e py27-sqlite
-The test suite as also requires the mock library. While
-mock is part of the Python standard library as of 3.3, previous versions
-will need to have it installed, and is available at::
+The tox runner contains a series of target combinations that can run
+against various combinations of databases. The test suite can be
+run against SQLite with "backend" tests also running against a Postgresql
+database::
- https://pypi.python.org/pypi/mock
+ tox -e py36-sqlite-postgresql
-RUNNING TESTS VIA SETUP.PY
---------------------------
-A plain vanilla run of all tests using sqlite can be run via setup.py, and
-requires that pytest is installed::
+Or to run just "backend" tests against a MySQL database::
- $ python setup.py test
+ tox -e py36-mysql-backendonly
+Running against backends other than SQLite requires that a database of that
+vendor be available at a specific URL. See "Setting Up Databases" below
+for details.
-RUNNING ALL TESTS - PYTEST
---------------------------
-To run all tests::
+The py.test Engine
+==================
- $ py.test
+Both the tox runner and the setup.py runner are using py.test to invoke
+the test suite. Within the realm of py.test, SQLAlchemy itself is adding
+a large series of option and customizations to the py.test runner using
+plugin points, to allow for SQLAlchemy's multiple database support,
+database setup/teardown and connectivity, multi process support, as well as
+lots of skip / database selection rules.
-The pytest configuration in setup.cfg will point the runner at the
-test/ directory, where it consumes a conftest.py file that gets everything
-else up and running.
+Running tests with py.test directly grants more immediate control over
+database options and test selection.
+A generic py.test run looks like::
-RUNNING ALL TESTS - NOSE
---------------------------
+ py.test -n4
-When using Nose, a bootstrap script is provided which sets up sys.path
-as well as installs the nose plugin::
+Above, the full test suite will run against SQLite, using four processes.
+If the "-n" flag is not used, the pytest-xdist is skipped and the tests will
+run linearly, which will take a pretty long time.
- $ ./sqla_nose.py
+The py.test command line is more handy for running subsets of tests and to
+quickly allow for custom database connections. Example::
-Assuming all tests pass, this is a very unexciting output. To make it more
-interesting::
+ py.test --dburi=postgresql+psycopg2://scott:tiger@localhost/test test/sql/test_query.py
- $ ./sqla_nose.py -v
+Above will run the tests in the test/sql/test_query.py file (a pretty good
+file for basic "does this database work at all?" to start with) against a
+running Postgresql database at the given URL.
-RUNNING INDIVIDUAL TESTS
----------------------------------
+The py.test frontend can also run tests against multiple kinds of databases
+at once - a large subset of tests are marked as "backend" tests, which will
+be run against each available backend, and additionally lots of tests are targeted
+at specific backends only, which only run if a matching backend is made available.
+For example, to run the test suite against both Postgresql and MySQL at the same time::
-Any directory of test modules can be run at once by specifying the directory
-path, and a specific file can be specified as well::
+ py.test -n4 --db postgresql --db mysql
- $ py.test test/dialect
- $ py.test test/orm/test_mapper.py
+Setting Up Databases
+====================
-When using nose, the setup.cfg currently sets "where" to "test/", so the
-"test/" prefix is omitted::
+The test suite identifies several built-in database tags that run against
+a pre-set URL. These can be seen using --dbs::
- $ ./sqla_nose.py dialect/
-
- $ ./sqla_nose.py orm/test_mapper.py
-
-With Nose, it is often more intuitive to specify tests as module paths::
-
- $ ./sqla_nose.py test.orm.test_mapper
-
-Nose can also specify a test class and optional method using this syntax::
-
- $ ./sqla_nose.py test.orm.test_mapper:MapperTest.test_utils
-
-With pytest, the -k flag is used to limit tests::
-
- $ py.test test/orm/test_mapper.py -k "MapperTest and test_utils"
-
-
-COMMAND LINE OPTIONS
---------------------
-
-SQLAlchemy-specific options are added to both runners, which are viewable
-within the help screen. With pytest, these options are easier to locate
-as they are underneath the "sqlalchemy" grouping::
-
- $ py.test --help
-
- $ ./sqla_nose.py --help
-
-The --help screen is a combination of common nose options and options which
-the SQLAlchemy nose plugin adds. The most commonly SQLAlchemy-specific
-options used are '--db' and '--dburi'.
-
-Both pytest and nose support the same set of SQLAlchemy options, though
-pytest features a bit more capability with them.
-
-
-DATABASE TARGETS
-----------------
-
-Tests will target an in-memory SQLite database by default. To test against
-another database, use the --dburi option with any standard SQLAlchemy URL::
-
- --dburi=postgresql://user:password@localhost/test
-
-If you'll be running the tests frequently, database aliases can save a lot of
-typing. The --dbs option lists the built-in aliases and their matching URLs::
-
- $ py.test --dbs
+ $ py.test --dbs=.
Available --db options (use --dburi to override)
- mysql mysql://scott:tiger@127.0.0.1:3306/test
- oracle oracle://scott:tiger@127.0.0.1:1521
- postgresql postgresql://scott:tiger@127.0.0.1:5432/test
- [...]
-
-To run tests against an aliased database::
-
- $ py.test --db postgresql
-
-This list of database urls is present in the setup.cfg file. The list
-can be modified/extended by adding a file ``test.cfg`` at the
-top level of the SQLAlchemy source distribution which includes
-additional entries::
-
+ default sqlite:///:memory:
+ firebird firebird://sysdba:masterkey@localhost//Users/classic/foo.fdb
+ mssql mssql+pyodbc://scott:tiger@ms_2008
+ mssql_pymssql mssql+pymssql://scott:tiger@ms_2008
+ mysql mysql://scott:tiger@127.0.0.1:3306/test?charset=utf8
+ oracle oracle://scott:tiger@127.0.0.1:1521
+ oracle8 oracle://scott:tiger@127.0.0.1:1521/?use_ansi=0
+ pg8000 postgresql+pg8000://scott:tiger@127.0.0.1:5432/test
+ postgresql postgresql://scott:tiger@127.0.0.1:5432/test
+ postgresql_psycopg2cffi postgresql+psycopg2cffi://scott:tiger@127.0.0.1:5432/test
+ pymysql mysql+pymysql://scott:tiger@127.0.0.1:3306/test?charset=utf8
+ sqlite sqlite:///:memory:
+ sqlite_file sqlite:///querytest.db
+
+What those mean is that if you have a database running that can be accessed
+by the above URL, you can run the test suite against it using ``--db <name>``.
+
+The URLs are present in the ``setup.cfg`` file. You can make your own URLs by
+creating a new file called ``test.cfg`` and adding your own ``[db]`` section::
+
+ # test.cfg file
[db]
- postgresql=postgresql://myuser:mypass@localhost/mydb
+ my_postgresql=postgresql://username:pass@hostname/dbname
-Your custom entries will override the defaults and you'll see them reflected
-in the output of --dbs.
+Above, we can now run the tests with ``my_postgresql``::
-MULTIPLE DATABASE TARGETS
--------------------------
+ py.test --db my_postgresql
-As of SQLAlchemy 0.9.4, the test runner supports **multiple databases at once**.
-This doesn't mean that the entire test suite runs for each database, but
-instead specific test suites may do so, while other tests may choose to
-run on a specific target out of those available. For example, if the tests underneath
-test/dialect/ are run, the majority of these tests are either specific to
-a particular backend, or are marked as "multiple", meaning they will run repeatedly
-for each database in use. If one runs the test suite as follows::
-
- $ py.test test/dialect --db sqlite --db postgresql --db mysql
-
-The tests underneath test/dialect/test_suite.py will be tripled up, running
-as appropriate for each target database, whereas dialect-specific tests
-within test/dialect/mysql, test/dialect/postgresql/ test/dialect/test_sqlite.py
-should run fully with no skips, as each suite has its target database available.
-
-The multiple targets feature is available both under pytest and nose,
-however when running nose, the "multiple runner" feature won't be available;
-instead, the first database target will be used.
-
-When running with multiple targets, tests that don't prefer a specific target
-will be run against the first target specified. Putting sqlite first in
-the list will lead to a much faster suite as the in-memory database is
-extremely fast for setting up and tearing down tables.
+We can also override the existing names in our ``test.cfg`` file, so that we can run
+with the tox runner also::
+ # test.cfg file
+ [db]
+ postgresql=postgresql://username:pass@hostname/dbname
+Now when we run ``tox -e py27-postgresql``, it will use our custom URL instead
+of the fixed one in setup.cfg.
-DATABASE CONFIGURATION
-----------------------
+Database Configuration
+======================
-Use an empty database and a database user with general DBA privileges.
-The test suite will be creating and dropping many tables and other DDL, and
-preexisting tables will interfere with the tests.
+The test runner will by default create and drop tables within the default
+database that's in the database URL, *unless* the multiprocessing option
+is in use via the py.test "-n" flag, which invokes pytest-xdist. The
+multiprocessing option is **enabled by default** for both the tox runner
+and the setup.py frontend. When multiprocessing is used, the SQLAlchemy
+testing framework will create a new database for each process, and then
+tear it down after the test run is complete. So it will be necessary
+for the database user to have access to CREATE DATABASE in order for this
+to work.
Several tests require alternate usernames or schemas to be present, which
are used to test dotted-name access scenarios. On some databases such
test=# create extension hstore;
CREATE EXTENSION
- MYSQL: Default storage engine should be "MyISAM". Tests that require
- "InnoDB" as the engine will specify this explicitly.
+ Full-text search configuration should be set to English, else
+ several tests of ``.match()`` will fail. This can be set (if it isn't so
+ already) with:
- ORACLE: a user named "test_schema" is created.
+ ALTER DATABASE test SET default_text_search_config = 'pg_catalog.english'
+
+ ORACLE: a user named "test_schema" is created in addition to the default
+ user.
The primary database user needs to be able to create and drop tables,
synonyms, and constraints within the "test_schema" user. For this
grant dba to scott;
- SYBASE: Similar to Oracle, "test_schema" is created as a user, and the
- primary test user needs to have the "sa_role".
-
- It's also recommended to turn on "trunc log on chkpt" and to use a
- separate transaction log device - Sybase basically seizes up when
- the transaction log is full otherwise.
-
- A full series of setup assuming sa/master:
-
- disk init name="translog", physname="/opt/sybase/data/translog.dat", size="10M"
- create database sqlalchemy on default log on translog="10M"
- sp_dboption sqlalchemy, "trunc log on chkpt", true
- sp_addlogin scott, "tiger7"
- sp_addlogin test_schema, "tiger7"
- use sqlalchemy
- sp_adduser scott
- sp_adduser test_schema
- grant all to scott
- sp_role "grant", sa_role, scott
-
- Sybase will still freeze for up to a minute when the log becomes
- full. To manually dump the log::
-
- dump tran sqlalchemy with truncate_only
-
MSSQL: Tests that involve multiple connections require Snapshot Isolation
ability implemented on the test database in order to prevent deadlocks that
will occur with record locking isolation. This feature is only available
ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON
- MSSQL+zxJDBC: Trying to run the unit tests on Windows against SQL Server
- requires using a test.cfg configuration file as the cmd.exe shell won't
- properly pass the URL arguments into the nose test runner.
-
- POSTGRESQL: Full-text search configuration should be set to English, else
- several tests of ``.match()`` will fail. This can be set (if it isn't so
- already) with:
-
- ALTER DATABASE test SET default_text_search_config = 'pg_catalog.english'
-
CONFIGURING LOGGING
-------------------
Any log target can be directed to the console with command line options, such
as::
- $ ./sqla_nose.py test.orm.unitofwork --log-info=sqlalchemy.orm.mapper \
+ $ ./py.test test/orm/test_unitofwork.py -s \
--log-debug=sqlalchemy.pool --log-info=sqlalchemy.engine
-This would log mapper configuration, connection pool checkouts, and SQL
-statement execution.
-
+Above we add the py.test "-s" flag so that standard out is not suppressed.
-BUILT-IN COVERAGE REPORTING
-------------------------------
-Coverage is tracked using the coverage plugins built for pytest or nose::
-
- $ py.test test/sql/test_query --cov=sqlalchemy
-
- $ ./sqla_nose.py test.sql.test_query --with-coverage
-
-BIG COVERAGE TIP !!! There is an issue where existing .pyc files may
-store the incorrect filepaths, which will break the coverage system. If
-coverage numbers are coming out as low/zero, try deleting all .pyc files.
DEVELOPING AND TESTING NEW DIALECTS
-----------------------------------
See the file README.dialects.rst for detail on dialects.
-TESTING WITH MULTIPLE PYTHON VERSIONS USING TOX
------------------------------------------------
-
-If you want to test across multiple versions of Python, you may find `tox
-<http://tox.testrun.org/>`_ useful. SQLAlchemy includes a tox.ini file::
-
- tox -e full
-
-SQLAlchemy uses tox mostly for pre-fab testing configurations, to simplify
-configuration of Jenkins jobs, and *not* for testing different Python
-interpreters simultaneously. You can of course create whatever alternate
-tox.ini file you want.
-
-Environments include::
-
- "full" - runs a full py.test
-
- "coverage" - runs a py.test plus coverage, skipping memory/timing
- intensive tests
-
- "pep8" - runs flake8 against the codebase (useful with --diff to check
- against a patch)
-
-
-PARALLEL TESTING
-----------------
-
-Parallel testing is supported using the Pytest xdist plugin. Supported
-databases currently include sqlite, postgresql, and mysql. The username
-for the database should have CREATE DATABASE and DROP DATABASE privileges.
-After installing pytest-xdist, testing is run adding the -n<num> option.
-For example, to run against sqlite, mysql, postgresql with four processes::
-
- tox -e -- -n 4 --db sqlite --db postgresql --db mysql
-
-Each backend has a different scheme for setting up the database. PostgreSQL
-still needs the "test_schema" and "test_schema_2" schemas present, as the
-parallel databases are created using the base database as a "template".
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
