--- /dev/null
+========================
+Developing new Dialects
+========================
+
+.. note::
+
+ When studying this file, it's probably a good idea to also
+ familiarize with the README.unittests.rst file, which discusses
+ SQLAlchemy's usage and extension of the Nose test runner.
+
+While SQLAlchemy includes many dialects within the core distribution, the
+trend for new dialects should be that they are published as external
+projects. SQLAlchemy has since version 0.5 featured a "plugin" system
+which allows external dialects to be integrated into SQLAlchemy using
+standard setuptools entry points. As of version 0.8, this system has
+been enhanced, so that a dialect can also be "plugged in" at runtime.
+
+On the testing side, SQLAlchemy as of 0.8 also includes a "dialect
+compliance suite" that is usable by third party libraries. There is no
+longer a strong need for a new dialect to run through SQLAlchemy's full
+testing suite, as a large portion of these tests do not have
+dialect-sensitive functionality. The "dialect compliance suite" should
+be viewed as the primary target for new dialects, and as it continues
+to grow and mature it should become a more thorough and efficient system
+of testing new dialects.
+
+The file structure of a dialect is typically similar to the following::
+
+ sqlalchemy-<dialect>/
+ setup.py
+ setup.cfg
+ run_tests.py
+ sqlalchemy_<dialect>/
+ __init__.py
+ base.py
+ <dbapi>.py
+ test/
+ __init__.py
+ requirements.py
+ test_suite.py
+ test_<dialect_specific_test>.py
+ ...
+
+An example of this structure can be seen in the Access dialect at
+https://bitbucket.org/zzzeek/sqlalchemy-access/.
+
+Key aspects of this file layout include::
+
+* setup.py - should specify setuptools entrypoints, allowing the
+ dialect to be usable from create_engine(), e.g.:
+
+ entry_points={
+ 'sqlalchemy.dialects': [
+ 'access = sqlalchemy_access.pyodbc:AccessDialect_pyodbc',
+ 'access.pyodbc = sqlalchemy_access.pyodbc:AccessDialect_pyodbc',
+ ]
+ }
+
+Above, the two entrypoints ``access`` and ``access.pyodbc`` allow URLs to be
+used such as::
+
+ create_engine("access://user:pw@dsn")
+
+ create_engine("access+pyodbc://user:pw@dsn")
+
+* setup.cfg - this file contains the traditional contents such as [egg_info]
+ and [nosetests] directives, but also contains new directives that are used
+ by SQLAlchemy's testing framework. E.g. for Access::
+
+ [egg_info]
+ tag_build = dev
+
+ [nosetests]
+ with-sqla_testing = true
+ where = test
+ cover-package = sqlalchemy-access
+ with-coverage = 1
+ cover-erase = 1
+
+ [sqla_testing]
+ requirement_cls=test.requirements:Requirements
+ profile_file=.profiles.txt
+
+ [db]
+ default=access+pyodbc://admin@access_test
+ sqlite=sqlite:///:memory:
+
+ Above, the ``[sqla_testing]`` section contains configuration used by
+ SQLAlchemy's test plugin.The ``[nosetests]`` section includes the
+ directive ``with-sql_testing = true``, which indicates to Nose that
+ the SQLAlchemy nose plugin should be used.
+
+* run_tests.py - The plugin is provided with SQLAlchemy, however is not
+ plugged into Nose automatically; instead, a ``run_tests.py`` script
+ should be composed as a front end to Nose, such that SQLAlchemy's plugin
+ will be correctly installed.
+
+ run_tests.py has two parts. One optional, but probably helpful, step
+ is that it installs your third party dialect into SQLAlchemy without
+ using the setuptools entrypoint system; this allows your dialect to
+ be present without any explicit setup.py step needed. The other
+ step is to import SQLAlchemy's nose runner and invoke it. An
+ example run_tests.py file looks like the following::
+
+ from sqlalchemy.dialects import registry
+
+ registry.register("access", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc")
+ registry.register("access.pyodbc", "sqlalchemy_access.pyodbc", "AccessDialect_pyodbc")
+
+ from sqlalchemy.testing import runner
+
+ runner.main()
+
+ Where above, the ``registry`` module, introduced in SQLAlchemy 0.8, provides
+ an in-Python means of installing the dialect entrypoints without the use
+ of setuptools, using the ``registry.register()`` function in a way that
+ is similar to the ``entry_points`` directive we placed in our ``setup.py``.
+ The call to ``runner.main()`` then runs the Nose front end, which installs
+ SQLAlchemy's testing plugins. Invoking our custom runner looks like the
+ following::
+
+ $ python run_tests.py -v
+
+* requirements.py - The ``requirements.py`` file is where directives
+ regarding database and dialect capabilities are set up.
+ SQLAlchemy's tests are often annotated with decorators that mark
+ tests as "skip" or "fail" for particular backends. Over time, this
+ system has been refined such that specific database and DBAPI names
+ are mentioned less and less, in favor of @requires directives which
+ state a particular capability. The requirement directive is linked
+ to target dialects using a ``Requirements`` subclass. The custom
+ ``Requirements`` subclass is specified in the ``requirements.py`` file
+ and is made available to SQLAlchemy's test runner using the
+ ``requirement_cls`` directive inside the ``[sqla_testing]`` section.
+
+ For a third-party dialect, the custom ``Requirements`` class can
+ usually specify a simple yes/no answer for a particular system. For
+ example, a requirements file that specifies a database that supports
+ the RETURNING construct but does not support reflection of tables
+ might look like this::
+
+ # test/requirements.py
+
+ from sqlalchemy.testing.requirements import SuiteRequirements
+
+ from sqlalchemy.testing import exclusions
+
+ class Requirements(SuiteRequirements):
+ @property
+ def table_reflection(self):
+ return exclusions.closed()
+
+ @property
+ def returning(self):
+ return exclusions.open()
+
+ The ``SuiteRequirements`` class in
+ ``sqlalchemy.testing.requirements`` contains a large number of
+ requirements rules, which attempt to have reasonable defaults. The
+ tests will report on those requirements found as they are run.
+
+* test_suite.py - Finally, the ``test_suite.py`` module represents a
+ Nose test suite, which pulls in the actual SQLAlchemy test suite.
+ To pull in the suite as a whole, it can be imported in one step::
+
+ # test/test_suite.py
+
+ from sqlalchemy.testing.suite import *
+
+ That's all that's needed - the ``sqlalchemy.testing.suite`` package
+ contains an ever expanding series of tests, most of which should be
+ annotated with specific requirement decorators so that they can be
+ fully controlled. To specifically modify some of the tests, they can
+ be imported by name and subclassed::
+
+ from sqlalchemy.testing.suite import *
+
+ from sqlalchemy.testing.suite import ComponentReflectionTest as _ComponentReflectionTest
+
+ class ComponentReflectionTest(_ComponentReflectionTest):
+ @classmethod
+ def define_views(cls, metadata, schema):
+ # bypass the "define_views" section of the
+ # fixture
+ return
+
+
SQLALCHEMY UNIT TESTS
=====================
-SQLAlchemy unit tests by default run using Python's built-in sqlite3
+SQLAlchemy unit tests by default run using Python's built-in sqlite3
module. If running on Python 2.4, pysqlite must be installed.
-Unit tests are run using nose. Nose is available at:
+Unit tests are run using nose. Nose is available at::
http://pypi.python.org/pypi/nose/
This plugin is invoked when the test runner script provided with
SQLAlchemy is used.
-**NOTE:** - the nose plugin is no longer installed by setuptools as of
+**NOTE:** - the nose plugin is no longer installed by setuptools as of
version 0.7 ! Use "python setup.py test" or "./sqla_nose.py".
RUNNING TESTS VIA SETUP.PY
$ python setup.py test
-The -v flag also works here:
+The -v flag also works here::
$ python setup.py test -v
-RUNNING ALL TESTS
+RUNNING ALL TESTS
------------------
-To run all tests:
+To run all tests::
$ ./sqla_nose.py
If you're running the tests on Microsoft Windows, then there is an additional
-argument that must be passed to ./sqla_nose.py:
+argument that must be passed to ./sqla_nose.py::
> ./sqla_nose.py --first-package-wins
sys.modules if it sees a package with the same name in a different location.
Setting this argument disables that behavior.
-Assuming all tests pass, this is a very unexciting output. To make it more
-interesting:
+Assuming all tests pass, this is a very unexciting output. To make it more
+interesting::
$ ./sqla_nose.py -v
RUNNING INDIVIDUAL TESTS
-------------------------
Any directory of test modules can be run at once by specifying the directory
-path:
+path::
$ ./sqla_nose.py test/dialect
-Any test module can be run directly by specifying its module name:
+Any test module can be run directly by specifying its module name::
$ ./sqla_nose.py test.orm.test_mapper
-To run a specific test within the module, specify it as module:ClassName.methodname:
+To run a specific test within the module, specify it as module:ClassName.methodname::
$ ./sqla_nose.py test.orm.test_mapper:MapperTest.test_utils
COMMAND LINE OPTIONS
--------------------
-Help is available via --help:
+Help is available via --help::
$ ./sqla_nose.py --help
-The --help screen is a combination of common nose options and options which
-the SQLAlchemy nose plugin adds. The most commonly SQLAlchemy-specific
+The --help screen is a combination of common nose options and options which
+the SQLAlchemy nose plugin adds. The most commonly SQLAlchemy-specific
options used are '--db' and '--dburi'.
----------------
Tests will target an in-memory SQLite database by default. To test against
-another database, use the --dburi option with any standard SQLAlchemy URL:
+another database, use the --dburi option with any standard SQLAlchemy URL::
--dburi=postgresql://user:password@localhost/test
are used to test dotted-name access scenarios. On some databases such
as Oracle or Sybase, these are usernames, and others such as Postgresql
and MySQL they are schemas. The requirement applies to all backends
-except SQLite and Firebird. The names are:
+except SQLite and Firebird. The names are::
test_schema
test_schema_2 (only used on Postgresql)
-Please refer to your vendor documentation for the proper syntax to create
+Please refer to your vendor documentation for the proper syntax to create
these namespaces - the database user must have permission to create and drop
tables within these schemas. Its perfectly fine to run the test suite
without these namespaces present, it only means that a handful of tests which
expect them to be present will fail.
-Additional steps specific to individual databases are as follows:
+Additional steps specific to individual databases are as follows::
MYSQL: Default storage engine should be "MyISAM". Tests that require
"InnoDB" as the engine will specify this explicitly.
grant dba to scott;
SYBASE: Similar to Oracle, "test_schema" is created as a user, and the
- primary test user needs to have the "sa_role".
-
+ primary test user needs to have the "sa_role".
+
It's also recommended to turn on "trunc log on chkpt" and to use a
- separate transaction log device - Sybase basically seizes up when
+ separate transaction log device - Sybase basically seizes up when
the transaction log is full otherwise.
- A full series of setup assuming sa/master:
+ A full series of setup assuming sa/master:
disk init name="translog", physname="/opt/sybase/data/translog.dat", size="10M"
create database sqlalchemy on default log on translog="10M"
sp_role "grant", sa_role, scott
Sybase will still freeze for up to a minute when the log becomes
- full. To manually dump the log:
+ full. To manually dump the log::
dump tran sqlalchemy with truncate_only
properly pass the URL arguments into the nose test runner.
If you'll be running the tests frequently, database aliases can save a lot of
-typing. The --dbs option lists the built-in aliases and their matching URLs:
+typing. The --dbs option lists the built-in aliases and their matching URLs::
$ ./sqla_nose.py --dbs
Available --db options (use --dburi to override)
postgresql postgresql://scott:tiger@127.0.0.1:5432/test
[...]
-To run tests against an aliased database:
+To run tests against an aliased database::
$ ./sqla_nose.py --db=postgresql
-To customize the URLs with your own users or hostnames, make a simple .ini
-file called `test.cfg` at the top level of the SQLAlchemy source distribution
-or a `.satest.cfg` in your home directory:
+To customize the URLs with your own users or hostnames, create a file
+called `test.cfg` at the top level of the SQLAlchemy source distribution.
+This file is in Python config format, and contains a [db] section which
+lists out additional database configurations::
[db]
postgresql=postgresql://myuser:mypass@localhost/mydb
-------------------
SQLAlchemy logs its activity and debugging through Python's logging package.
Any log target can be directed to the console with command line options, such
-as:
+as::
$ ./sqla_nose.py test.orm.unitofwork --log-info=sqlalchemy.orm.mapper \
--log-debug=sqlalchemy.pool --log-info=sqlalchemy.engine
BUILT-IN COVERAGE REPORTING
------------------------------
-Coverage is tracked using Nose's coverage plugin. See the nose
-documentation for details. Basic usage is:
+Coverage is tracked using Nose's coverage plugin. See the nose
+documentation for details. Basic usage is::
$ ./sqla_nose.py test.sql.test_query --with-coverage
store the incorrect filepaths, which will break the coverage system. If
coverage numbers are coming out as low/zero, try deleting all .pyc files.
-TESTING NEW DIALECTS
---------------------
-You can use the SQLAlchemy test suite to test any new database dialect in
-development. All possible database features will be exercised by default.
-Test decorators are provided that can exclude unsupported tests for a
-particular dialect. You'll see them all over the source, feel free to add
-your dialect to them or apply new decorations to existing tests as required.
-
-It's fine to start out with very broad exclusions, e.g. "2-phase commit is not
-supported on this database" and later refine that as needed "2-phase commit is
-not available until server version 8".
-
-To be considered for inclusion in the SQLAlchemy distribution, a dialect must
-be integrated with the standard test suite. Dialect-specific tests can be
-placed in the 'dialects/' directory. Comprehensive testing of
-database-specific column types and their proper reflection are a very good
-place to start.
-
-When working through the tests, start with 'engine' and 'sql' tests. 'engine'
-performs a wide range of transaction tests that might deadlock on a brand-new
-dialect- try disabling those if you're having problems and revisit them later.
-
-Once the 'sql' tests are passing, the 'orm' tests should pass as well, modulo
-any adjustments needed for SQL features the ORM uses that might not be
-available in your database. But if an 'orm' test requires changes to your
-dialect or the SQLAlchemy core to pass, there's a test missing in 'sql'! Any
-time you can spend boiling down the problem to it's essential sql roots and
-adding a 'sql' test will be much appreciated.
-
-The test suite is very effective at illuminating bugs and inconsistencies in
-an underlying DB-API (or database!) implementation. Workarounds are almost
-always possible. If you hit a wall, join us on the mailing list or, better,
-IRC!
+DEVELOPING AND TESTING NEW DIALECTS
+-----------------------------------
+See the new file README.dialects.rst for detail on dialects.
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
