From: Mike Bayer Date: Tue, 6 Nov 2018 20:13:03 +0000 (-0500) Subject: Improve documentation re: Session.binds and partitioning strategies X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=4c75dead12bf7b3c9340426ac48ed56effb9ac85;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Improve documentation re: Session.binds and partitioning strategies Update documentation to include background on arbitrary superclass usage, add full cross-linking between all related methods and parameters. De-emphasize "twophase" and document that it is not well-supported in drivers. Change-Id: Id99894bb62cc506e896c9aa7c256e9f6e602243e (cherry picked from commit 1f13c8c833ebd96c09d1499b2504aa4950dca709) --- diff --git a/doc/build/orm/extensions/horizontal_shard.rst b/doc/build/orm/extensions/horizontal_shard.rst index deb97a74ce..69faf9bb33 100644 --- a/doc/build/orm/extensions/horizontal_shard.rst +++ b/doc/build/orm/extensions/horizontal_shard.rst @@ -1,3 +1,5 @@ +.. _horizontal_sharding_toplevel: + Horizontal Sharding =================== diff --git a/doc/build/orm/persistence_techniques.rst b/doc/build/orm/persistence_techniques.rst index ccbe82c02c..7c1db2dd96 100644 --- a/doc/build/orm/persistence_techniques.rst +++ b/doc/build/orm/persistence_techniques.rst @@ -443,32 +443,105 @@ The above mapping upon INSERT will look like: .. _session_partitioning: -Partitioning Strategies -======================= +Partitioning Strategies (e.g. multiple database backends per Session) +===================================================================== Simple Vertical Partitioning ---------------------------- -Vertical partitioning places different kinds of objects, or different tables, -across multiple databases:: +Vertical partitioning places different classes, class hierarchies, +or mapped tables, across multiple databases, by configuring the +:class:`.Session` with the :paramref:`.Session.binds` argument. This +argument receives a dictionary that contains any combination of +ORM-mapped classes, arbitrary classes within a mapped hierarchy (such +as declarative base classes or mixins), :class:`.Table` objects, +and :class:`.Mapper` objects as keys, which then refer typically to +:class:`.Engine` or less typically :class:`.Connection` objects as targets. +The dictionary is consulted whenever the :class:`.Session` needs to +emit SQL on behalf of a particular kind of mapped class in order to locate +the appropriate source of database connectivity:: engine1 = create_engine('postgresql://db1') engine2 = create_engine('postgresql://db2') - Session = sessionmaker(twophase=True) + Session = sessionmaker() # bind User operations to engine 1, Account operations to engine 2 Session.configure(binds={User:engine1, Account:engine2}) session = Session() -Above, operations against either class will make usage of the :class:`.Engine` -linked to that class. Upon a flush operation, similar rules take place -to ensure each class is written to the right database. +Above, SQL operations against either class will make usage of the :class:`.Engine` +linked to that class. The functionality is comprehensive across both +read and write operations; a :class:`.Query` that is against entities +mapped to ``engine1`` (determined by looking at the first entity in the +list of items requested) will make use of ``engine1`` to run the query. A +flush operation will make use of **both** engines on a per-class basis as it +flushes objects of type ``User`` and ``Account``. + +In the more common case, there are typically base or mixin classes that can be +used to distinguish between operations that are destined for different database +connections. The :paramref:`.Session.binds` argument can accomodate any +arbitrary Python class as a key, which will be used if it is found to be in the +``__mro__`` (Python method resolution order) for a particular mapped class. +Supposing two declarative bases are representing two different database +connections:: + + BaseA = declarative_base() + + BaseB = declarative_base() + + class User(BaseA): + # ... + + class Address(BaseA): + # ... + -The transactions among the multiple databases can optionally be coordinated -via two phase commit, if the underlying backend supports it. See -:ref:`session_twophase` for an example. + class GameInfo(BaseB): + # ... + + class GameStats(BaseB): + # ... + + + Session = sessionmaker() + + # all User/Address operations will be on engine 1, all + # Game operations will be on engine 2 + Session.configure(binds={BaseA:engine1, BaseB:engine2}) + +Above, classes which descend from ``BaseA`` and ``BaseB`` will have their +SQL operations routed to one of two engines based on which superclass +they descend from, if any. In the case of a class that descends from more +than one "bound" superclass, the superclass that is highest in the target +class' hierarchy will be chosen to represent which engine should be used. + +.. seealso:: + + :paramref:`.Session.binds` + + +Coordination of Transactions for a multiple-engine Session +---------------------------------------------------------- + +One caveat to using multiple bound engines is in the case where a commit +operation may fail on one backend after the commit has succeeded on another. +This is an inconsistency problem that in relational databases is solved +using a "two phase transaction", which adds an additional "prepare" step +to the commit sequence that allows for multiple databases to agree to commit +before actually completing the transaction. + +Due to limited support within DBAPIs, SQLAlchemy has limited support for two- +phase transactions across backends. Most typically, it is known to work well +with the PostgreSQL backend and to a lesser extent with the MySQL backend. +However, the :class:`.Session` is fully capable of taking advantage of the two +phase transaction feature when the backend supports it, by setting the +:paramref:`.Session.use_twophase` flag within :class:`.sessionmaker` or +:class:`.Session`. See :ref:`session_twophase` for an example. + + +.. _session_custom_partitioning: Custom Vertical Partitioning ---------------------------- @@ -517,13 +590,19 @@ This approach can be combined with multiple :class:`.MetaData` objects, using an approach such as that of using the declarative ``__abstract__`` keyword, described at :ref:`declarative_abstract`. +.. seealso:: + + `Django-style Database Routers in SQLAlchemy `_ - blog post on a more comprehensive example of :meth:`.Session.get_bind` + Horizontal Partitioning ----------------------- Horizontal partitioning partitions the rows of a single table (or a set of -tables) across multiple databases. - -See the "sharding" example: :ref:`examples_sharding`. +tables) across multiple databases. The SQLAlchemy :class:`.Session` +contains support for this concept, however to use it fully requires that +:class:`.Session` and :class:`.Query` subclasses are used. A basic version +of these subclasses are available in the :ref:`horizontal_sharding_toplevel` +ORM extension. An example of use is at: :ref:`examples_sharding`. .. _bulk_operations: diff --git a/lib/sqlalchemy/orm/session.py b/lib/sqlalchemy/orm/session.py index d198de77ea..69c0c397d6 100644 --- a/lib/sqlalchemy/orm/session.py +++ b/lib/sqlalchemy/orm/session.py @@ -642,26 +642,37 @@ class Session(_SessionClassMethods): operations performed by this session will execute via this connectable. - :param binds: An optional dictionary which contains more granular - "bind" information than the ``bind`` parameter provides. This - dictionary can map individual :class`.Table` - instances as well as :class:`~.Mapper` instances to individual - :class:`.Engine` or :class:`.Connection` objects. Operations which - proceed relative to a particular :class:`.Mapper` will consult this - dictionary for the direct :class:`.Mapper` instance as - well as the mapper's ``mapped_table`` attribute in order to locate - a connectable to use. The full resolution is described in the - :meth:`.Session.get_bind`. - Usage looks like:: + :param binds: A dictionary which may specify any number of + :class:`.Engine` or :class:`.Connection` objects as the source of + connectivity for SQL operations on a per-entity basis. The keys + of the dictionary consist of any series of mapped classes, + arbitrary Python classes that are bases for mapped classes, + :class:`.Table` objects and :class:`.Mapper` objects. The + values of the dictionary are then instances of :class:`.Engine` + or less commonly :class:`.Connection` objects. Operations which + proceed relative to a particular mapped class will consult this + dictionary for the closest matching entity in order to determine + which :class:`.Engine` should be used for a particular SQL + operation. The complete heuristics for resolution are + described at :meth:`.Session.get_bind`. Usage looks like:: Session = sessionmaker(binds={ SomeMappedClass: create_engine('postgresql://engine1'), - somemapper: create_engine('postgresql://engine2'), - some_table: create_engine('postgresql://engine3'), + SomeDeclarativeBase: create_engine('postgresql://engine2'), + some_mapper: create_engine('postgresql://engine3'), + some_table: create_engine('postgresql://engine4'), }) - Also see the :meth:`.Session.bind_mapper` - and :meth:`.Session.bind_table` methods. + .. seealso:: + + :ref:`session_partitioning` + + :meth:`.Session.bind_mapper` + + :meth:`.Session.bind_table` + + :meth:`.Session.get_bind` + :param \class_: Specify an alternate class other than ``sqlalchemy.orm.session.Session`` which should be used by the @@ -1254,22 +1265,52 @@ class Session(_SessionClassMethods): "Not an acceptable bind target: %s" % key) def bind_mapper(self, mapper, bind): - """Associate a :class:`.Mapper` with a "bind", e.g. a :class:`.Engine` - or :class:`.Connection`. + """Associate a :class:`.Mapper` or arbitrary Python class with a + "bind", e.g. an :class:`.Engine` or :class:`.Connection`. - The given mapper is added to a lookup used by the + The given entity is added to a lookup used by the :meth:`.Session.get_bind` method. + :param mapper: a :class:`.Mapper` object, or an instance of a mapped + class, or any Python class that is the base of a set of mapped + classes. + + :param bind: an :class:`.Engine` or :class:`.Connection` object. + + .. seealso:: + + :ref:`session_partitioning` + + :paramref:`.Session.binds` + + :meth:`.Session.bind_table` + + """ self._add_bind(mapper, bind) def bind_table(self, table, bind): - """Associate a :class:`.Table` with a "bind", e.g. a :class:`.Engine` + """Associate a :class:`.Table` with a "bind", e.g. an :class:`.Engine` or :class:`.Connection`. - The given mapper is added to a lookup used by the + The given :class:`.Table` is added to a lookup used by the :meth:`.Session.get_bind` method. + :param table: a :class:`.Table` object, which is typically the target + of an ORM mapping, or is present within a selectable that is + mapped. + + :param bind: an :class:`.Engine` or :class:`.Connection` object. + + .. seealso:: + + :ref:`session_partitioning` + + :paramref:`.Session.binds` + + :meth:`.Session.bind_mapper` + + """ self._add_bind(table, bind) @@ -1293,7 +1334,10 @@ class Session(_SessionClassMethods): The order of resolution is: 1. if mapper given and session.binds is present, - locate a bind based on mapper. + locate a bind based first on the mapper in use, then + on the mapped class in use, then on any base classes that are + present in the ``__mro__`` of the mapped class, from more specific + superclasses to more general. 2. if clause given and session.binds is present, locate a bind based on :class:`.Table` objects found in the given clause present in session.binds. @@ -1308,6 +1352,11 @@ class Session(_SessionClassMethods): 6. No bind can be found, :exc:`~sqlalchemy.exc.UnboundExecutionError` is raised. + Note that the :meth:`.Session.get_bind` method can be overridden on + a user-defined subclass of :class:`.Session` to provide any kind + of bind resolution scheme. See the example at + :ref:`session_custom_partitioning`. + :param mapper: Optional :func:`.mapper` mapped class or instance of :class:`.Mapper`. The bind can be derived from a :class:`.Mapper` @@ -1324,6 +1373,16 @@ class Session(_SessionClassMethods): for a bound element, typically a :class:`.Table` associated with bound :class:`.MetaData`. + .. seealso:: + + :ref:`session_partitioning` + + :paramref:`.Session.binds` + + :meth:`.Session.bind_mapper` + + :meth:`.Session.bind_table` + """ if mapper is clause is None: