--- /dev/null
+.. change::
+ :tags: orm, feature
+ :tickets: 4736
+
+ Added new loader option method :meth:`.Load.options` which allows loader
+ options to be constructed hierarchically, so that many sub-options can be
+ applied to a particular path without needing to call :func:`.defaultload`
+ many times. Thanks to Alessio Bogon for the idea.
+
Deferred Column Loading
=======================
-This feature allows particular columns of a table be loaded only
+Deferred column loading allows particular columns of a table be loaded only
upon direct access, instead of when the entity is queried using
:class:`.Query`. This feature is useful when one wants to avoid
loading a large text or binary field into memory when it's not needed.
photo2 = deferred(Column(Binary), group='photos')
photo3 = deferred(Column(Binary), group='photos')
-You can defer or undefer columns at the :class:`~sqlalchemy.orm.query.Query`
-level using options, including :func:`.orm.defer` and :func:`.orm.undefer`::
+.. _deferred_options:
- from sqlalchemy.orm import defer, undefer
+Deferred Column Loader Query Options
+------------------------------------
+
+Columns can be marked as "deferred" or reset to "undeferred" at query time
+using options which are passed to the :meth:`.Query.options` method; the most
+basic query options are :func:`.orm.defer` and
+:func:`.orm.undefer`::
+
+ from sqlalchemy.orm import defer
+ from sqlalchemy.orm import undefer
query = session.query(Book)
- query = query.options(defer('summary'))
- query = query.options(undefer('excerpt'))
+ query = query.options(defer('summary'), undefer('excerpt'))
query.all()
+Above, the "summary" column will not load until accessed, and the "excerpt"
+column will load immediately even if it was mapped as a "deferred" column.
+
:func:`.orm.deferred` attributes which are marked with a "group" can be undeferred
using :func:`.orm.undefer_group`, sending in the group name::
query = session.query(Book)
query.options(undefer_group('photos')).all()
-Load Only Cols
---------------
+.. _deferred_loading_w_multiple:
-An arbitrary set of columns can be selected as "load only" columns, which will
-be loaded while deferring all other columns on a given entity, using :func:`.orm.load_only`::
+Deferred Loading across Multiple Entities
+-----------------------------------------
- from sqlalchemy.orm import load_only
+To specify column deferral for a :class:`.Query` that loads multiple types of
+entities at once, the deferral options may be specified more explicitly using
+class-bound attributes, rather than string names::
- session.query(Book).options(load_only("summary", "excerpt"))
+ from sqlalchemy.orm import defer
-.. versionadded:: 0.9.0
+ query = session.query(Book, Author).join(Book.author)
+ query = query.options(defer(Author.bio))
+
+Column deferral options may also indicate that they take place along various
+relationship paths, which are themselves often :ref:`eagerly loaded
+<loading_toplevel>` with loader options. All relationship-bound loader options
+support chaining onto additional loader options, which include loading for
+further levels of relationships, as well as onto column-oriented attributes at
+that path. Such as, to load ``Author`` instances, then joined-eager-load the
+``Author.books`` collection for each author, then apply deferral options to
+column-oriented attributes onto each ``Book`` entity from that relationship,
+the :func:`.joinedload` loader option can be combined with the :func:`.load_only`
+option (described later in this section) to defer all ``Book`` columns except
+those explicitly specified::
-.. _deferred_loading_w_multiple:
+ from sqlalchemy.orm import joinedload
-Deferred Loading with Multiple Entities
----------------------------------------
+ query = session.query(Author)
+ query = query.options(
+ joinedload(Author.books).load_only(Book.summary, Book.excerpt),
+ )
-To specify column deferral options within a :class:`.Query` that loads multiple types
-of entity, the :class:`.Load` object can specify which parent entity to start with::
+Option structures as above can also be organized in more complex ways, such
+as hierarchically using the :meth:`.Load.options`
+method, which allows multiple sub-options to be chained to a common parent
+option at once. Any mixture of string names and class-bound attribute objects
+may be used::
- from sqlalchemy.orm import Load
+ from sqlalchemy.orm import defer
+ from sqlalchemy.orm import joinedload
+ from sqlalchemy.orm import load_only
- query = session.query(Book, Author).join(Book.author)
+ query = session.query(Author)
query = query.options(
- Load(Book).load_only("summary", "excerpt"),
- Load(Author).defer("bio")
+ joinedload(Author.book).options(
+ load_only("summary", "excerpt"),
+ joinedload(Book.citations).options(
+ joinedload(Citation.author),
+ defer(Citation.fulltext)
+ )
+ )
)
-To specify column deferral options along the path of various relationships,
-the options support chaining, where the loading style of each relationship
-is specified first, then is chained to the deferral options. Such as, to load
-``Book`` instances, then joined-eager-load the ``Author``, then apply deferral
-options to the ``Author`` entity::
+.. versionadded:: 1.3.6 Added :meth:`.Load.options` to allow easier
+ construction of hierarchies of loader options.
- from sqlalchemy.orm import joinedload
+Another way to apply options to a path is to use the :func:`.orm.defaultload`
+function. This function is used to indicate a particular path within a loader
+option structure without actually setting any options at that level, so that further
+sub-options may be applied. The :func:`.orm.defaultload` function can be used
+to create the same structure as we did above using :meth:`.Load.options` as::
- query = session.query(Book)
+ query = session.query(Author)
query = query.options(
- joinedload(Book.author).load_only("summary", "excerpt"),
- )
+ joinedload(Author.book).load_only("summary", "excerpt"),
+ defaultload(Author.book).joinedload(Book.citations).joinedload(Citation.author),
+ defaultload(Author.book).defaultload(Book.citations).defer(Citation.fulltext)
+ )
-In the case where the loading style of parent relationships should be left
-unchanged, use :func:`.orm.defaultload`::
+.. seealso::
- from sqlalchemy.orm import defaultload
+ :ref:`relationship_loader_options` - targeted towards relationship loading
- query = session.query(Book)
+Load Only and Wildcard Options
+------------------------------
+
+The ORM loader option system supports the concept of "wildcard" loader options,
+in which a loader option can be passed an asterisk ``"*"`` to indicate that
+a particular option should apply to all applicable attributes of a mapped
+class. Such as, if we wanted to load the ``Book`` class but only
+the "summary" and "excerpt" columns, we could say::
+
+ from sqlalchemy.orm import defer
+ from sqlalchemy.orm import undefer
+
+ session.query(Book).options(
+ defer('*'), undefer("summary"), undefer("excerpt"))
+
+Above, the :func:`.defer` option is applied using a wildcard to all column
+attributes on the ``Book`` class. Then, the :func:`.undefer` option is used
+against the "summary" and "excerpt" fields so that they are the only columns
+loaded up front. A query for the above entity will include only the "summary"
+and "excerpt" fields in the SELECT, along with the primary key columns which
+are always used by the ORM.
+
+A similar function is available with less verbosity by using the
+:func:`.orm.load_only` option. This is a so-called **exclusionary** option
+which will apply deferred behavior to all column attributes except those
+that are named::
+
+ from sqlalchemy.orm import load_only
+
+ session.query(Book).options(load_only("summary", "excerpt"))
+
+Wildcard and Exclusionary Options with Multiple-Entity Queries
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Wildcard options and exclusionary options such as :func:`.load_only` may
+only be applied to a single entity at a time within a :class:`.Query`. To
+suit the less common case where a :class:`.Query` is returning multiple
+primary entities at once, a special calling style may be required in order
+to apply a wildcard or exclusionary option, which is to use the
+:class:`.Load` object to indicate the starting entity for a deferral option.
+Such as, if we were loading ``Book`` and ``Author`` at once, the :class:`.Query`
+will raise an informative error if we try to apply :func:`.load_only` to
+both at once. Using :class:`.Load` looks like::
+
+ from sqlalchemy.orm import Load
+
+ query = session.query(Book, Author).join(Book.author)
query = query.options(
- defaultload(Book.author).load_only("summary", "excerpt"),
+ Load(Book).load_only("summary", "excerpt")
)
-.. versionadded:: 0.9.0 support for :class:`.Load` and other options which
- allow for better targeting of deferral options.
+Above, :class:`.Load` is used in conjunction with the exclusionary option
+:func:`.load_only` so that the deferral of all other columns only takes
+place for the ``Book`` class and not the ``Author`` class. Again,
+the :class:`.Query` object should raise an informative error message when
+the above calling style is actually required that describes those cases
+where explicit use of :class:`.Load` is needed.
Column Deferral API
-------------------
.. _relationship_loader_options:
-Controlling Loading via Options
--------------------------------
+Relationship Loading with Loader Options
+----------------------------------------
The other, and possibly more common way to configure loading strategies
-is to set them up on a per-query basis against specific attributes. Very detailed
+is to set them up on a per-query basis against specific attributes using the
+:meth:`.Query.options` method. Very detailed
control over relationship loading is available using loader options;
the most common are
:func:`~sqlalchemy.orm.joinedload`,
of a particular attribute, the :func:`.defaultload` method/function may be used::
session.query(A).options(
- defaultload("atob").
- joinedload("btoc")).all()
+ defaultload(A.atob).
+ joinedload(B.btoc)).all()
+
+A similar approach can be used to specify multiple sub-options at once, using
+the :meth:`.Load.options` method::
+
+ session.query(A).options(
+ defaultload(A.atob).options(
+ joinedload(B.btoc),
+ joinedload(B.btod)
+ )).all()
+
+.. versionadded:: 1.3.6 added :meth:`.Load.options`
+
+
+.. seealso::
+
+ :ref:`deferred_loading_w_multiple` - illustrates examples of combining
+ relationship and column-oriented loader options.
+
.. note:: The loader options applied to an object's lazy-loaded collections
are **"sticky"** to specific object instances, meaning they will persist
return self.add_columns(column)
def options(self, *args):
- """Return a new Query object, applying the given list of
+ """Return a new :class:`.Query` object, applying the given list of
mapper options.
Most supplied options regard changing how column- and
- relationship-mapped attributes are loaded. See the sections
- :ref:`deferred` and :doc:`/orm/loading_relationships` for reference
- documentation.
+ relationship-mapped attributes are loaded.
+
+ .. seealso::
+
+ :ref:`deferred_options`
+
+ :ref:`relationship_loader_options`
"""
return self._options(False, *args)
.. seealso::
- :ref:`loading_toplevel`
+ :ref:`deferred_options`
+
+ :ref:`deferred_loading_w_multiple`
+
+ :ref:`relationship_loader_options`
"""
strategy = tuple(sorted(strategy.items()))
return strategy
+ def _apply_to_parent(self, parent, applied, bound):
+ raise NotImplementedError(
+ "Only 'unbound' loader options may be used with the "
+ "Load.options() method"
+ )
+
+ @_generative
+ def options(self, *opts):
+ r"""Apply a series of options as sub-options to this :class:`.Load`
+ object.
+
+ E.g.::
+
+ query = session.query(Author)
+ query = query.options(
+ joinedload(Author.book).options(
+ load_only("summary", "excerpt"),
+ joinedload(Book.citations).options(
+ joinedload(Citation.author)
+ )
+ )
+ )
+
+ :param \*opts: A series of loader option objects (ultimately
+ :class:`.Load` objects) which should be applied to the path
+ specified by this :class:`.Load` object.
+
+ .. versionadded:: 1.3.6
+
+ .. seealso::
+
+ :func:`.defaultload`
+
+ :ref:`relationship_loader_options`
+
+ :ref:`deferred_loading_w_multiple`
+
+ """
+ apply_cache = {}
+ bound = not isinstance(self, _UnboundLoad)
+ if bound:
+ raise NotImplementedError(
+ "The options() method is currently only supported "
+ "for 'unbound' loader options"
+ )
+ for opt in opts:
+ opt._apply_to_parent(self, apply_cache, bound)
+
@_generative
def set_relationship_strategy(
self, attr, strategy, propagate_to_loaders=True
):
strategy = self._coerce_strat(strategy)
-
self.is_class_strategy = False
self.propagate_to_loaders = propagate_to_loaders
# if the path is a wildcard, this will set propagate_to_loaders=False
def _set_path_strategy(self):
self._to_bind.append(self)
+ def _apply_to_parent(self, parent, applied, bound):
+ if self in applied:
+ return applied[self]
+
+ cloned = self._generate()
+
+ applied[self] = cloned
+
+ cloned.strategy = self.strategy
+ if self.path:
+ attr = self.path[-1]
+ if isinstance(attr, util.string_types) and attr.endswith(
+ _DEFAULT_TOKEN
+ ):
+ attr = attr.split(":")[0] + ":" + _WILDCARD_TOKEN
+ cloned._generate_path(
+ parent.path + self.path[0:-1], attr, self.strategy, None
+ )
+
+ # these assertions can go away once the "sub options" API is
+ # mature
+ assert cloned.propagate_to_loaders == self.propagate_to_loaders
+ assert cloned.is_class_strategy == self.is_class_strategy
+ assert cloned.is_opts_only == self.is_opts_only
+
+ new_to_bind = {
+ elem._apply_to_parent(parent, applied, bound)
+ for elem in self._to_bind
+ }
+ cloned._to_bind = parent._to_bind
+ cloned._to_bind.extend(new_to_bind)
+ cloned.local_opts.update(self.local_opts)
+
+ return cloned
+
def _generate_path(self, path, attr, for_strategy, wildcard_key):
if (
wildcard_key
.. seealso::
+ :meth:`.Load.options` - allows for complex hierarchical
+ loader option structures with less verbosity than with individual
+ :func:`.defaultload` directives.
+
:ref:`relationship_loader_options`
:ref:`deferred_loading_w_multiple`
from sqlalchemy.orm import column_property
from sqlalchemy.orm import create_session
from sqlalchemy.orm import defaultload
+from sqlalchemy.orm import defer
from sqlalchemy.orm import exc as orm_exc
from sqlalchemy.orm import joinedload
from sqlalchemy.orm import lazyload
self._assert_attrs(opts, {"foo": "bar", "bat": "hoho"})
+class SubOptionsTest(PathTest, QueryTest):
+ run_create_tables = False
+ run_inserts = None
+ run_deletes = None
+
+ def _assert_opts(self, q, sub_opt, non_sub_opts):
+ existing_attributes = q._attributes
+ q._attributes = q._attributes.copy()
+ attr_a = {}
+
+ for val in sub_opt._to_bind:
+ val._bind_loader(
+ [ent.entity_zero for ent in q._mapper_entities],
+ q._current_path,
+ attr_a,
+ False,
+ )
+
+ q._attributes = existing_attributes.copy()
+
+ attr_b = {}
+
+ for opt in non_sub_opts:
+ for val in opt._to_bind:
+ val._bind_loader(
+ [ent.entity_zero for ent in q._mapper_entities],
+ q._current_path,
+ attr_b,
+ False,
+ )
+
+ for k, l in attr_b.items():
+ if not l.strategy:
+ del attr_b[k]
+
+ def strat_as_tuple(strat):
+ return (
+ strat.strategy,
+ strat.local_opts,
+ strat.propagate_to_loaders,
+ strat._of_type,
+ strat.is_class_strategy,
+ strat.is_opts_only,
+ )
+
+ eq_(
+ {path: strat_as_tuple(load) for path, load in attr_a.items()},
+ {path: strat_as_tuple(load) for path, load in attr_b.items()},
+ )
+
+ def test_one(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+ sub_opt = joinedload(User.orders).options(
+ joinedload(Order.items).options(defer(Item.description)),
+ defer(Order.description),
+ )
+ non_sub_opts = [
+ joinedload(User.orders),
+ defaultload(User.orders)
+ .joinedload(Order.items)
+ .defer(Item.description),
+ defaultload(User.orders).defer(Order.description),
+ ]
+
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_two(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+ sub_opt = defaultload(User.orders).options(
+ joinedload(Order.items),
+ defaultload(Order.items).options(subqueryload(Item.keywords)),
+ defer(Order.description),
+ )
+ non_sub_opts = [
+ defaultload(User.orders)
+ .joinedload(Order.items)
+ .subqueryload(Item.keywords),
+ defaultload(User.orders).defer(Order.description),
+ ]
+
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_three(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+ sub_opt = defaultload(User.orders).options(defer("*"))
+ non_sub_opts = [defaultload(User.orders).defer("*")]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_four(self):
+ User, Address, Order, Item, SubItem, Keyword = self.classes(
+ "User", "Address", "Order", "Item", "SubItem", "Keyword"
+ )
+ sub_opt = joinedload(User.orders).options(
+ defer(Order.description),
+ joinedload(Order.items).options(
+ joinedload(Item.keywords).options(defer(Keyword.name)),
+ defer(Item.description),
+ ),
+ )
+ non_sub_opts = [
+ joinedload(User.orders),
+ defaultload(User.orders).defer(Order.description),
+ defaultload(User.orders).joinedload(Order.items),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .joinedload(Item.keywords),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .defer(Item.description),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .defaultload(Item.keywords)
+ .defer(Keyword.name),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_four_strings(self):
+ User, Address, Order, Item, SubItem, Keyword = self.classes(
+ "User", "Address", "Order", "Item", "SubItem", "Keyword"
+ )
+ sub_opt = joinedload("orders").options(
+ defer("description"),
+ joinedload("items").options(
+ joinedload("keywords").options(defer("name")),
+ defer("description"),
+ ),
+ )
+ non_sub_opts = [
+ joinedload(User.orders),
+ defaultload(User.orders).defer(Order.description),
+ defaultload(User.orders).joinedload(Order.items),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .joinedload(Item.keywords),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .defer(Item.description),
+ defaultload(User.orders)
+ .defaultload(Order.items)
+ .defaultload(Item.keywords)
+ .defer(Keyword.name),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_five(self):
+ User, Address, Order, Item, SubItem, Keyword = self.classes(
+ "User", "Address", "Order", "Item", "SubItem", "Keyword"
+ )
+ sub_opt = joinedload(User.orders).options(load_only(Order.description))
+ non_sub_opts = [
+ joinedload(User.orders),
+ defaultload(User.orders).load_only(Order.description),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_five_strings(self):
+ User, Address, Order, Item, SubItem, Keyword = self.classes(
+ "User", "Address", "Order", "Item", "SubItem", "Keyword"
+ )
+ sub_opt = joinedload("orders").options(load_only("description"))
+ non_sub_opts = [
+ joinedload(User.orders),
+ defaultload(User.orders).load_only(Order.description),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_invalid_one(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+
+ # these options are "invalid", in that User.orders -> Item.keywords
+ # is not a path. However, the "normal" option is not generating
+ # an error for now, which is bad, but we're testing here only that
+ # it works the same way, so there you go. If and when we make this
+ # case raise, then both cases should raise in the same way.
+ sub_opt = joinedload(User.orders).options(
+ joinedload(Item.keywords), joinedload(Order.items)
+ )
+ non_sub_opts = [
+ joinedload(User.orders).joinedload(Item.keywords),
+ defaultload(User.orders).joinedload(Order.items),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_invalid_two(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+
+ # these options are "invalid", in that User.orders -> Item.keywords
+ # is not a path. However, the "normal" option is not generating
+ # an error for now, which is bad, but we're testing here only that
+ # it works the same way, so there you go. If and when we make this
+ # case raise, then both cases should raise in the same way.
+ sub_opt = joinedload("orders").options(
+ joinedload("keywords"), joinedload("items")
+ )
+ non_sub_opts = [
+ joinedload(User.orders).joinedload(Item.keywords),
+ defaultload(User.orders).joinedload(Order.items),
+ ]
+ sess = Session()
+ self._assert_opts(sess.query(User), sub_opt, non_sub_opts)
+
+ def test_not_implemented_fromload(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+
+ assert_raises_message(
+ NotImplementedError,
+ r"The options\(\) method is currently only supported "
+ "for 'unbound' loader options",
+ Load(User).joinedload(User.orders).options,
+ joinedload(Order.items),
+ )
+
+ def test_not_implemented_toload(self):
+ User, Address, Order, Item, SubItem = self.classes(
+ "User", "Address", "Order", "Item", "SubItem"
+ )
+
+ assert_raises_message(
+ NotImplementedError,
+ r"Only 'unbound' loader options may be used with the "
+ r"Load.options\(\) method",
+ joinedload(User.orders).options,
+ Load(Order).joinedload(Order.items),
+ )
+
+
class CacheKeyTest(PathTest, QueryTest):
run_create_tables = False
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
