From: Mike Bayer Date: Thu, 20 Jun 2019 19:37:59 +0000 (-0400) Subject: Add Load.options() for hierchical construction of loader options X-Git-Tag: rel_1_3_6~28 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ce9de9fff675a96cb4aaef5d7d34f93dc9ad7002;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Add Load.options() for hierchical construction of loader options Added new loader option method :meth:`.Load.options` which allows loader options to be constructed hierarchically, so that many sub-options can be applied to a particular path without needing to call :func:`.defaultload` many times. Thanks to Alessio Bogon for the idea. Also applies a large pass to the loader option documentation which needed improvement. Fixes: #4736 Change-Id: I93c453e30a20c074f27e87cf7e95b13dd3f2b494 (cherry picked from commit f0d1a5364fa8a9585b709239f85c4092439c4cd8) --- diff --git a/doc/build/changelog/unreleased_13/4736.rst b/doc/build/changelog/unreleased_13/4736.rst new file mode 100644 index 0000000000..3e5d30023b --- /dev/null +++ b/doc/build/changelog/unreleased_13/4736.rst @@ -0,0 +1,9 @@ +.. change:: + :tags: orm, feature + :tickets: 4736 + + Added new loader option method :meth:`.Load.options` which allows loader + options to be constructed hierarchically, so that many sub-options can be + applied to a particular path without needing to call :func:`.defaultload` + many times. Thanks to Alessio Bogon for the idea. + diff --git a/doc/build/orm/loading_columns.rst b/doc/build/orm/loading_columns.rst index 8766dac1b4..d128651453 100644 --- a/doc/build/orm/loading_columns.rst +++ b/doc/build/orm/loading_columns.rst @@ -11,7 +11,7 @@ This section presents additional options regarding the loading of columns. Deferred Column Loading ======================= -This feature allows particular columns of a table be loaded only +Deferred column loading allows particular columns of a table be loaded only upon direct access, instead of when the entity is queried using :class:`.Query`. This feature is useful when one wants to avoid loading a large text or binary field into memory when it's not needed. @@ -57,16 +57,26 @@ separately when it is accessed:: photo2 = deferred(Column(Binary), group='photos') photo3 = deferred(Column(Binary), group='photos') -You can defer or undefer columns at the :class:`~sqlalchemy.orm.query.Query` -level using options, including :func:`.orm.defer` and :func:`.orm.undefer`:: +.. _deferred_options: - from sqlalchemy.orm import defer, undefer +Deferred Column Loader Query Options +------------------------------------ + +Columns can be marked as "deferred" or reset to "undeferred" at query time +using options which are passed to the :meth:`.Query.options` method; the most +basic query options are :func:`.orm.defer` and +:func:`.orm.undefer`:: + + from sqlalchemy.orm import defer + from sqlalchemy.orm import undefer query = session.query(Book) - query = query.options(defer('summary')) - query = query.options(undefer('excerpt')) + query = query.options(defer('summary'), undefer('excerpt')) query.all() +Above, the "summary" column will not load until accessed, and the "excerpt" +column will load immediately even if it was mapped as a "deferred" column. + :func:`.orm.deferred` attributes which are marked with a "group" can be undeferred using :func:`.orm.undefer_group`, sending in the group name:: @@ -75,59 +85,137 @@ using :func:`.orm.undefer_group`, sending in the group name:: query = session.query(Book) query.options(undefer_group('photos')).all() -Load Only Cols --------------- +.. _deferred_loading_w_multiple: -An arbitrary set of columns can be selected as "load only" columns, which will -be loaded while deferring all other columns on a given entity, using :func:`.orm.load_only`:: +Deferred Loading across Multiple Entities +----------------------------------------- - from sqlalchemy.orm import load_only +To specify column deferral for a :class:`.Query` that loads multiple types of +entities at once, the deferral options may be specified more explicitly using +class-bound attributes, rather than string names:: - session.query(Book).options(load_only("summary", "excerpt")) + from sqlalchemy.orm import defer -.. versionadded:: 0.9.0 + query = session.query(Book, Author).join(Book.author) + query = query.options(defer(Author.bio)) + +Column deferral options may also indicate that they take place along various +relationship paths, which are themselves often :ref:`eagerly loaded +` with loader options. All relationship-bound loader options +support chaining onto additional loader options, which include loading for +further levels of relationships, as well as onto column-oriented attributes at +that path. Such as, to load ``Author`` instances, then joined-eager-load the +``Author.books`` collection for each author, then apply deferral options to +column-oriented attributes onto each ``Book`` entity from that relationship, +the :func:`.joinedload` loader option can be combined with the :func:`.load_only` +option (described later in this section) to defer all ``Book`` columns except +those explicitly specified:: -.. _deferred_loading_w_multiple: + from sqlalchemy.orm import joinedload -Deferred Loading with Multiple Entities ---------------------------------------- + query = session.query(Author) + query = query.options( + joinedload(Author.books).load_only(Book.summary, Book.excerpt), + ) -To specify column deferral options within a :class:`.Query` that loads multiple types -of entity, the :class:`.Load` object can specify which parent entity to start with:: +Option structures as above can also be organized in more complex ways, such +as hierarchically using the :meth:`.Load.options` +method, which allows multiple sub-options to be chained to a common parent +option at once. Any mixture of string names and class-bound attribute objects +may be used:: - from sqlalchemy.orm import Load + from sqlalchemy.orm import defer + from sqlalchemy.orm import joinedload + from sqlalchemy.orm import load_only - query = session.query(Book, Author).join(Book.author) + query = session.query(Author) query = query.options( - Load(Book).load_only("summary", "excerpt"), - Load(Author).defer("bio") + joinedload(Author.book).options( + load_only("summary", "excerpt"), + joinedload(Book.citations).options( + joinedload(Citation.author), + defer(Citation.fulltext) + ) + ) ) -To specify column deferral options along the path of various relationships, -the options support chaining, where the loading style of each relationship -is specified first, then is chained to the deferral options. Such as, to load -``Book`` instances, then joined-eager-load the ``Author``, then apply deferral -options to the ``Author`` entity:: +.. versionadded:: 1.3.6 Added :meth:`.Load.options` to allow easier + construction of hierarchies of loader options. - from sqlalchemy.orm import joinedload +Another way to apply options to a path is to use the :func:`.orm.defaultload` +function. This function is used to indicate a particular path within a loader +option structure without actually setting any options at that level, so that further +sub-options may be applied. The :func:`.orm.defaultload` function can be used +to create the same structure as we did above using :meth:`.Load.options` as:: - query = session.query(Book) + query = session.query(Author) query = query.options( - joinedload(Book.author).load_only("summary", "excerpt"), - ) + joinedload(Author.book).load_only("summary", "excerpt"), + defaultload(Author.book).joinedload(Book.citations).joinedload(Citation.author), + defaultload(Author.book).defaultload(Book.citations).defer(Citation.fulltext) + ) -In the case where the loading style of parent relationships should be left -unchanged, use :func:`.orm.defaultload`:: +.. seealso:: - from sqlalchemy.orm import defaultload + :ref:`relationship_loader_options` - targeted towards relationship loading - query = session.query(Book) +Load Only and Wildcard Options +------------------------------ + +The ORM loader option system supports the concept of "wildcard" loader options, +in which a loader option can be passed an asterisk ``"*"`` to indicate that +a particular option should apply to all applicable attributes of a mapped +class. Such as, if we wanted to load the ``Book`` class but only +the "summary" and "excerpt" columns, we could say:: + + from sqlalchemy.orm import defer + from sqlalchemy.orm import undefer + + session.query(Book).options( + defer('*'), undefer("summary"), undefer("excerpt")) + +Above, the :func:`.defer` option is applied using a wildcard to all column +attributes on the ``Book`` class. Then, the :func:`.undefer` option is used +against the "summary" and "excerpt" fields so that they are the only columns +loaded up front. A query for the above entity will include only the "summary" +and "excerpt" fields in the SELECT, along with the primary key columns which +are always used by the ORM. + +A similar function is available with less verbosity by using the +:func:`.orm.load_only` option. This is a so-called **exclusionary** option +which will apply deferred behavior to all column attributes except those +that are named:: + + from sqlalchemy.orm import load_only + + session.query(Book).options(load_only("summary", "excerpt")) + +Wildcard and Exclusionary Options with Multiple-Entity Queries +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Wildcard options and exclusionary options such as :func:`.load_only` may +only be applied to a single entity at a time within a :class:`.Query`. To +suit the less common case where a :class:`.Query` is returning multiple +primary entities at once, a special calling style may be required in order +to apply a wildcard or exclusionary option, which is to use the +:class:`.Load` object to indicate the starting entity for a deferral option. +Such as, if we were loading ``Book`` and ``Author`` at once, the :class:`.Query` +will raise an informative error if we try to apply :func:`.load_only` to +both at once. Using :class:`.Load` looks like:: + + from sqlalchemy.orm import Load + + query = session.query(Book, Author).join(Book.author) query = query.options( - defaultload(Book.author).load_only("summary", "excerpt"), + Load(Book).load_only("summary", "excerpt") ) -.. versionadded:: 0.9.0 support for :class:`.Load` and other options which - allow for better targeting of deferral options. +Above, :class:`.Load` is used in conjunction with the exclusionary option +:func:`.load_only` so that the deferral of all other columns only takes +place for the ``Book`` class and not the ``Author`` class. Again, +the :class:`.Query` object should raise an informative error message when +the above calling style is actually required that describes those cases +where explicit use of :class:`.Load` is needed. Column Deferral API ------------------- diff --git a/doc/build/orm/loading_relationships.rst b/doc/build/orm/loading_relationships.rst index d53631cc2a..1a60693cd6 100644 --- a/doc/build/orm/loading_relationships.rst +++ b/doc/build/orm/loading_relationships.rst @@ -97,11 +97,12 @@ further background. .. _relationship_loader_options: -Controlling Loading via Options -------------------------------- +Relationship Loading with Loader Options +---------------------------------------- The other, and possibly more common way to configure loading strategies -is to set them up on a per-query basis against specific attributes. Very detailed +is to set them up on a per-query basis against specific attributes using the +:meth:`.Query.options` method. Very detailed control over relationship loading is available using loader options; the most common are :func:`~sqlalchemy.orm.joinedload`, @@ -145,8 +146,26 @@ stated. To navigate along a path without changing the existing loader style of a particular attribute, the :func:`.defaultload` method/function may be used:: session.query(A).options( - defaultload("atob"). - joinedload("btoc")).all() + defaultload(A.atob). + joinedload(B.btoc)).all() + +A similar approach can be used to specify multiple sub-options at once, using +the :meth:`.Load.options` method:: + + session.query(A).options( + defaultload(A.atob).options( + joinedload(B.btoc), + joinedload(B.btod) + )).all() + +.. versionadded:: 1.3.6 added :meth:`.Load.options` + + +.. seealso:: + + :ref:`deferred_loading_w_multiple` - illustrates examples of combining + relationship and column-oriented loader options. + .. note:: The loader options applied to an object's lazy-loaded collections are **"sticky"** to specific object instances, meaning they will persist diff --git a/lib/sqlalchemy/orm/query.py b/lib/sqlalchemy/orm/query.py index 713fa326cf..13d4334c43 100644 --- a/lib/sqlalchemy/orm/query.py +++ b/lib/sqlalchemy/orm/query.py @@ -1511,13 +1511,17 @@ class Query(object): return self.add_columns(column) def options(self, *args): - """Return a new Query object, applying the given list of + """Return a new :class:`.Query` object, applying the given list of mapper options. Most supplied options regard changing how column- and - relationship-mapped attributes are loaded. See the sections - :ref:`deferred` and :doc:`/orm/loading_relationships` for reference - documentation. + relationship-mapped attributes are loaded. + + .. seealso:: + + :ref:`deferred_options` + + :ref:`relationship_loader_options` """ return self._options(False, *args) diff --git a/lib/sqlalchemy/orm/strategy_options.py b/lib/sqlalchemy/orm/strategy_options.py index 912b6b5503..b444a03025 100644 --- a/lib/sqlalchemy/orm/strategy_options.py +++ b/lib/sqlalchemy/orm/strategy_options.py @@ -61,7 +61,11 @@ class Load(Generative, MapperOption): .. seealso:: - :ref:`loading_toplevel` + :ref:`deferred_options` + + :ref:`deferred_loading_w_multiple` + + :ref:`relationship_loader_options` """ @@ -319,12 +323,59 @@ class Load(Generative, MapperOption): strategy = tuple(sorted(strategy.items())) return strategy + def _apply_to_parent(self, parent, applied, bound): + raise NotImplementedError( + "Only 'unbound' loader options may be used with the " + "Load.options() method" + ) + + @_generative + def options(self, *opts): + r"""Apply a series of options as sub-options to this :class:`.Load` + object. + + E.g.:: + + query = session.query(Author) + query = query.options( + joinedload(Author.book).options( + load_only("summary", "excerpt"), + joinedload(Book.citations).options( + joinedload(Citation.author) + ) + ) + ) + + :param \*opts: A series of loader option objects (ultimately + :class:`.Load` objects) which should be applied to the path + specified by this :class:`.Load` object. + + .. versionadded:: 1.3.6 + + .. seealso:: + + :func:`.defaultload` + + :ref:`relationship_loader_options` + + :ref:`deferred_loading_w_multiple` + + """ + apply_cache = {} + bound = not isinstance(self, _UnboundLoad) + if bound: + raise NotImplementedError( + "The options() method is currently only supported " + "for 'unbound' loader options" + ) + for opt in opts: + opt._apply_to_parent(self, apply_cache, bound) + @_generative def set_relationship_strategy( self, attr, strategy, propagate_to_loaders=True ): strategy = self._coerce_strat(strategy) - self.is_class_strategy = False self.propagate_to_loaders = propagate_to_loaders # if the path is a wildcard, this will set propagate_to_loaders=False @@ -490,6 +541,41 @@ class _UnboundLoad(Load): def _set_path_strategy(self): self._to_bind.append(self) + def _apply_to_parent(self, parent, applied, bound): + if self in applied: + return applied[self] + + cloned = self._generate() + + applied[self] = cloned + + cloned.strategy = self.strategy + if self.path: + attr = self.path[-1] + if isinstance(attr, util.string_types) and attr.endswith( + _DEFAULT_TOKEN + ): + attr = attr.split(":")[0] + ":" + _WILDCARD_TOKEN + cloned._generate_path( + parent.path + self.path[0:-1], attr, self.strategy, None + ) + + # these assertions can go away once the "sub options" API is + # mature + assert cloned.propagate_to_loaders == self.propagate_to_loaders + assert cloned.is_class_strategy == self.is_class_strategy + assert cloned.is_opts_only == self.is_opts_only + + new_to_bind = { + elem._apply_to_parent(parent, applied, bound) + for elem in self._to_bind + } + cloned._to_bind = parent._to_bind + cloned._to_bind.extend(new_to_bind) + cloned.local_opts.update(self.local_opts) + + return cloned + def _generate_path(self, path, attr, for_strategy, wildcard_key): if ( wildcard_key @@ -1324,6 +1410,10 @@ def defaultload(loadopt, attr): .. seealso:: + :meth:`.Load.options` - allows for complex hierarchical + loader option structures with less verbosity than with individual + :func:`.defaultload` directives. + :ref:`relationship_loader_options` :ref:`deferred_loading_w_multiple` diff --git a/test/orm/test_options.py b/test/orm/test_options.py index b75cbf2052..bf099e7e6d 100644 --- a/test/orm/test_options.py +++ b/test/orm/test_options.py @@ -10,6 +10,7 @@ from sqlalchemy.orm import class_mapper from sqlalchemy.orm import column_property from sqlalchemy.orm import create_session from sqlalchemy.orm import defaultload +from sqlalchemy.orm import defer from sqlalchemy.orm import exc as orm_exc from sqlalchemy.orm import joinedload from sqlalchemy.orm import lazyload @@ -1543,6 +1544,252 @@ class LocalOptsTest(PathTest, QueryTest): self._assert_attrs(opts, {"foo": "bar", "bat": "hoho"}) +class SubOptionsTest(PathTest, QueryTest): + run_create_tables = False + run_inserts = None + run_deletes = None + + def _assert_opts(self, q, sub_opt, non_sub_opts): + existing_attributes = q._attributes + q._attributes = q._attributes.copy() + attr_a = {} + + for val in sub_opt._to_bind: + val._bind_loader( + [ent.entity_zero for ent in q._mapper_entities], + q._current_path, + attr_a, + False, + ) + + q._attributes = existing_attributes.copy() + + attr_b = {} + + for opt in non_sub_opts: + for val in opt._to_bind: + val._bind_loader( + [ent.entity_zero for ent in q._mapper_entities], + q._current_path, + attr_b, + False, + ) + + for k, l in attr_b.items(): + if not l.strategy: + del attr_b[k] + + def strat_as_tuple(strat): + return ( + strat.strategy, + strat.local_opts, + strat.propagate_to_loaders, + strat._of_type, + strat.is_class_strategy, + strat.is_opts_only, + ) + + eq_( + {path: strat_as_tuple(load) for path, load in attr_a.items()}, + {path: strat_as_tuple(load) for path, load in attr_b.items()}, + ) + + def test_one(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + sub_opt = joinedload(User.orders).options( + joinedload(Order.items).options(defer(Item.description)), + defer(Order.description), + ) + non_sub_opts = [ + joinedload(User.orders), + defaultload(User.orders) + .joinedload(Order.items) + .defer(Item.description), + defaultload(User.orders).defer(Order.description), + ] + + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_two(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + sub_opt = defaultload(User.orders).options( + joinedload(Order.items), + defaultload(Order.items).options(subqueryload(Item.keywords)), + defer(Order.description), + ) + non_sub_opts = [ + defaultload(User.orders) + .joinedload(Order.items) + .subqueryload(Item.keywords), + defaultload(User.orders).defer(Order.description), + ] + + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_three(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + sub_opt = defaultload(User.orders).options(defer("*")) + non_sub_opts = [defaultload(User.orders).defer("*")] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_four(self): + User, Address, Order, Item, SubItem, Keyword = self.classes( + "User", "Address", "Order", "Item", "SubItem", "Keyword" + ) + sub_opt = joinedload(User.orders).options( + defer(Order.description), + joinedload(Order.items).options( + joinedload(Item.keywords).options(defer(Keyword.name)), + defer(Item.description), + ), + ) + non_sub_opts = [ + joinedload(User.orders), + defaultload(User.orders).defer(Order.description), + defaultload(User.orders).joinedload(Order.items), + defaultload(User.orders) + .defaultload(Order.items) + .joinedload(Item.keywords), + defaultload(User.orders) + .defaultload(Order.items) + .defer(Item.description), + defaultload(User.orders) + .defaultload(Order.items) + .defaultload(Item.keywords) + .defer(Keyword.name), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_four_strings(self): + User, Address, Order, Item, SubItem, Keyword = self.classes( + "User", "Address", "Order", "Item", "SubItem", "Keyword" + ) + sub_opt = joinedload("orders").options( + defer("description"), + joinedload("items").options( + joinedload("keywords").options(defer("name")), + defer("description"), + ), + ) + non_sub_opts = [ + joinedload(User.orders), + defaultload(User.orders).defer(Order.description), + defaultload(User.orders).joinedload(Order.items), + defaultload(User.orders) + .defaultload(Order.items) + .joinedload(Item.keywords), + defaultload(User.orders) + .defaultload(Order.items) + .defer(Item.description), + defaultload(User.orders) + .defaultload(Order.items) + .defaultload(Item.keywords) + .defer(Keyword.name), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_five(self): + User, Address, Order, Item, SubItem, Keyword = self.classes( + "User", "Address", "Order", "Item", "SubItem", "Keyword" + ) + sub_opt = joinedload(User.orders).options(load_only(Order.description)) + non_sub_opts = [ + joinedload(User.orders), + defaultload(User.orders).load_only(Order.description), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_five_strings(self): + User, Address, Order, Item, SubItem, Keyword = self.classes( + "User", "Address", "Order", "Item", "SubItem", "Keyword" + ) + sub_opt = joinedload("orders").options(load_only("description")) + non_sub_opts = [ + joinedload(User.orders), + defaultload(User.orders).load_only(Order.description), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_invalid_one(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + + # these options are "invalid", in that User.orders -> Item.keywords + # is not a path. However, the "normal" option is not generating + # an error for now, which is bad, but we're testing here only that + # it works the same way, so there you go. If and when we make this + # case raise, then both cases should raise in the same way. + sub_opt = joinedload(User.orders).options( + joinedload(Item.keywords), joinedload(Order.items) + ) + non_sub_opts = [ + joinedload(User.orders).joinedload(Item.keywords), + defaultload(User.orders).joinedload(Order.items), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_invalid_two(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + + # these options are "invalid", in that User.orders -> Item.keywords + # is not a path. However, the "normal" option is not generating + # an error for now, which is bad, but we're testing here only that + # it works the same way, so there you go. If and when we make this + # case raise, then both cases should raise in the same way. + sub_opt = joinedload("orders").options( + joinedload("keywords"), joinedload("items") + ) + non_sub_opts = [ + joinedload(User.orders).joinedload(Item.keywords), + defaultload(User.orders).joinedload(Order.items), + ] + sess = Session() + self._assert_opts(sess.query(User), sub_opt, non_sub_opts) + + def test_not_implemented_fromload(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + + assert_raises_message( + NotImplementedError, + r"The options\(\) method is currently only supported " + "for 'unbound' loader options", + Load(User).joinedload(User.orders).options, + joinedload(Order.items), + ) + + def test_not_implemented_toload(self): + User, Address, Order, Item, SubItem = self.classes( + "User", "Address", "Order", "Item", "SubItem" + ) + + assert_raises_message( + NotImplementedError, + r"Only 'unbound' loader options may be used with the " + r"Load.options\(\) method", + joinedload(User.orders).options, + Load(Order).joinedload(Order.items), + ) + + class CacheKeyTest(PathTest, QueryTest): run_create_tables = False