From: Mike Bayer Date: Thu, 10 Sep 2020 15:56:34 +0000 (-0400) Subject: Add more docs for populate_existing(); link with_for_update X-Git-Tag: rel_1_3_20~25 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ccf8907d3675e4b183f4bcd98fb73ea72951e085;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Add more docs for populate_existing(); link with_for_update The populate_existing() method is actually changing to be an execution option, however it has almost no mention in the narrative docs so add docs in terms of the 1.x version first, including that we mention you almost definitely want to use this method if you are also using with_for_update(). Fixes: #5572 Fixes: #4774 Change-Id: Ieca916400622c1ffc1ae81204132a02a0983594c (cherry picked from commit e78c8f84958a031056684d7056aff0eab6ced226) --- diff --git a/doc/build/faq/sessions.rst b/doc/build/faq/sessions.rst index 835ce301ff..d9972cb86b 100644 --- a/doc/build/faq/sessions.rst +++ b/doc/build/faq/sessions.rst @@ -6,6 +6,7 @@ Sessions / Queries :class: faq :backlinks: none +.. _faq_session_identity: I'm re-loading data with my Session but it isn't seeing changes that I committed elsewhere ------------------------------------------------------------------------------------------ diff --git a/doc/build/orm/session_basics.rst b/doc/build/orm/session_basics.rst index 40715a7049..f04fce842f 100644 --- a/doc/build/orm/session_basics.rst +++ b/doc/build/orm/session_basics.rst @@ -732,6 +732,74 @@ required after a flush fails, even though the underlying transaction will have been rolled back already - this is so that the overall nesting pattern of so-called "subtransactions" is consistently maintained. +Expiring / Refreshing +--------------------- + +An important consideration that will often come up when using the +:class:`_orm.Session` is that of dealing with the state that is present on +objects that have been loaded from the database, in terms of keeping them +synchronized with the current state of the transaction. The SQLAlchemy +ORM is based around the concept of an :term:`identity map` such that when +an object is "loaded" from a SQL query, there will be a unique Python +object instance maintained corresponding to a particular database identity. +This means if we emit two separate queries, each for the same row, and get +a mapped object back, the two queries will have returned the same Python +object:: + + >>> u1 = session.query(User).filter(id=5).first() + >>> u2 = session.query(User).filter(id=5).first() + >>> u1 is u2 + True + +Following from this, when the ORM gets rows back from a query, it will +**skip the population of attributes** for an object that's already loaded. +The design assumption here is to assume a transaction that's perfectly +isolated, and then to the degree that the transaction isn't isolated, the +application can take steps on an as-needed basis to refresh objects +from the database transaction. The FAQ entry at :ref:`faq_session_identity` +discusses this concept in more detail. + +When an ORM mapped object is loaded into memory, there are three general +ways to refresh its contents with new data from the current transaction: + +* **the expire() method** - the :meth:`_orm.Session.expire` method will + erase the contents of selected or all attributes of an object, such that they + will be loaded from the database when they are next accessed, e.g. using + a :term:`lazy loading` pattern:: + + session.expire(u1) + u1.some_attribute # <-- lazy loads from the transaction + .. + +* **the refresh() method** - closely related is the :meth:`_orm.Session.refresh` + method, which does everything the :meth:`_orm.Session.expire` method does + but also emits one or more SQL queries immediately to actually refresh + the contents of the object:: + + session.refresh(u1) # <-- emits a SQL query + u1.some_attribute # <-- is refreshed from the transaction + + .. + +* **the populate_existing() method** - this method is actually on the + :class:`_orm.Query` object as :meth:`_orm.Query.populate_existing` + and indicates that it should return objects that are unconditionally + re-populated from their contents in the database:: + + u2 = session.query(User).populate_existing().filter(id=5).first() + + .. + +Further discussion on the refresh / expire concept can be found at +:ref:`session_expire`. + +.. seealso:: + + :ref:`session_expire` + + :ref:`faq_session_identity` + + .. _session_committing: Committing diff --git a/doc/build/orm/session_state_management.rst b/doc/build/orm/session_state_management.rst index 1a5168eb63..40e50cda15 100644 --- a/doc/build/orm/session_state_management.rst +++ b/doc/build/orm/session_state_management.rst @@ -506,6 +506,12 @@ attributes to be marked as expired:: # expire only attributes obj1.attr1, obj1.attr2 session.expire(obj1, ['attr1', 'attr2']) +The :meth:`.Session.expire_all` method allows us to essentially call +:meth:`.Session.expire` on all objects contained within the :class:`.Session` +at once:: + + session.expire_all() + The :meth:`~.Session.refresh` method has a similar interface, but instead of expiring, it emits an immediate SELECT for the object's row immediately:: @@ -519,11 +525,14 @@ be that of a column-mapped attribute:: # reload obj1.attr1, obj1.attr2 session.refresh(obj1, ['attr1', 'attr2']) -The :meth:`.Session.expire_all` method allows us to essentially call -:meth:`.Session.expire` on all objects contained within the :class:`.Session` -at once:: +An alternative method of refreshing which is often more flexible is to +use the :meth:`_orm.Query.populate_existing` method of :class:`_orm.Query`. +With this option, all of the ORM objects returned by the :class:`_orm.Query` +will be refreshed with the contents of what was loaded in the SELECT:: + + for user in session.query(User).populate_existing().filter(User.name.in_(['a', 'b', 'c'])): + print(user) # will be refreshed for those columns that came back from the query - session.expire_all() What Actually Loads ~~~~~~~~~~~~~~~~~~~ @@ -631,6 +640,10 @@ transactions, an understanding of the isolation behavior in effect is essential. :meth:`.Session.refresh` + :meth:`_orm.Query.populate_existing` - :class:`_orm.Query` method that refreshes + all matching objects in the identity map against the results of a + SELECT statement. + :term:`isolation` - glossary explanation of isolation which includes links to Wikipedia. diff --git a/lib/sqlalchemy/orm/query.py b/lib/sqlalchemy/orm/query.py index 8f4b3b0aca..7de2d939c8 100644 --- a/lib/sqlalchemy/orm/query.py +++ b/lib/sqlalchemy/orm/query.py @@ -1188,6 +1188,11 @@ class Query(object): after rollback or commit handles object state automatically. This method is not intended for general use. + .. seealso:: + + :ref:`session_expire` - in the ORM :class:`_orm.Session` + documentation + """ self._populate_existing = True @@ -1786,7 +1791,7 @@ class Query(object): E.g.:: - q = sess.query(User).with_for_update(nowait=True, of=User) + q = sess.query(User).populate_existing().with_for_update(nowait=True, of=User) The above query on a PostgreSQL backend will render like:: @@ -1796,13 +1801,24 @@ class Query(object): supersedes the :meth:`_query.Query.with_lockmode` method. + .. note:: It is generally a good idea to combine the use of the + :meth:`_orm.Query.populate_existing` method when using the + :meth:`_orm.Query.with_for_update` method. The purpose of + :meth:`_orm.Query.populate_existing` is to force all the data read + from the SELECT to be populated into the ORM objects returned, + even if these objects are already in the :term:`identity map`. + .. seealso:: :meth:`_expression.GenerativeSelect.with_for_update` - Core level method with full argument and behavioral description. - """ + :meth:`_orm.Query.populate_existing` - overwrites attributes of + objects already loaded in the identity map. + + """ # noqa: E501 + self._for_update_arg = LockmodeArg( read=read, nowait=nowait, diff --git a/lib/sqlalchemy/orm/session.py b/lib/sqlalchemy/orm/session.py index 1402d7c103..b3cc1334db 100644 --- a/lib/sqlalchemy/orm/session.py +++ b/lib/sqlalchemy/orm/session.py @@ -1678,6 +1678,8 @@ class Session(_SessionClassMethods): :meth:`.Session.expire_all` + :meth:`_orm.Query.populate_existing` + """ try: state = attributes.instance_state(instance) @@ -1748,6 +1750,8 @@ class Session(_SessionClassMethods): :meth:`.Session.refresh` + :meth:`_orm.Query.populate_existing` + """ for state in self.identity_map.all_states(): state._expire(state.dict, self.identity_map._modified) @@ -1786,6 +1790,8 @@ class Session(_SessionClassMethods): :meth:`.Session.refresh` + :meth:`_orm.Query.populate_existing` + """ try: state = attributes.instance_state(instance)