Mike Bayer [Tue, 9 Aug 2022 17:31:14 +0000 (13:31 -0400)]
validate mapped collection key is loaded
Changed the attribute access method used by
:func:`_orm.attribute_mapped_collection` and
:func:`_orm.column_mapped_collection`, used when populating the dictionary,
to assert that the data value on the object to be used as the dictionary
key is actually present, and is not instead using "None" due to the
attribute never being actually assigned. This is used to prevent a
mis-population of None for a key when assigning via a backref where the
"key" attribute on the object is not yet assigned.
As the failure mode here is a transitory condition that is not typically
persisted to the database, and is easy to produce via the constructor of
the class based on the order in which parameters are assigned, it is very
possible that many applications include this behavior already which is
silently passed over. To accommodate for applications where this error is
now raised, a new parameter
:paramref:`_orm.attribute_mapped_collection.ignore_unpopulated_attribute`
is also added to both :func:`_orm.attribute_mapped_collection` and
:func:`_orm.column_mapped_collection` that instead causes the erroneous
backref assignment to be skipped.
Mike Bayer [Tue, 16 Aug 2022 18:25:12 +0000 (14:25 -0400)]
refine transfer of cached ORM options for selectin, lazy
Fixed issue involving :func:`_orm.with_loader_criteria` where a closure
variable used as bound parameter value within the lambda would not carry
forward correctly into additional relationship loaders such as
:func:`_orm.selectinload` and :func:`_orm.lazyload` after the statement
were cached, using the stale originally-cached value instead.
This change brings forth a good refinement where we finally realize
we shouldn't be testing every ORM option with lots of switches, we
just let the option itself be given "here is your uncached version,
you are cached, tell us what to do!". the current decision is
that strategy loader options used the cached in all cases as they
always have, with_loader_criteria uses the uncached, because the
uncached will have been invoked with new closure state that we
definitely need. The only
edge that might not work is if with_loader_criteria referenced
an entity that is local to the query, namely a specific AliasedInsp,
however that's not a documented case for this. if we had to do that,
then we perhaps would introduce a more complex reconcilation
logic, and this would also give us the hook to do that.
Mike Bayer [Wed, 17 Aug 2022 17:06:51 +0000 (13:06 -0400)]
fill out all distinguising fields for AliasedInsp
Hardened the cache key strategy for the :func:`_orm.aliased` and
:func:`_orm.with_polymorphic` constructs. While no issue involving actual
statements being cached can easily be demonstrated (if at all), these two
constructs were not including enough of what makes them unique in their
cache keys for caching on the aliased construct alone to be accurate.
Introduced the type :class:`_postgresql.JSONPATH` that can be used
in cast expressions. This is required by some PostgreSQL dialects
when using functions such as ``jsonb_path_exists`` or
``jsonb_path_match`` that accept a ``jsonpath`` as input.
Mike Bayer [Wed, 10 Aug 2022 14:53:11 +0000 (10:53 -0400)]
Propagate key for collection events
Added new parameter :paramref:`_orm.AttributeEvents.include_key`, which
will include the dictionary or list key for operations such as
``__setitem__()`` (e.g. ``obj[key] = value``) and ``__delitem__()`` (e.g.
``del obj[key]``), using a new keyword parameter "key" or "keys", depending
on event, e.g. :paramref:`_orm.AttributeEvents.append.key`,
:paramref:`_orm.AttributeEvents.bulk_replace.keys`. This allows event
handlers to take into account the key that was passed to the operation and
is of particular importance for dictionary operations working with
:class:`_orm.MappedCollection`.
Mike Bayer [Fri, 12 Aug 2022 22:35:14 +0000 (18:35 -0400)]
restore Connectable and other changes for oslo.db
at this point oslo.db should be passing tests however
they seem to be importing "Connectable", which we had
temporarily removed, it's now a different kind of object
but restore it to the engine import space which seems
to be where they're pulling it from to see if oslo.db
can run some tests now.
RamonWill [Tue, 25 Aug 2020 00:17:19 +0000 (20:17 -0400)]
Add support for Partitioning and Sample pages on mysql
Add support for Partitioning and Sample pages on MySQL and MariaDB
reflected options.
The options are stored in the table dialect options dictionary, so
the following keyword need to be prefixed with ``mysql_`` or ``mariadb_``
depending on the backend.
Supported options are:
These options are also reflected when loading a table from database,
and will populate the table :attr:`_schema.Table.dialect_options`.
Pull request courtesy of Ramon Will.
Mike Bayer [Thu, 11 Aug 2022 14:20:49 +0000 (10:20 -0400)]
reorg bulk persistence into a separate module
This restores persistence.py to only functions that are used
by unitofwork.py, and all the "bulk" stuff gets its own
module bulk_persistence.py. Also fixes up the ORM context
class hierarchy for bulk.
This is all ahead of the ORM-insert changes coming in, so that
the later review can be about logic and not about reorganization.
Daniel Hall [Sat, 30 Jul 2022 19:12:20 +0000 (15:12 -0400)]
Support comments on MSSQL
Added support table and column comments on MSSQL when
creating a table. Added support for reflecting table comments.
Thanks to Daniel Hall for the help in this pull request.
Mike Bayer [Wed, 10 Aug 2022 16:40:17 +0000 (12:40 -0400)]
doc fixes
* fixed erroneous use of mapped_column() in m2m relationship Table
* Fill in full imports for some relationship examples that had
partial imports; examples that have no imports, leave empty for now
* converted joined/single inh mappings to annotated style
* We have a problem with @declared_attr in that the error message
is wrong if the mapped_column() returned doesnt have a type, and/or
mapped_column() with @declared_attr doesnt use the annotation
* fix thing where sphinx with undoc-members global setting seems to
no longer tolerate ":attribute:" entries in autodoc classes, which
is fine we can document the annotations now
* Fix mapper params in inheritance to be on Mapper
* add missing changelog file for instances remove
Mike Bayer [Tue, 9 Aug 2022 13:50:57 +0000 (09:50 -0400)]
improve names for datatypes section
Continuing along #8362, if we look at the document here without
"this is the same text I've read for 14 years", we begin to see
that the title "Column and Data types" makes no sense at all,
is there a "column type" and a "Data type"? I guess what I was
thinking at that time is that a type can be placed on a
Column, or it can be used whenever you have, you know, "data".
The phrase "SQL expression" wasn't discovered yet.
"SQL Datatype" is not spectacular but at least it's one term.
the new intro then is focused on the hierarchy layout so let's
name it that.
not amazing, but better than the duplicate name that made
no sense before.
Mike Bayer [Fri, 5 Aug 2022 21:25:05 +0000 (17:25 -0400)]
deep compare CTEs before considering them conflicting
Fixed issue where referencing a CTE multiple times in conjunction with a
polymorphic SELECT could result in multiple "clones" of the same CTE being
constructed, which would then trigger these two CTEs as duplicates. To
resolve, the two CTEs are deep-compared when this occurs to ensure that
they are equivalent, then are treated as equivalent.
Mike Bayer [Wed, 3 Aug 2022 20:55:23 +0000 (16:55 -0400)]
translate joined inheritance cols in UPDATE/DELETE
Fixed issue in ORM enabled UPDATE when the statement is created against a
joined-inheritance subclass, updating only local table columns, where the
"fetch" synchronization strategy would not render the correct RETURNING
clause for databases that use RETURNING for fetch synchronization.
Also adjusts the strategy used for RETURNING in UPDATE FROM and
DELETE FROM statements.
Also fixes MariaDB which does not support RETURNING with
DELETE..USING. this was not caught in tests because
"fetch" strategy wasn't tested. so also adjust the ORMDMLState
classes to look for "extra froms" first before adding
RETURNING, add new parameters to interfaces for
"update_returning_multitable" and "delete_returning_multitable".
A new execution option is_delete_using=True, described in the
changelog message, is added to allow the ORM to know up front
if a certain statement should have a SELECT up front
for "fetch" strategy.
Mike Bayer [Thu, 4 Aug 2022 14:27:59 +0000 (10:27 -0400)]
implement PG ranges/multiranges agnostically
Ranges now work using a new Range object,
multiranges as lists of Range objects (this is what
asyncpg does. not sure why psycopg has a "Multirange"
type).
psycopg, psycopg2, and asyncpg are currently supported.
It's not clear how to make ranges work with pg8000, likely
needs string conversion; this is straightforward with the
new archicture and can be added later.
Mike Bayer [Tue, 2 Aug 2022 20:18:18 +0000 (16:18 -0400)]
include column.default, column.onupdate in eager_defaults
Fixed bug in the behavior of the :paramref:`_orm.Mapper.eager_defaults`
parameter such that client-side SQL default or onupdate expressions in the
table definition alone will trigger a fetch operation using RETURNING or
SELECT when the ORM emits an INSERT or UPDATE for the row. Previously, only
server side defaults established as part of table DDL and/or server-side
onupdate expressions would trigger this fetch, even though client-side SQL
expressions would be included when the fetch was rendered.
zeeeeb [Tue, 28 Jun 2022 23:05:08 +0000 (19:05 -0400)]
fixes: #7156 - Adds support for PostgreSQL MultiRange type
This adds functionality for PostgreSQL MultiRange type, as discussed in Issue #7156.
As far as I can tell, only psycopg provides a [Multirange adaptation](https://www.psycopg.org/psycopg3/docs/basic/pgtypes.html#multirange-adaptation). Psycopg2 only supports a [Range adaptation/data type](https://www.psycopg.org/psycopg3/docs/basic/pgtypes.html#multirange-adaptation).
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 3 Aug 2022 22:59:05 +0000 (18:59 -0400)]
deprecate Query.instances()
this method no longer does the thing that it was originally
intended to do, which is to get ORM results from arbitrary
result sets. Modern patterns should supersede the
use of this construct.
Mike Bayer [Wed, 3 Aug 2022 18:08:32 +0000 (14:08 -0400)]
ensure RETURNING renders in stringify w/ no server version
just in my own testing, if I say insert().return_defaults()
and stringify, I should see it, so make sure all the dialects
default to "insert_returning" etc. , with downgrade on
server version check.
Mike Bayer [Wed, 3 Aug 2022 16:08:54 +0000 (12:08 -0400)]
send in the dragons on async_scoped_session
make it clear that async_scoped_session.remove() must
be called, else memory will build up. Generally
discourage the whole pattern as well, as this is a
"framework" pattern and we don't really want to be supporting
frameworks. Also indicate that scopefunc must be idempotent
and lightweight.
Mike Bayer [Tue, 2 Aug 2022 18:51:49 +0000 (14:51 -0400)]
reword yield_per a bit more
I'm still not satisified with this section as it is still
too wordy and dense, but at least let's put a better description
of what yield_per actually is and why one might use it at the top.
Fixed issue where the SQL Server dialect's query for the current isolation
level would fail on Azure Synapse Analytics, due to the way in which this
database handles transaction rollbacks after an error has occurred. The
initial query has been modified to no longer rely upon catching an error
when attempting to detect the appropriate system view. Additionally, to
better support this database's very specific "rollback" behavior,
implemented new parameter ``ignore_no_transaction_on_rollback`` indicating
that a rollback should ignore Azure Synapse error 'No corresponding
transaction found. (111214)', which is raised if no transaction is present
in conflict with the Python DBAPI.
Mike Bayer [Tue, 2 Aug 2022 15:34:35 +0000 (11:34 -0400)]
fix up SAVEPOINT docs
these contained a factual error that the entire session is
expired, which is no longer the case (I can't find exactly
when this was changed). Additionally, added a PostgreSQL
specific example w/ IntegrityError as this is the most
common case for this. Tried to tighten up other language
and make it as clear as possible.
Added new parameter to SQLite for reflection methods called
``sqlite_include_internal=True``; when omitted, local tables that start
with the prefix ``sqlite_``, which per SQLite documentation are noted as
"internal schema" tables such as the ``sqlite_sequence`` table generated to
support "AUTOINCREMENT" columns, will not be included in reflection methods
that return lists of local objects. This prevents issues for example when
using Alembic autogenerate, which previously would consider these
SQLite-generated tables as being remove from the model.
Mike Bayer [Wed, 27 Jul 2022 15:36:57 +0000 (11:36 -0400)]
implement tuple-slices from .c collections
Added new syntax to the ``.c`` collection on all :class:`.FromClause`
objects allowing tuples of keys to be passed to ``__getitem__()``, along
with support for ``select()`` handling of ``.c`` collections directly,
allowing the syntax ``select(table.c['a', 'b', 'c'])`` to be possible. The
sub-collection returned is itself a :class:`.ColumnCollection` which is
also directly consumable by :func:`_sql.select` and similar now.
Mike Bayer [Mon, 1 Aug 2022 14:29:13 +0000 (10:29 -0400)]
repair psycopg2 (and psycopg) multiple hosts format
Fixed issue in psycopg2 dialect where the "multiple hosts" feature
implemented for :ticket:`4392`, where multiple ``host:port`` pairs could be
passed in the query string as
``?host=host1:port1&host=host2:port2&host=host3:port3`` was not implemented
correctly, as it did not propagate the "port" parameter appropriately.
Connections that didn't use a different "port" likely worked without issue,
and connections that had "port" for some of the entries may have
incorrectly passed on that hostname. The format is now corrected to pass
hosts/ports appropriately.
As part of this change, maintained support for another multihost style that
worked unintentionally, which is comma-separated
``?host=h1,h2,h3&port=p1,p2,p3``. This format is more consistent with
libpq's query-string format, whereas the previous format is inspired by a
different aspect of libpq's URI format but is not quite the same thing.
If the two styles are mixed together, an error is raised as this is
ambiguous.
The PostgreSQL dialect now supports reflection of expression based indexes.
The reflection is supported both when using
:meth:`_engine.Inspector.get_indexes` and when reflecting a
:class:`_schema.Table` using :paramref:`_schema.Table.autoload_with`.
Thanks to immerrr and Aidan Kane for the help on this ticket.
Mike Bayer [Thu, 7 Jul 2022 16:07:39 +0000 (12:07 -0400)]
Use FETCH FIRST N ROWS / OFFSET for Oracle LIMIT/OFFSET
Oracle will now use FETCH FIRST N ROWS / OFFSET syntax for limit/offset
support by default for Oracle 12c and above. This syntax was already
available when :meth:`_sql.Select.fetch` were used directly, it's now
implied for :meth:`_sql.Select.limit` and :meth:`_sql.Select.offset` as
well.
I'm currently setting this up so that the new syntax renders
in Oracle using POSTCOMPILE binds. I really have no indication
if Oracle's SQL optimizer would be better with params
here, so that it can cache the SQL plan, or if it expects
hardcoded numbers for these. Since we had reports that the previous
ROWNUM thing really needed hardcoded ints, let's guess
for now that hardcoded ints would be preferable. it can be turned
off with a single boolean if users report that they'd prefer
real bound values.
Mike Bayer [Thu, 23 Jun 2022 20:43:49 +0000 (16:43 -0400)]
add an extra load for non-new but unloaded
Made an improvement to the "deferred" / "load_only" set of strategy options
where if a certain object is loaded from two different logical paths within
one query, attributes that have been configured by at least one of the
options to be populated will be populated in all cases, even if other load
paths for that same object did not set this option. previously, it was
based on randomness as to which "path" addressed the object first.
Mike Bayer [Tue, 19 Jul 2022 17:03:51 +0000 (13:03 -0400)]
fixes for mypy 0.971
things that were passing with 0.961 need adjustment.
it seems mypy has become very pedantic about the difference
between importing from a module vs. accessing members of that
module as instance variables, so adjust the preloaded
typing block to be explicitly instance variables, since that's
how the accessor works in any case.
Mike Bayer [Tue, 19 Jul 2022 14:50:05 +0000 (10:50 -0400)]
check for TypeDecorator when handling getitem
Fixed issue where :class:`.TypeDecorator` would not correctly proxy the
``__getitem__()`` operator when decorating the :class:`.ARRAY` datatype,
without explicit workarounds.
Mike Bayer [Mon, 18 Jul 2022 16:52:28 +0000 (12:52 -0400)]
implement executemany RETURNING for Oracle
this works straight out of the box as we can expand
upon what we did for #6245 to also receive for multiple
statements. Oracle "fast ORM insert" then is basically done.
Federico Caselli [Fri, 17 Jun 2022 21:12:39 +0000 (23:12 +0200)]
add shield() in aexit
Added ``asyncio.shield()`` to the connection and session release process
specifically within the ``__aexit__()`` context manager exit, when using
:class:`.AsyncConnection` or :class:`.AsyncSession` as a context manager
that releases the object when the context manager is complete. This appears
to help with task cancellation when using alternate concurrency libraries
such as ``anyio``, ``uvloop`` that otherwise don't provide an async context
for the connection pool to release the connection properly during task
cancellation.
Mike Bayer [Mon, 18 Jul 2022 12:48:55 +0000 (08:48 -0400)]
add contextmanager typing, open run_sync typing
was missing AsyncConnection type for the async
context manager.
fixing that revealed that _SyncConnectionCallable
and _SyncSessionCallable protocols are infeasible because
the given callable can have a lot of different signatures
that are compatible.
Mike Bayer [Sun, 17 Jul 2022 15:32:27 +0000 (11:32 -0400)]
use concat() directly for contains, startswith, endswith
Adjusted the SQL compilation for string containment functions
``.contains()``, ``.startswith()``, ``.endswith()`` to force the use of the
string concatenation operator, rather than relying upon the overload of the
addition operator, so that non-standard use of these operators with for
example bytestrings still produces string concatenation operators.
To accommodate this, needed to add a new _rconcat operator function,
which is private, as well as a fallback in concat_op() that works
similarly to Python builtin ops.
Mike Bayer [Sat, 16 Jul 2022 20:19:15 +0000 (16:19 -0400)]
implement column._merge()
this takes the user-defined args of one Column and merges
them into the not-user-defined args of another Column.
Implemented within the pep-593 column transfer operation
to begin to make this new feature more robust.
work may still be needed for constraints etc. but
in theory everything from the left side annotated column
should take effect for the right side if not otherwise
specified on the right.
Mike Bayer [Fri, 15 Jul 2022 16:25:22 +0000 (12:25 -0400)]
make anno-only Mapped[] column available for mixins
Documentation is relying on the recently improved
behavior of produce_column_copies() to make sure everything is
available on cls for a declared_attr. therefore for anno-only
attribute, we also need to generate the mapped_column() up front
before scan is called.
noticed in pylance, allow @declared_attr to recognize
@classmethod also which allows letting typing tools know
something is explicitly a classmethod
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
