Mike Bayer [Thu, 20 Oct 2022 16:05:33 +0000 (12:05 -0400)]
collect annotation earlier for mapped_column present
Fixed issue with new dataclass mapping feature where arguments passed to
the dataclasses API could sometimes be mis-ordered when dealing with mixins
that override :func:`_orm.mapped_column` declarations, leading to
initializer problems.
the change made here is specific to the test case given which regards
mapped_column() most specifically. cases that involve relationship()
etc. are not tested here, however mapped_column() is the only attribute
that's implicit without an instance given on the right side, and is also
most common for mixins. not clear if there are more issues in this
area, however it appears that we need only adjust the order in which we
accommodate grabbing the annotations in order to affect how dataclasses
sees the class; that is, we have control over ``__annotations__`` here
so dont have to worry about ``cls.__dict__``.
Mike Bayer [Wed, 19 Oct 2022 13:41:44 +0000 (09:41 -0400)]
more many-to-one typing
since we are typing centric, note this configuration
as we have just supported in #8668.
Note also I am just taking "backref" out of the basic
version of the docs here totally, this doc is already
a lot to read / take in without making it even more
confusing; backref still has an entirely dedicated
docs page which can have all the additional behaviors
of backref() described.
Additionally, get other "optional" forms to work including
``cls | None`` and ``Union[cls, None]``.
Mike Bayer [Wed, 19 Oct 2022 13:17:03 +0000 (09:17 -0400)]
rename tables to _table for basic relationships
the names "parent" / "child" are confusing for new users
in that they are used for table names as well as
"back_populates='parent'", use a disambiguated name.
In this change, there's now overlap between the
variable named "association_table" and the table name
"association_table". not sure of a better naming system.
Mike Bayer [Wed, 19 Oct 2022 01:01:05 +0000 (21:01 -0400)]
de-optionalize union types to support Optional for m2o
Fixed bug in new ORM typed declarative mappings where we did not include
the ability to use ``Optional[]`` in the type annotation for a many-to-one
relationship, even though this is common.
Mike Bayer [Tue, 18 Oct 2022 17:25:06 +0000 (13:25 -0400)]
call super().__init_subclass__(); support GenericAlias
Improved the :class:`.DeclarativeBase` class so that when combined with
other mixins like :class:`.MappedAsDataclass`, the order of the classes may
be in either order.
Added support for mapped classes that are also ``Generic`` subclasses,
to be specified as a ``GenericAlias`` object (e.g. ``MyClass[str]``)
within statements and calls to :func:`_sa.inspect`.
Mike Bayer [Tue, 18 Oct 2022 13:44:37 +0000 (09:44 -0400)]
further qualify pyodbc setinputsizes types for long stirngs
Fixed regression caused by SQL Server pyodbc change :ticket:`8177` where we
now use ``setinputsizes()`` by default; for VARCHAR, this fails if the
character size is greater than 4000 (or 2000, depending on data) characters
as the incoming datatype is NVARCHAR, which has a limit of 4000 characters,
despite the fact that VARCHAR can handle unlimited characters. Additional
pyodbc-specific typing information is now passed to ``setinputsizes()``
when the datatype's size is > 2000 characters. The change is also applied
to the :class:`.JSON` type which was also impacted by this issue for large
JSON serializations.
The :class:`.Sequence` construct restores itself to the DDL behavior it
had prior to the 1.4 series, where creating a :class:`.Sequence` with
no additional arguments will emit a simple ``CREATE SEQUENCE`` instruction
**without** any additional parameters for "start value". For most backends,
this is how things worked previously in any case; **however**, for
MS SQL Server, the default value on this database is
``-2**63``; to prevent this generally impractical default
from taking effect on SQL Server, the :paramref:`.Sequence.start` parameter
should be provided. As usage of :class:`.Sequence` is unusual
for SQL Server which for many years has standardized on ``IDENTITY``,
it is hoped that this change has minimal impact.
Mike Bayer [Mon, 17 Oct 2022 19:09:01 +0000 (15:09 -0400)]
update SEQUENCE docs ahead of default change
for backport to 1.4 as well, remove references to
Firebird, and also revert "associate Sequence with MetaData"
step as this is not needed usually, just note that schema
is not shared. encourage users to use IDENTITY instead.
Mike Bayer [Mon, 17 Oct 2022 15:53:07 +0000 (11:53 -0400)]
simplify unmapped col eval fallback
Removed the warning that emits when using ORM-enabled update/delete
regarding evaluation of columns by name, first added in :ticket:`4073`;
this warning actually covers up a scenario that otherwise could populate
the wrong Python value for an ORM mapped attribute depending on what the
actual column is, so this deprecated case is removed. In 2.0, ORM enabled
update/delete uses "auto" for "synchronize_session", which should do the
right thing automatically for any given UPDATE expression.
Mike Bayer [Sun, 16 Oct 2022 22:30:44 +0000 (18:30 -0400)]
repair type qualify in _ServerDefaultType; other pyright tweaks
as we haven't done full pylance / pyright strict typing internally,
some of the things pyright reports on specifically will leak
out into user code, such as this mapped_column() issue.
So we will have to look more closely at pyright strict
mode going forward for the release.
Fixed typing issue where pylance strict mode would report "partially
unknown" datatype for the :func:`_orm.mapped_column` construct.
Also repaired a trailing comma and pyright complaining about overloads
for orm.composite.
Mike Bayer [Sat, 15 Oct 2022 19:20:21 +0000 (15:20 -0400)]
accommodate arbitrary embedded params in insertmanyvalues
Fixed bug in new "insertmanyvalues" feature where INSERT that included a
subquery with :func:`_sql.bindparam` inside of it would fail to render
correctly in "insertmanyvalues" format. This affected psycopg2 most
directly as "insertmanyvalues" is used unconditionally with this driver.
Mike Bayer [Sat, 15 Oct 2022 15:12:25 +0000 (11:12 -0400)]
disable isort in pyproject.toml
disable isort, for IDEs that just default isort to be turned on, e.g. vscode.
we use flake8-import-order for import sorting, using zimports to actually
reformat code. isort is nicer in many ways but doesn't have our
"import *" fixer and also is not 100% compatible with flake8-import-order.
Mike Bayer [Tue, 11 Oct 2022 21:01:43 +0000 (17:01 -0400)]
implement autobegin=False option
Added new parameter :paramref:`_orm.Session.autobegin`, which when set to
``False`` will prevent the :class:`_orm.Session` from beginning a
transaction implicitly. The :meth:`_orm.Session.begin` method must be
called explicitly first in order to proceed with operations, otherwise an
error is raised whenever any operation would otherwise have begun
automatically. This option can be used to create a "safe"
:class:`_orm.Session` that won't implicitly start new transactions.
As part of this change, also added a new status variable
:class:`_orm.SessionTransaction.origin` which may be useful for event
handling code to be aware of the origin of a particular
:class:`_orm.SessionTransaction`.
Mike Bayer [Mon, 10 Oct 2022 18:03:04 +0000 (14:03 -0400)]
warn for no polymorphic identity w/ poly hierarchy
A warning is emitted when attempting to configure a mapped class within an
inheritance hierarchy where the mapper is not given any polymorphic
identity, however there is a polymorphic discriminator column assigned.
Such classes should be abstract if they never intend to load directly.
Mike Bayer [Tue, 11 Oct 2022 18:22:40 +0000 (14:22 -0400)]
enable check same thread for aiosqlite
to do this we have to invent our own isolation level
setter based on their current internals. however
now we can ensure thread-safe access. we are trying
to resolve an issue where test suite on CI seems to fail
around the same time each time.
Mike Bayer [Fri, 7 Oct 2022 18:03:16 +0000 (14:03 -0400)]
rename MappedCollection and related
For consistency with the prominent ORM concept :class:`_orm.Mapped`, the
names of the dictionary-oriented collections,
:func:`_orm.attribute_mapped_collection`,
:func:`_orm.column_mapped_collection`, and :class:`_orm.MappedCollection`,
are changed to :func:`_orm.attribute_keyed_dict`,
:func:`_orm.column_keyed_dict` and :class:`_orm.KeyFuncDict`, using the
phrase "dict" to minimize any confusion against the term "mapped". The old
names will remain indefinitely with no schedule for removal.
Docs here are also updated for typing as we can type
these collections as ``Mapped[dict[str, cls]]``, don't need
KeyFuncDict / MappedCollection for these
Mike Bayer [Fri, 7 Oct 2022 15:25:08 +0000 (11:25 -0400)]
dont mutate bind_arguments incoming dictionary
The :paramref:`_orm.Session.execute.bind_arguments` dictionary is no longer
mutated when passed to :meth:`_orm.Session.execute` and similar; instead,
it's copied to an internal dictionary for state changes. Among other
things, this fixes and issue where the "clause" passed to the
:meth:`_orm.Session.get_bind` method would be incorrectly referring to the
:class:`_sql.Select` construct used for the "fetch" synchronization
strategy, when the actual query being emitted was a :class:`_dml.Delete` or
:class:`_dml.Update`. This would interfere with recipes for "routing
sessions".
Mike Bayer [Mon, 26 Sep 2022 18:38:44 +0000 (14:38 -0400)]
implement write-only colletions, typing for dynamic
For 2.0, we provide a truly "larger than memory collection"
implementation, a write-only collection that will never
under any circumstances implicitly load the entire
collection, even during flush.
This is essentially a much more "strict" version
of the "dynamic" loader, which in fact has a lot of
scenarios that it loads the full backing collection
into memory, mostly defeating its purpose.
Typing constructs are added that support
both the new feature WriteOnlyMapping as well as the
legacy feature DynamicMapping. These have been
integrated with "annotion based mapping" so that
relationship() uses these annotations to configure
the loader strategy as well.
additional changes:
* the docs triggered a conflict in hybrid's
"transformers" section, this section is hard-coded
to Query using a pattern that doesnt seem to have
any use and isn't part of the current select()
interface, so just removed this section
* As the docs for WriteOnlyMapping are very long,
collections.rst is broken up into two pages now.
Mike Bayer [Thu, 29 Sep 2022 16:56:23 +0000 (12:56 -0400)]
reorganize Mapped[] super outside of MapperProperty
We made all the MapperProperty classes a subclass
of Mapped[] to allow declarative mappings to name
Mapped[] on the left side. this was cheating a bit because
MapperProperty is not actually a descriptor, and the mapping
process replaces the object with InstrumentedAttribute at
mapping time, which is the actual Mapped[] descriptor.
But now in I6929f3da6e441cad92285e7309030a9bac4e429d we
are considering making the "cheating" a little more extensive
by putting DynamicMapped / WriteOnlyMapped in Relationship's
hierarchy, which need a flat out "type: ignore" to work.
Instead of pushing more cheats into the core classes, move
out the "Declarative"-facing versions of these classes to be
typing only: Relationship, Composite, Synonym, and MappedSQLExpression
added for ColumnProperty. Keep the internals expressed on the
old names, RelationshipProperty, CompositeProperty, SynonymProperty,
ColumnProprerty, which will remain "pure" with fully correct typing.
then have the typing only endpoints be where the "cheating"
and "type: ignores" have to happen, so that these are more or less
slightly better forms of "Any".
Make if_exists and if_not_exists flags on ddl statements match compiler
Added ``if_exists`` and ``if_not_exists`` parameters for all "Create" /
"Drop" constructs including :class:`.CreateSequence`,
:class:`.DropSequence`, :class:`.CreateIndex`, :class:`.DropIndex`, etc.
allowing generic "IF EXISTS" / "IF NOT EXISTS" phrases to be rendered
within DDL. Pull request courtesy Jesse Bakker.
John Bodley [Fri, 30 Sep 2022 01:58:58 +0000 (21:58 -0400)]
adjust MySQL view reflection for non-standard MySQL variants
Adjusted the regular expression used to match "CREATE VIEW" when
testing for views to work more flexibly, no longer requiring the
special keyword "ALGORITHM" in the middle, which was intended to be
optional but was not working correctly. The change allows view reflection
to work more completely on MySQL-compatible variants such as StarRocks.
Pull request courtesy John Bodley.
Mike Bayer [Mon, 3 Oct 2022 01:22:11 +0000 (21:22 -0400)]
the future is here
the autodoc for the "future" Engine / Connection were removed,
so all these links weren't working. Replace all _future
for these with _engine. There was just one _future pointing
to select, changed that separately.
Mike Bayer [Sun, 2 Oct 2022 02:24:38 +0000 (22:24 -0400)]
add disable doctest tag for autodoc test suite
ahead of trying to get everything formatted, some more
flexibility so that we can use doctest for all
python + sql code, while still being able to tell the
test suite to not run doctests on a sample. All of the
"non-console python with SQL" in the docs is because I was
showing an example that I didn't want tested.
Mike Bayer [Fri, 30 Sep 2022 20:58:26 +0000 (16:58 -0400)]
fix "entry points"
the formatting here wasn't round tripping without manually
adjusting the indentation. this suggests that "check" mode
is not doing the exact same thing as "write". maybe
file.write_text() is changing something.
Added script to format code in the rst documentation using black.
This is also added to the lint tox job to ensure that the code
in the docs is properly formatted.
Mike Bayer [Tue, 27 Sep 2022 14:23:28 +0000 (10:23 -0400)]
correct test criteria for mssql
this test was failing on mssql b.c. mssql
reports false for supports_default_metavalue. however
insertmanyvalues is still used in this case, and the
assert SQL is against the default dialect in any case.
Mike Bayer [Mon, 26 Sep 2022 12:54:59 +0000 (08:54 -0400)]
adjust tests for sqlites w/o returning
the sqlite builds on github actions seem to be very
inconsistent about versions and many don't support
RETURNING. ensure any tests that depend on RETURNING present
are marked as such.
Mike Bayer [Thu, 18 Aug 2022 17:56:50 +0000 (13:56 -0400)]
New ORM Query Guide featuring DML support
reviewers: these docs publish periodically at:
https://docs.sqlalchemy.org/en/gerrit/4042/orm/queryguide/index.html
See the "last generated" timestamp near the bottom of the
page to ensure the latest version is up
Change includes some other adjustments:
* small typing fixes for end-user benefit
* removal of a bunch of old examples for patterns that nobody
uses or aren't really what we promote now
* modernization of some examples, including inheritance
Mike Bayer [Sun, 25 Sep 2022 18:56:22 +0000 (14:56 -0400)]
warn for local-only column in remote side
A warning is emitted in ORM configurations when an explicit
:func:`_orm.remote` annotation is applied to columns that are local to the
immediate mapped class, when the referenced class does not include any of
the same table columns. Ideally this would raise an error at some point as
it's not correct from a mapping point of view.
Mike Bayer [Sun, 7 Aug 2022 16:14:19 +0000 (12:14 -0400)]
ORM bulk insert via execute
* ORM Insert now includes "bulk" mode that will run
essentially the same process as session.bulk_insert_mappings;
interprets the given list of values as ORM attributes for
key names
* ORM UPDATE has a similar feature, without RETURNING support,
for session.bulk_update_mappings
* Added support for upserts to do RETURNING ORM objects as well
* ORM UPDATE/DELETE with list of parameters + WHERE criteria
is a not implemented; use connection
* ORM UPDATE/DELETE defaults to "auto" synchronize_session;
use fetch if RETURNING is present, evaluate if not, as
"fetch" is much more efficient (no expired object SELECT problem)
and less error prone if RETURNING is available
UPDATE: howver this is inefficient! please continue to
use evaluate for simple cases, auto can move to fetch
if criteria not evaluable
* "Evaluate" criteria will now not preemptively
unexpire and SELECT attributes that were individually
expired. Instead, if evaluation of the criteria indicates that
the necessary attrs were expired, we expire the object
completely (delete) or expire the SET attrs unconditionally
(update). This keeps the object in the same unloaded state
where it will refresh those attrs on the next pass, for
this generally unusual case. (originally #5664)
* Core change! update/delete rowcount comes from len(rows)
if RETURNING was used. SQLite at least otherwise did not
support this. adjusted test_rowcount accordingly
* ORM DELETE with a list of parameters at all is also a not
implemented as this would imply "bulk", and there is no
bulk_delete_mappings (could be, but we dont have that)
* ORM insert().values() with single or multi-values translates
key names based on ORM attribute names
* ORM returning() implemented for insert, update, delete;
explcit returning clauses now interpret rows in an ORM
context, with support for qualifying loader options as well
* session.bulk_insert_mappings() assigns polymorphic identity
if not set.
* explicit RETURNING + synchronize_session='fetch' is now
supported with UPDATE and DELETE.
* expanded return_defaults() to work with DELETE also.
* added support for composite attributes to be present
in the dictionaries used by bulk_insert_mappings and
bulk_update_mappings, which is also the new ORM bulk
insert/update feature, that will expand the composite
values into their individual mapped attributes the way they'd
be on a mapped instance.
* bulk UPDATE supports "synchronize_session=evaluate", is the
default. this does not apply to session.bulk_update_mappings,
just the new version
* both bulk UPDATE and bulk INSERT, the latter with or without
RETURNING, support *heterogenous* parameter sets.
session.bulk_insert/update_mappings did this, so this feature
is maintained. now cursor result can be both horizontally
and vertically spliced :)
This is now a long story with a lot of options, which in
itself is a problem to be able to document all of this
in some way that makes sense. raising exceptions for
use cases we haven't supported is pretty important here
too, the tradition of letting unsupported things just not work
is likely not a good idea at this point, though there
are still many cases that aren't easily avoidable
Mike Bayer [Mon, 18 Jul 2022 19:08:37 +0000 (15:08 -0400)]
implement batched INSERT..VALUES () () for executemany
the feature is enabled for all built in backends
when RETURNING is used,
except for Oracle that doesn't need it, and on
psycopg2 and mssql+pyodbc it is used for all INSERT statements,
not just those that use RETURNING.
third party dialects would need to opt in to the new feature
by setting use_insertmanyvalues to True.
Also adds dialect-level guards against using returning
with executemany where we dont have an implementation to
suit it. execute single w/ returning still defers to the
server without us checking.
Tighten password security by removing `URL.__str__`
For improved security, the :class:`_url.URL` object will now use password
obfuscation by default when ``str(url)`` is called. To stringify a URL with
cleartext password, the :meth:`_url.URL.render_as_string` may be used,
passing the :paramref:`_url.URL.render_as_string.hide_password` parameter
as ``False``. Thanks to our contributors for this pull request.
Mike Bayer [Fri, 23 Sep 2022 19:17:57 +0000 (15:17 -0400)]
remove should_nest behavior for contains_eager()
Fixed regression for 1.4 in :func:`_orm.contains_eager` where the "wrap in
subquery" logic of :func:`_orm.joinedload` would be inadvertently triggered
for use of the :func:`_orm.contains_eager` function with similar statements
(e.g. those that use ``distinct()``, ``limit()`` or ``offset()``). This is
not appropriate for :func:`_orm.contains_eager` which has always had the
contract that the user-defined SQL statement is unmodified with the
exception of adding the appropriate columns.
Also includes an adjustment to the assertion in Label._make_proxy()
which was there to prevent a fixed label name from being anonymized;
if the label is already anonymous, the change should proceed.
This logic was being hit before the contains_eager behavior was
adjusted. With the adjustment, this code is not used.
Mike Bayer [Fri, 23 Sep 2022 17:20:16 +0000 (13:20 -0400)]
update whatsnew/ migration
The migration20 guide had a lot of "whats new" stuff in it,
which is moved to the whatsnew20 document.
Also moved most of the content regarding the reflection rewrite
into the whatsnew20 document and added some performance comparisons
to 1.4.
The sectioning for these two documents, as has happened for some
other documents I'm working on, has been modified to only use the
double-equals operator at the top. This is strictly because my
vscode seems to be able to show me an .rst outline in the side panel
if I do it this way, and otherwise it's blank.
Mike Bayer [Tue, 20 Sep 2022 16:21:14 +0000 (12:21 -0400)]
auto-cast PG range types
Range type handling has been enhanced so that it automatically
renders type casts, so that in-place round trips for statements that don't
provide the database with any context don't require the :func:`_sql.cast`
construct to be explicit for the database to know the desired type.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
