Yossi [Mon, 1 Dec 2025 17:06:12 +0000 (12:06 -0500)]
[typing] Fix type error when passing Mapped columns to values()
This adjusts the _DMLOnlyColumnArgument type to be a more
focused _OnlyColumnArgument type where we also add a more tightly
focused coercion, while still allowing ORM attributes to be used
as arguments.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Closes: #13012
Pull-request: https://github.com/sqlalchemy/sqlalchemy/pull/13012
Pull-request-sha: 5ebb402c686abf1090e5b83e3489dfca4908efdf
Mike Bayer [Thu, 27 Nov 2025 06:22:38 +0000 (01:22 -0500)]
drop a 400 ton anvil on oracle 23c
this DB is extremely erratic in being able to connect. Add
a brute force connection retrier to all engines everywhere
(which for oracledb we can fortunately use their built-in feature
that also works).
This actually works and I can see it pausing under load, reconnecting,
and succeeding. the problem is that absolutely every engine everywhere
needs this routine otherwise an engine without a retrier in it will
crash. That then necessitates digging into testing_engine(),
making sure testing_engine() is used everywhere an engine that's going
to connect is used, then dealing with the fallout from that.
We also simplify some older workarounds for cx_oracle and
hack into config/provision to make oracledb seem like the primary
DBAPI for most tests.
testing_engine has been completely overhauled, making use of a new
post_configure_testing_engine() hook which moves and refines
the SQLite pool sharing and savepoint logic all into sqlite/provision.py
and also allows for cx_oracle to apply a retry event handler.
Mike Bayer [Wed, 26 Nov 2025 06:36:57 +0000 (01:36 -0500)]
stop using MyISAM; more oracle struggles
getting some fails on mariadb12 and likely 11 which appear to
be related to calling in MyISAM, which is not used in
modern mysql/mariadb. see if we can just remove this whole
thing and rely on default engines for mariadb/mysql.
this change also removes the "ignore errors" part of the
run deletes for the TablesTest fixture, which was resulting
in compound failures, and apparently a lot of tests were relying
on it skipping nonexistent tables. rather than check for that
we should just improve the tests and probably increase use of
pytest style fixtures overall.
this change also identifies and fixes that memusage_w_backend
tests were running for all backends with a tag like
py314_mysql_backendonly; the memusage tests should basically
never be run as part of the whole suite since they are entirely
unreliable within a full scale test run.
dialect suite tests are also further broken out into those where
every driver should be exercised (i.e. __backend__, for tests that
test datatypes going out and coming back from the database as well
as identity/autoincrement kinds of tests) vs. those where only
one driver per backend is needed (i.e. __sparse_driver_backend__,
for tests like reflection, DDL, CTEs, etc.).
we are also trying to get a --low-connections option that actually
works. changed this so that the testing reaper aggressively disposes
the "global" engines (one per backend / driver) after test classes
are done and before any testing_engine() call. This definitely
works, however some monitoring with PG shows the number of connections
still has brief bursts for some reason. it should be much more
effective than before though as oracle 23/26 really does not handle
more than a few connections.
this change reverts oracle to oracle18c for now in setup.cfg;
further work will be needed to determine if oracle23c can be
run with this test suite
Mike Bayer [Tue, 25 Nov 2025 00:33:18 +0000 (19:33 -0500)]
test oracle 23c, mariadb12; reduce backend use
one particular vector test wont run on oracle 23c free, so
just disable it.
added better skips for the rest of the vector tests and
fixed a deprecation issue.
this will be the first run on the new oracle23 on CI so we'll have to
see how this goes.
Also adjust for mariabdb12 being overly helpful with regards
to stale row updates.
as we are having trouble getting 23c to pass throug transaction
tests, i noted we have an explosion of tests due to the multiple
drivers, so this patch introduces __sparse_driver_backend__
for all tests where we want variety of
database server but there's no need to test every driver.
This should dramatically reduce the size of the test suite run
Shamil [Wed, 19 Nov 2025 13:02:28 +0000 (08:02 -0500)]
Fix type hint for with_for_update() to support tuples of table classes
Fixed typing issue where :meth:`.Select.with_for_update` would not support
lists of ORM entities in the :paramref:`.Select.with_for_update.of`
parameter. Pull request courtesy Shamil.
Mike Bayer [Tue, 18 Nov 2025 20:10:05 +0000 (15:10 -0500)]
propagate _scalar_type() for SelectStatementGrouping
Fixed issue where using the :meth:`.ColumnOperators.in_` operator with a
nested :class:`.CompoundSelect` statement (e.g. an ``INTERSECT`` of
``UNION`` queries) would raise a :class:`NotImplementedError` when the
nested compound select was the first argument to the outer compound select.
The ``_scalar_type()`` internal method now properly handles nested compound
selects.
Mike Bayer [Mon, 17 Nov 2025 21:32:54 +0000 (16:32 -0500)]
update lint setup
We are stuck on flake8 because we rely on many plugins with
specific behaviors. The situation has calcified where:
1. the whole world uses ruff
2. nobody cares about import order linting or all the other stuff
we do, and/or similar but not quite the same things are embedded
deeply into ruff which would require us giving up a lot of our
standards (like isort)
3. flake8 is absolutely never going to support pyproject.
4. flake8-pyproject works for this
beyond that, for t string support we want to make it easy
to get onto py3.14, so here we update black to the latest which
appears to fix some missing symbols for py3.14 t strings.
we should also migrate the remaining sqlalchemy test config
from setup.cfg to pyproject.toml but that should likely be
2.1 only
Mike Bayer [Sat, 15 Nov 2025 16:41:34 +0000 (11:41 -0500)]
bump nox to latest main + fix
apparently 2.0 was behind so this copies
the files with 2.0-specific modifications from the below change id
add pyv to file template; use = for all custom args
adding "test/" to pytest doesnt work because then we can't indicate
a specific set of test files. use = for all sqlalchemy-custom
parameters instead to avoid [1]
Yannick PÉROUX [Tue, 4 Nov 2025 17:58:03 +0000 (12:58 -0500)]
Typing: fix type of func.coalesce when used with hybrid properties
Fixed typing issue where :class:`.coalesce` would not return the correct
return type when a nullable form of that argument were passed, even though
this function is meant to select the non-null entry among possibly null
arguments. Pull request courtesy Yannick PÉROUX.
JetDrag [Wed, 5 Nov 2025 03:47:02 +0000 (22:47 -0500)]
feat: Support MySQL FOR SHARE locking syntax.
Added support for MySQL 8.0.1 + ``FOR SHARE`` to be emitted for the
:meth:`.Select.with_for_uddate` method, which offers compatibility with
``NOWAIT`` and ``SKIP LOCKED``. The new syntax is used only for MySQL when
version 8.0.1 or higher is detected. Pull request courtesy JetDrag.
Mike Bayer [Tue, 4 Nov 2025 14:13:45 +0000 (09:13 -0500)]
clarify Core / ORM insert parameter behaviors
it seems to have gotten lost in our newer docs that we're looking
at the first dict only for core insert. add sections to both
INSERT tutorials explaining this difference
Pat Buxton [Mon, 3 Nov 2025 14:49:33 +0000 (09:49 -0500)]
Add order by clause to dialect tests to ensure expected result order
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Failing for Starrocks dialect currently without overrides, the amended dialect tests require an order by clause to ensure the expected result.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
Mike Bayer [Sat, 1 Nov 2025 13:56:18 +0000 (09:56 -0400)]
use the default driver for sparse backend
the memusage tests use sparse_backend but should use the
main driver in a set; they were using the pysqlite_numeric
dialect which is not a real dialect and apparently runs
dramatically slower for the memusage tests since it generates
more memory artifacts.
Mike Bayer [Sat, 1 Nov 2025 02:57:28 +0000 (22:57 -0400)]
fix sqlite regex for quoted fk, pk names
Fixed issue where SQLite dialect would fail to reflect constraint names
that contained uppercase letters or other characters requiring quoting. The
regular expressions used to parse primary key, foreign key, and unique
constraint names from the ``CREATE TABLE`` statement have been updated to
properly handle both quoted and unquoted constraint names.
Mike Bayer [Fri, 31 Oct 2025 15:51:37 +0000 (11:51 -0400)]
ensure util.get_annotations() is used
Fixed issue in Python 3.14 where dataclass transformation would fail when
a mapped class using :class:`.MappedAsDataclass` included a
:func:`.relationship` referencing a class that was not available at
runtime (e.g., within a ``TYPE_CHECKING`` block). This occurred when using
Python 3.14's :pep:`649` deferred annotations feature, which is the
default behavior without a ``from __future__ import annotations``
directive.
Mike Bayer [Tue, 28 Oct 2025 20:20:54 +0000 (16:20 -0400)]
try not to rely on GC working at all for tests
GC seems to be working in fewer cases across different interpreters,
so add a new step to force an InstanceState to act as though
it got dereferenced, for those cases where we have del'ed the object,
done a gc collect, and we know it should not be reachable.
Mike Bayer [Tue, 28 Oct 2025 19:22:45 +0000 (15:22 -0400)]
set nox min version only for newer sys python
The min version here fails on github actions because the runners
there use the single version of python for the install, and the newer
nox is not available on older pythons like 3.7.
We probably dont need the min version anyway, as since we have just
switched to venv in 5e62ea65167c, the misinterpretation of
"python3.14t" doesn't seem to happen anyway. however, all versions
of nox seem willing to install a 3.14t interpreter when 3.14 is
requested, and a 3.14t is sooner in the path. does not seem to
happen the other way around though (e.g. when 3.14t is requested,
it does not use a 3.14)
Anyway, for better predictability leave a min nox version in place
where we can do so for now.
Mike Bayer [Tue, 28 Oct 2025 14:03:09 +0000 (10:03 -0400)]
rewrite pool subsecond test to use mocking
the timing here is too sensitive to play out reliably
on CI machines particularly free threaded, so mock the time()
callable instead and ensure with subsecond clock intervals
we do the right math
Mike Bayer [Tue, 28 Oct 2025 18:06:14 +0000 (14:06 -0400)]
use the nox venv backend
This backend seems to run from the start against the resolved
interpreter, which we want. The virtualenv option, which is
the default, has the problem indicated at [1].
Mike Bayer [Sat, 18 Oct 2025 20:43:22 +0000 (16:43 -0400)]
improve sqlite reflection regex for unusual names/formats
A series of improvements have been made for reflection of CHECK constraints
on SQLite. The reflection logic now correctly handles table names
containing the strings "CHECK" or "CONSTRAINT", properly supports all four
SQLite identifier quoting styles (double quotes, single quotes, brackets,
and backticks) for constraint names, and accurately parses CHECK constraint
expressions containing parentheses within string literals using balanced
parenthesis matching with string context tracking. Big thanks to
GruzdevAV for new test cases and implementation ideas.
Mike Bayer [Mon, 29 Sep 2025 03:44:41 +0000 (23:44 -0400)]
add nox support (but dont switch out fully)
This backports the nox change from main/2.1 so that we have full
nox support available for the 2.0 series, however does not modify
any existing docs or test systems.
A noxfile.py has been added to allow testing with nox. This is a direct
port of 2.1's move to nox, however leaves the tox.ini file in place and
retains all test documentation in terms of tox. Version 2.1 will move to
nox fully, including deprecation warnings for tox and new testing
documentation.
krave1986 [Mon, 20 Oct 2025 20:06:31 +0000 (04:06 +0800)]
Fix missing back_populates in Note.item relationship in example code (#12925)
Without back_populates, the subsequent code would not automatically populate Item.notes or generate the key, which contradicts what the documentation intends to demonstrate.
krave1986 [Wed, 15 Oct 2025 19:37:10 +0000 (03:37 +0800)]
Update collection_api.rst (#12912)
In the Dictionary Collections section, the example code incorrectly calls .items() on the dictionary but shows dictionary output instead of the items() method's actual return value.
Mike Bayer [Wed, 15 Oct 2025 18:47:38 +0000 (14:47 -0400)]
fully copy_internals for AnnotatedFromClause for straight cloned traverse
Fixed issue where using :meth:`_sql.Select.params` to replace bound
parameters in a query could fail for some cases where the parameters
were embedded in subqueries or CTEs when ORM classes were involved,
due to issues with internal query traversal for these cases.
Mike Bayer [Tue, 14 Oct 2025 21:19:26 +0000 (17:19 -0400)]
Support warnings in exclusions
this adds a new feature to exclusions ``warns_if()`` which applies
the expect_warnings() context manager to a test method. Additionally,
at the class level these requirements can be extracted from a
``__requirements__`` directive and also added to global Python warnings
filter using catch_warnings().
Allen Chen [Fri, 3 Oct 2025 02:53:34 +0000 (22:53 -0400)]
Fix mssql index column order
Fixed issue where the index reflection for SQL Server would
not correctly return the order of the column inside an index
when the order of the columns in the index did not match the
order of the columns in the table.
Pull request courtesy of Allen Chen.
Mike Bayer [Thu, 9 Oct 2025 16:43:16 +0000 (12:43 -0400)]
dont generate skipped suite names
For a suite that is ``__backend__``, if no tests will run, then
don't include that suite in the result at all, rather than
keeping them there and having them skip. the skip suites are
producing lots of noise in junit files
Mike Bayer [Wed, 8 Oct 2025 15:09:59 +0000 (11:09 -0400)]
correct for FK reflecting on PK / index col at the same time
Fixed issue in the MSSQL dialect's foreign key reflection query where
duplicate rows could be returned when a foreign key column and its
referenced primary key column have the same name, and both the referencing
and referenced tables have indexes with the same name. This resulted in an
"ForeignKeyConstraint with duplicate source column references are not
supported" error when attempting to reflect such tables. The query has been
corrected to exclude indexes on the child table when looking for unique
indexes referenced by foreign keys.
Mike Bayer [Tue, 7 Oct 2025 18:55:44 +0000 (14:55 -0400)]
Add Executable traverse internals to Executable subclasses and turn tests on
Fixed a caching issue where :func:`_orm.with_loader_criteria` would
incorrectly reuse cached bound parameter values when used with
:class:`_sql.CompoundSelect` constructs such as :func:`_sql.union`. The
issue was caused by the cache key for compound selects not including the
execution options that are part of the :class:`_sql.Executable` base class,
which :func:`_orm.with_loader_criteria` uses to apply its criteria
dynamically. The fix ensures that compound selects and other executable
constructs properly include execution options in their cache key traversal.
Denis Laxalde [Fri, 3 Oct 2025 19:10:08 +0000 (15:10 -0400)]
Add type annotations to indexable extension code
A typing test case (plain_files/ext/indexable.py) is also added.
In order to make the methods of index_property conform with type
definitions of `fget`, `fset` and `fdel` arguments of hybrid_property,
we need to make the signature of protocols
(e.g. `_HybridGetterType`) `__call__`) method positional only.
Pablo Estevez [Sat, 23 Aug 2025 12:33:47 +0000 (08:33 -0400)]
type pysqlite
<!-- Provide a general summary of your proposed changes in the Title field above -->
type pysqlite from dialects.
type some related code on pysqlite.py
related to #6810
<!-- Describe your changes in detail -->
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical / small typing error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Add explicit multi-threaded tests and support free-threaded build
Implemented initial support for free-threaded Python by adding new tests
and reworking the test harness and GitHub Actions to include Python 3.13t
and Python 3.14t in test runs. Two concurrency issues have been identified
and fixed: the first involves initialization of the ``.c`` collection on a
``FromClause``, a continuation of :ticket:`12302`, where an optional mutex
under free-threading is added; the second involves synchronization of the
pool "first_connect" event, which first received thread synchronization in
:ticket:`2964`, however under free-threading the creation of the mutex
itself runs under the same free-threading mutex. Initial pull request and
test suite courtesy Lysandros Nikolaou.
fix: pass dialect-specific kwargs to MetaData.reflect
Fixed issue where :meth:`_schema.MetaData.reflect` did not forward
dialect-specific keyword arguments to the :class:`_engine.Inspector`
methods, causing options like ``oracle_resolve_synonyms`` to be ignored
during reflection. The method now ensures that all extra kwargs passed to
:meth:`_schema.MetaData.reflect` are forwarded to
:meth:`_engine.Inspector.get_table_names` and related reflection methods.
Pull request courtesy Lukáš Kožušník.
Mike Bayer [Sat, 20 Sep 2025 22:31:25 +0000 (18:31 -0400)]
remove _py3k suffixes from test files
on the theme of 2.0/2.1 are relatively similar right now, do
some more cleanup removing the py3k suffixes that we dont need anymore.
I'm not sure that the logic that would search for these suffixes is
even present anymore and I would guess we removed it when we removed
py2k support. However, I am planning on possibly bringing it back
for some of the py314 stuff, so I still want to be able to do
suffix stuff. that will be for another patch.
Mike Bayer [Sat, 20 Sep 2025 18:08:55 +0000 (14:08 -0400)]
Use ARRAY type for any_(), all_() coercion
Fixed issue where the :func:`_sql.any_` and :func:`_sql.all_` aggregation
operators would not correctly coerce the datatype of the compared value, in
those cases where the compared value were not a simple int/str etc., such
as a Python ``Enum`` or other custom value. This would lead to execution
time errors for these values. This issue is essentially the same as
:ticket:`6515` which was for the now-legacy :meth:`.ARRAY.any` and
:meth:`.ARRAY.all` methods.
Mike Bayer [Sat, 20 Sep 2025 15:27:12 +0000 (11:27 -0400)]
refactor test_sqlite into a package
this is many years overdue, let's do it while 2.1/2.0 are in
sync and we are far away from 1.4 now
for the 2.0 cherry pick this was 99% a clean cherry pick;
adding only test_deprecated_serializer_args, removing
one cachekeysuite, and a 3.8 check for
test_determinsitic_parameter (yes it's misspelled)
Tip ten Brink [Mon, 15 Sep 2025 12:58:40 +0000 (08:58 -0400)]
Fix get_columns sqlite reflection rejecting tables with WITHOUT_ROWID and/or STRICT for generated column case
Fixed issue where SQLite table reflection would fail for tables using
``WITHOUT ROWID`` and/or ``STRICT`` table options when the table contained
generated columns. The regular expression used to parse ``CREATE TABLE``
statements for generated column detection has been updated to properly
handle these SQLite table options that appear after the column definitions.
Pull request courtesy Tip ten Brink.
Mike Bayer [Tue, 9 Sep 2025 19:16:45 +0000 (15:16 -0400)]
Add function mapped_as_dataclass
Added new decorator :func:`_orm.mapped_as_dataclass`, which is a function
based form of :meth:`_orm.registry.mapped_as_dataclass`; the method form
:meth:`_orm.registry.mapped_as_dataclass` does not seem to be correctly
recognized within the scope of :pep:`681` in recent mypy versions.
The new function is tested and mentioned in the docs, in 2.1 in a
subsequent patch (probably the one that adds unmapped_dataclass also)
we'll switch this new decorator to be the prominent one.
also alphabetize mapping_api.rst. while the summary box at the top
auto-sorts, have the sidebar alpha also, it's kind of weird how
these were in no order at all
Mike Bayer [Sun, 7 Sep 2025 00:20:00 +0000 (20:20 -0400)]
doc updates, localize test fixtures
testing with types is inherently awkward and subject
to changes in python interpreters (such as all the recent python 3.14
stuff we had them fix), but in this suite we already have a lot of
types that are defined inline inside of test methods. so since that's
how many of the tests work anyway, organize the big series of pep-695
and pep-593 structures into fixtures or individual tests to make
the whole suite easier to follow. pyright complains quite a lot
about this, so if this becomes a bigger issue for say mypy /pep484
target, we may have to revisit (which I'd likely do with more ignores)
or if function/method-local type declarations with global becomes a runtime
issue in py3.15 or something, we can revisit then where we would in
theory need to convert the entire suite, which I'd do with a more
consistent naming style for everything.
but for now try to go with fixtures / local type declarations so that
we dont have to wonder where all these types are used.
For 2.0 this further repairs some merge mismatches between 2.1 and
2.0, with tests lining up more closely to where they were placed in
2.1.
Mike Bayer [Thu, 4 Sep 2025 01:44:33 +0000 (21:44 -0400)]
liberalize pep695 matching
after many days of discussion we are moving to liberalize the
matching rules used for pep695 to the simple rule that we will resolve
a pep695 type to its immediate ``__value__`` without requiring that
it be present in the type map, however without any further recursive
checks (that is, we will not resolve ``__value__`` of ``__value__``).
This allows the vast majority of simple uses of pep695 types to not
require entries in the type map, including when the type points
to a simple Python type or any type that is present in the type_map.
Also supported is resolution of generic pep695 types against the
right side, including for Annotated types.
The change here in 2.1 will form the base for a revised approach
to the RegistryEvents patch for #9832, which will still provide
the RegistryEvents.resolve_type_annotation hook. In 2.0, we need
to scale back the warnings that are emitted so portions of this patch
will also be backported including similar changes to the test suite.
Mike Bayer [Fri, 5 Sep 2025 13:29:34 +0000 (09:29 -0400)]
interpret NULL in PG enum array values
Fixed issue where selecting an enum array column containing NULL values
would fail to parse properly in the PostgreSQL dialect. The
:func:`._split_enum_values` function now correctly handles NULL entries by
converting them to Python ``None`` values.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
