Mike Bayer [Fri, 4 Mar 2022 23:27:24 +0000 (18:27 -0500)]
warn for enum length silently ignored
the "length" parameter is silently ignored when native_enum
is not passed as False. if native_enum is True, a non-native
VARCHAR can still be generated. Warn for this silent ignore
right now, consider having "length" used in all cases where
non-native enum is rendered likely in 2.0.
Mike Bayer [Fri, 4 Mar 2022 22:17:53 +0000 (17:17 -0500)]
fix type string formatting calls
Fixed type-related error messages that would fail for values that were
tuples, due to string formatting syntax, including compile of unsupported
literal values and invalid boolean values.
Mike Bayer [Fri, 4 Mar 2022 21:28:31 +0000 (16:28 -0500)]
correct for pytest discovery
Made corrections to the default pytest configuration so that test discovery
runs correctly; previously, a configuration error had the effect of
discovery locating the wrong files if a super-directory of the current
directory were named "test".
petit87 [Sat, 26 Feb 2022 21:46:32 +0000 (16:46 -0500)]
Fix repr for MySQL SET, generic Enum
Fixed issues in :class:`_mysql.SET` datatype as well as :class:`.Enum`
where the ``__repr__()`` method would not render all optional parameters in
the string output, impacting the use of these types in Alembic
autogenerate. Pull request for MySQL courtesy Yuki Nishimine.
Mike Bayer [Thu, 3 Mar 2022 02:43:53 +0000 (21:43 -0500)]
improve error raise for dialect/pool events w/ async engine
Fixed issues where a descriptive error message was not raised for some
classes of event listening with an async engine, which should instead be a
sync engine instance.
Mike Bayer [Wed, 2 Mar 2022 16:51:57 +0000 (11:51 -0500)]
prevent Mapped[] auto-column logic w/ fixed table
When a class has __table__, people will still want to
annotate the attributes on classes, so make sure a
Mapped annotation without a right side is only interpreted
as a column if there is no __table__
Mike Bayer [Mon, 28 Feb 2022 04:05:46 +0000 (23:05 -0500)]
pep484 + abc bases for assocaitionproxy
went to this one next as it was going to be hard,
and also exercises the ORM expression hierarchy a bit.
made some adjustments to SQLCoreOperations etc.
Mike Bayer [Thu, 17 Feb 2022 18:43:04 +0000 (13:43 -0500)]
pep-484 for engine
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection. cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.
Behavioral changes:
* create_connect_args() methods return a tuple of list,
dict, rather than a list of list, dict
* removed allow_chars parameter from
pyodbc connector ._get_server_version_info()
method
* the parameter list passed to do_executemany is now
a list in all cases. previously, this was being run
through dialect.execute_sequence_format, which
defaults to tuple and was only intended for individual
tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
class method and dialect.dbapi module object. added
a deprecation path for legacy dialects. it's not
really feasible to type a single attr as a classmethod
vs. module type. The "type_compiler" attribute also
has this problem with greater ability to work around,
left that one for now.
* lots of constants changing to be Enum, so that we can
type them. for fixed tuple-position constants in
cursor.py / compiler.py (which are used to avoid the
speed overhead of namedtuple), using Literal[value]
which seems to work well
* some tightening up in Row regarding __getitem__, which
we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
set_engine_execution_options event flows so that the
dictionary of options may be mutated within the event
hook, where it will then take effect as the actual
options used. Previously, changing the dict would
be silently ignored which seems counter-intuitive
and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
methods and attributes, including underscored ones, move
to interfaces. This is not fully ideal as it means
the Dialect/ExecutionContext interfaces aren't publicly
subclassable directly, but their current purpose
is more of documentation for dialect authors who should
(and certainly are) still be subclassing the DefaultXYZ
versions in all cases
Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.
Mike Bayer [Fri, 25 Feb 2022 17:40:21 +0000 (12:40 -0500)]
block pypy for now
the most recent pypy 7.3.8 series (3.7 and 3.9 included, likely
3.8 as well) have installed a 9 year old version of SQLite,
and additionally seem to have some other behavioral changes
like formatting of exception messages that is breaking
some tests. as we are waiting on a response at [1]
remove pypy testing for now.
zeeeeeb [Sat, 12 Feb 2022 19:00:02 +0000 (14:00 -0500)]
Implement generic Double and related fixed types
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.
Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.
As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.
Mike Bayer [Wed, 23 Feb 2022 17:24:31 +0000 (12:24 -0500)]
Add more nesting features to add_cte()
Added new parameter :paramref:`.HasCTE.add_cte.nest_here` to
:meth:`.HasCTE.add_cte` which will "nest" a given :class:`.CTE` at the
level of the parent statement. This parameter is equivalent to using the
:paramref:`.HasCTE.cte.nesting` parameter, but may be more intuitive in
some scenarios as it allows the nesting attribute to be set simultaneously
along with the explicit level of the CTE.
The :meth:`.HasCTE.add_cte` method also accepts multiple CTE objects.
Mike Bayer [Wed, 23 Feb 2022 18:43:03 +0000 (13:43 -0500)]
support cx_Oracle DPI disconnect codes
Added support to parse "DPI" error codes from cx_Oracle exception objects
such as ``DPI-1080`` and ``DPI-1010``, both of which now indicate a
disconnect scenario as of cx_Oracle 8.3.
Mike Bayer [Wed, 23 Feb 2022 17:50:36 +0000 (12:50 -0500)]
support add_cte() for TextualSelect
Fixed issue where the :meth:`.HasCTE.add_cte` method as called upon a
:class:`.TextualSelect` instance was not being accommodated by the SQL
compiler. The fix additionally adds more "SELECT"-like compiler behavior to
:class:`.TextualSelect` including that DML CTEs such as UPDATE and INSERT
may be accommodated.
Mike Bayer [Wed, 23 Feb 2022 19:02:07 +0000 (14:02 -0500)]
disable pyright from pep 484
new version of pyright is suddenly emitting an
error that makes no sense for a particular line of code,
breaking builds. As we already have mypy which is generally
more conservative in its checking, disable pyright from CI
for now until it changes its behaviors less often.
Mike Bayer [Mon, 21 Feb 2022 21:56:16 +0000 (16:56 -0500)]
undefer column name sooner to accommodate composites
This was from adding composite class introspection as a
means of determining column name, cols need to have their
normal attribute name set up already.
Federico Caselli [Sun, 20 Feb 2022 11:03:19 +0000 (12:03 +0100)]
Revert SQLAlchemy warnings to warnings.py
Configuring the warning filters in pyproject breaks tests if
no sqlalchemy is installed in the env, since the filters are
processed before loading conftest. It also may interfere
with coverage.
Mike Bayer [Sun, 20 Feb 2022 14:51:22 +0000 (09:51 -0500)]
improve reflection of inline UNIQUE constraints
Fixed issue where SQLite unique constraint reflection would not work
for an inline UNIQUE constraint where the column name had an underscore
in its name.
Added support for reflecting SQLite inline unique constraints where
the column names are formatted with SQLite "escape quotes" ``[]``
or `` ` ``, which are discarded by the database when producing the
column name.
This method I would assume got committed during the
1.4 engine refactor, where we moved from different kinds of
ResultProxy implementations to different strategy
classes instead. These strategies are set up by
dialects by setting "self.cursor_fetch_strategy"
in the execution context. The method here was
likely a previous iteration of that which got merged
but was never used.
Mike Bayer [Sat, 19 Feb 2022 19:11:19 +0000 (14:11 -0500)]
updates for mariadb connector 1.0.10
Fixed regression in mariadbconnector dialect as of mariadb connector 1.0.10
where the DBAPI no longer pre-buffers cursor.lastrowid. The dialect now
fetches this value proactively for situations where it applies.
test_invalidate_on_results seems to pass for mariadbconnector now.
the driver has likely changed how it buffers result sets. This is
a major change for them to make in a point release so we might
want to watch this in case they reverse course again.
Mike Bayer [Wed, 16 Feb 2022 04:43:51 +0000 (23:43 -0500)]
pep-484 for pool
also extends into some areas of utils, events and others
as needed.
Formalizes a public hierarchy for pool API,
with ManagesConnection -> PoolProxiedConnection /
ConnectionPoolEntry for connectionfairy / connectionrecord,
which are now what's exposed in the event API and other
APIs. all public API docs moved to the new objects.
Corrects the mypy plugin's check for sqlalchemy-stubs
not being insatlled, which has to be imported using the
dash in the name to be effective.
Mike Bayer [Sun, 13 Feb 2022 21:45:18 +0000 (16:45 -0500)]
pep-484 for sqlalchemy.event; use future annotations
__future__.annotations mode allows us to use non-string
annotations for argument and return types in most cases,
but more importantly it removes a large amount of runtime
overhead that would be spent in evaluating the annotations.
Mike Bayer [Tue, 15 Feb 2022 15:53:07 +0000 (10:53 -0500)]
correct for non-deterministic gc artifacts
Observed the tests here have different profiling
counts when run individually vs. as a group, and
this seems to be due to whether or not results of
each query are garbage collected or not. for
all but one test, ensuring results stay between
query runs seems to meet the current profiling
counts.
Mike Bayer [Mon, 14 Feb 2022 14:01:56 +0000 (09:01 -0500)]
remove **kw from AsyncSession.begin() / AsyncSession.begin_nested()
Removed the unused ``**kw`` arguments from
:class:`_asyncio.AsyncSession.begin` and
:class:`_asyncio.AsyncSession.begin_nested`. These kw aren't used and
appear to have been added to the API in error.
Mike Bayer [Mon, 24 Jan 2022 22:04:27 +0000 (17:04 -0500)]
establish mypy / typing approach for v2.0
large patch to get ORM / typing efforts started.
this is to support adding new test cases to mypy,
support dropping sqlalchemy2-stubs entirely from the
test suite, validate major ORM typing reorganization
to eliminate the need for the mypy plugin.
* New declarative approach which uses annotation
introspection, fixes: #7535
* Mapped[] is now at the base of all ORM constructs
that find themselves in classes, to support direct
typing without plugins
* Mypy plugin updated for new typing structures
* Mypy test suite broken out into "plugin" tests vs.
"plain" tests, and enhanced to better support test
structures where we assert that various objects are
introspected by the type checker as we expect.
as we go forward with typing, we will
add new use cases to "plain" where we can assert that
types are introspected as we expect.
* For typing support, users will be much more exposed to the
class names of things. Add these all to "sqlalchemy" import
space.
* Column(ForeignKey()) no longer needs to be `@declared_attr`
if the FK refers to a remote table
* composite() attributes mapped to a dataclass no longer
need to implement a `__composite_values__()` method
* with_variant() accepts multiple dialect names
Mike Bayer [Sat, 12 Feb 2022 15:50:45 +0000 (10:50 -0500)]
Raise and re-catch NameError from _ModNS
Fixed issue where using a fully qualified path for the classname in
:func:`_orm.relationship` that nonetheless contained an incorrect name for
path tokens that were not the first token, would fail to raise an
informative error and would instead fail randomly at a later step.
Mike Bayer [Wed, 22 Dec 2021 13:34:15 +0000 (08:34 -0500)]
use QueuePool for sqlite file databases
The SQLite dialect now defaults to :class:`_pool.QueuePool` when a file
based database is used. This is set along with setting the
``check_same_thread`` parameter to ``False``. It has been observed that the
previous approach of defaulting to :class:`_pool.NullPool`, which does not
hold onto database connections after they are released, did in fact have a
measurable negative performance impact. As always, the pool class is always
customizable via the :paramref:`_sa.create_engine.poolclass` parameter.
Mike Bayer [Tue, 8 Feb 2022 15:12:33 +0000 (10:12 -0500)]
Accommodate escaped_bind_names for defaults/insert params
Fixed issue in Oracle dialect where using a column name that requires
quoting when written as a bound parameter, such as ``"_id"``, would not
correctly track a Python generated default value due to the bound-parameter
rewriting missing this value, causing an Oracle error to be raised.
Arie Bovenberg [Sun, 6 Feb 2022 21:37:02 +0000 (16:37 -0500)]
add slotscheck to CI
As discussed in #7589, `slotscheck` can prevent slots-related mistakes from creeping back in.
Plan for now is to have slotscheck part of the "lint" tests
(renamed from pep8) that will run for CI and github actions.
To support slotscheck's runtime nature, slotscheck is
run twice, first with cython exts enabled and then
with them disabled via new environment variable.
Also added sqlalchemy[mypy] dependency to support slots
checking the mypy plugin.
Found and fixed one more `__slots__` issue by disabling C
exts.
Mike Bayer [Mon, 7 Feb 2022 17:08:51 +0000 (12:08 -0500)]
apply literal value resolution to String
Python string values for which a SQL type is determined from the type of
the value, mainly when using :func:`_sql.literal`, will now apply the
:class:`_types.String` type, rather than the :class:`_types.Unicode`
datatype, for Python string values that test as "ascii only" using Python
``str.isascii()``. If the string is not ``isascii()``, the
:class:`_types.Unicode` datatype will be bound instead, which was used in
all string detection previously. This behavior **only applies to in-place
detection of datatypes when using ``literal()`` or other contexts that have
no existing datatype**, which is not usually the case under normal
:class:`_schema.Column` comparison operations, where the type of the
:class:`_schema.Column` being compared always takes precedence.
Use of the :class:`_types.Unicode` datatype can determine literal string
formatting on backends such as SQL Server, where a literal value (i.e.
using ``literal_binds``) will be rendered as ``N'<value>'`` instead of
``'value'``. For normal bound value handling, the :class:`_types.Unicode`
datatype also may have implications for passing values to the DBAPI, again
in the case of SQL Server, the pyodbc driver supports the use of
:ref:`setinputsizes mode <mssql_pyodbc_setinputsizes>` which will handle
:class:`_types.String` versus :class:`_types.Unicode` differently.
Mike Bayer [Fri, 4 Feb 2022 14:04:49 +0000 (09:04 -0500)]
ensure exception raised for all stream w/ sync result
Fixed issue where the :meth:`_asyncio.AsyncSession.execute` method failed
to raise an informative exception if the ``stream_results`` execution
option were used, which is incompatible with a sync-style
:class:`_result.Result` object. An exception is now raised in this scenario
in the same way one is already raised when using ``stream_results`` in
conjunction with the :meth:`_asyncio.AsyncConnection.execute` method.
Additionally, for improved stability with state-sensitive dialects such as
asyncmy, the cursor is now closed when this error condition is raised;
previously with the asyncmy dialect, the connection would go into an
invalid state with unconsumed server side results remaining.
Mike Bayer [Fri, 4 Feb 2022 02:58:14 +0000 (21:58 -0500)]
fall back to SHOW VARIABLES for MySQL < 5.6
Fixed regression caused by :ticket:`7518` where changing the syntax "SHOW
VARIABLES" to "SELECT @@" broke compatibility with MySQL versions older
than 5.6, including early 5.0 releases. While these are very old MySQL
versions, a change in compatibility was not planned, so version-specific
logic has been restored to fall back to "SHOW VARIABLES" for MySQL server
versions < 5.6.
includes unrelated orm/test_expire ordering issue , only showing
up on 1.4 / py2.7 but seems to be passing by luck otherwise
Kurt McKee [Wed, 2 Feb 2022 19:46:40 +0000 (13:46 -0600)]
Remove duplicate word "other" in sentence (#7655)
Spotted this while reading the docs.
Word wrapping makes it less obvious, but the sentence previously read "The phrase after "ON UPDATE" or "ON DELETE" may also ~~other~~ allow other phrases".
Arie Bovenberg [Tue, 1 Feb 2022 20:08:19 +0000 (15:08 -0500)]
Fix overlapping slots, base classes without slots
Some `__slots__` were not in order.
Fixes #7527
### Description
I'm fixing two types of slots mistakes:
- [x] remove overlapping slots (i.e. slots already defined on a base class)
- [x] fix broken inheritance (i.e. slots class inheriting from a non-slots class)
- [x] slots added to base class `TransactionalContext`. It seemed to use two attributes, which I've added as slots.
- [x] empty slots removed from `ORMOption`. Its base class explicitly makes use of `__dict__` so empty slots don't add anything.
- [x] empty slots added to `PostLoader`. It doesn't appear to use any slots not already defined on its base classes.
- [x] empty slots added to `IterateMappersMixin`. It doesn't appear to use any slots not already defined on its subclasses.
- [x] empty slots added to `ImmutableContainer`. It doesn't use any fields.
- [x] empty slots added to `OperatorType`. It's a protocol.
- [x] empty slots added to `InternalTraversal`, `_HasTraversalDispatch`. They don't seem to use attributes on their own.
### Checklist
This pull request is:
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
Mike Bayer [Sun, 30 Jan 2022 19:25:36 +0000 (14:25 -0500)]
split out declarative varieties into the declarative mapping section
specific declarative styles like those for dataclasses and attrs
should be in the more reference-oriented "declarative_mapping"
section rather than the more introduction-oriented mapping_styles.
this will also make it easier for us to add still more ways of
mapping declaratively for 2.0.
Markus Gerstel [Wed, 26 Jan 2022 09:56:40 +0000 (04:56 -0500)]
Fix up Python logging metadata
Adjusted the logging for key SQLAlchemy components including
:class:`_engine.Engine`, :class:`_engine.Connection` to establish an
appropriate stack level parameter, so that the Python logging tokens
``funcName`` and ``lineno`` when used in custom logging formatters will
report the correct information, which can be useful when filtering log
output; supported on Python 3.8 and above. Pull request courtesy Markus
Gerstel.
Gilbert Gilb's [Sun, 23 Jan 2022 18:00:35 +0000 (13:00 -0500)]
Add compiler support for PostgreSQL "NOT VALID" constraints.
Added compiler support for the PostgreSQL ``NOT VALID`` phrase when rendering
DDL for the :class:`.CheckConstraint`, :class:`.ForeignKeyConstraint`
and :class:`.ForeignKey` schema constructs. Pull request courtesy
Gilbert Gilb's.
Mike Bayer [Mon, 24 Jan 2022 23:13:05 +0000 (18:13 -0500)]
restore set-as-superclass for OrderedSet
OrderedSet again subclasses set, spent some time
with the stubs at
https://github.com/python/typeshed/blob/master/stdlib/builtins.pyi#L887
to more deeply understand what they are doing here
so that we can type check fully.
Mike Bayer [Tue, 25 Jan 2022 16:01:25 +0000 (11:01 -0500)]
join to existing mark expr with "and"
ca48f461b2dcac2970829e4e0 considered an existing mark expression
plus legacy tags to be an error condition; however these can
be joined by "and" and will in our use case do the right thing.
The github action scripts make use of legacy tags. We can change
that also but I want to just make sure this combination works
fully as well.
Mike Bayer [Tue, 25 Jan 2022 05:45:30 +0000 (00:45 -0500)]
replace test tags with pytest.mark
replaced the __tags__ class attribute and the
--exclude-tags / --include-tags test runner options
with regular pytest.mark names
so that we can take advantage of mark expressions.
options --nomemory, --notimingintensive, --backend-only,
--exclude-tags, --include-tags remain as legacy but
make use of pytest mark for implemementation.
Added a "mypy" mark for the section of tests that are doing mypy
integration tests.
The __backend__ and __sparse_backend__ class attributes also
use pytest marks for their implementation, which also allows
the marks "backend" and "sparse_backend" to be used explicitly.
Also removed the no longer used "--cdecimal" option as this was
python 2 specific.
in theory, the usage of pytest marks could expand such that
the whole exclusions system would be based on it, but this
does not seem to have any advantage at the moment.
Mike Bayer [Sun, 9 Jan 2022 16:49:02 +0000 (11:49 -0500)]
mypy: sqlalchemy.util
Starting to set up practices and conventions to
get the library typed.
Key goals for typing are:
1. whole library can pass mypy without any strict
turned on.
2. we can incrementally turn on some strict flags on a per-package/
module basis, as here we turn on more strictness for sqlalchemy.util, exc,
and log
3. mypy ORM plugin tests work fully without sqlalchemy2-stubs
installed
4. public facing methods all have return types, major parameter
signatures filled in also
5. Foundational elements like util etc. are typed enough so that
we can use them in fully typed internals higher up the stack.
Conventions set up here:
1. we can use lots of config in setup.cfg to limit where mypy
is throwing errors and how detailed it should be in different
packages / modules. We can use this to push up gerrits
that will pass tests fully without everything being typed.
2. a new tox target pep484 is added. this links to a new jenkins
pep484 job that works across all projects (alembic, dogpile, etc.)
We've worked around some mypy bugs that will likely
be around for awhile, and also set up some core practices
for how to deal with certain things such as public_factory
modules (mypy won't accept a module from a callable at all,
so need to use simple type checking conditionals).
Mike Bayer [Sun, 23 Jan 2022 15:51:55 +0000 (10:51 -0500)]
after all that, use pytest warnings plugin
The warnings plugin lets us set the filters up
in the config, and as our filter requirements are now
simple we can just set this up.
additionally pytest now recommends pyproject.toml, since
we fully include this now, let's move it there.
the pytest logging plugin seems to not be any problem either
at the moment, so re-enable that. if it becomes apparent
whatever the problem was (which was probably that it was just
surprising, or something) we can disable it again and comment
what the reason was.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
