Mike Bayer [Wed, 16 Mar 2022 16:37:20 +0000 (12:37 -0400)]
remove intermediary _is_clone_of entries when cloning
Improvements in memory usage by the ORM, removing a significant set of
intermediary expression objects that are typically stored when a copy of an
expression object is created. These clones have been greatly reduced,
reducing the number of total expression objects stored in memory by
ORM mappings by about 30%.
note this change causes the tests to have a bit of a harder time with
GC, which we would assume is because mappings now have a lot more
garbage to clean up after mappers are configured. it remains
to be seen what the long term effects of this are.
Mike Bayer [Sun, 13 Mar 2022 17:37:11 +0000 (13:37 -0400)]
pep-484 - SQL column operations
note we are taking out the
ColumnOperartors[SQLCoreOperations] thing; not really clear
why that was needed and at the moment it seems I was likely
confused.
Mike Bayer [Mon, 14 Mar 2022 04:01:30 +0000 (00:01 -0400)]
restore an updated version of cascade backrefs
these sections were removed totally from SQLAlchemy 2.0
as the ``cascade_backrefs`` option was removed. However,
the current behavior is still one that should be documented,
so restore the two removed sections in a re-worded form
that presents the current behavior as well as some
explaination.
Mike Bayer [Tue, 8 Mar 2022 22:14:41 +0000 (17:14 -0500)]
pep-484: sqlalchemy.sql pass one
sqlalchemy.sql will require many passes to get all
modules even gradually typed. Will have to pick and
choose what modules can be strictly typed vs. which
can be gradual.
in this patch, emphasis is on visitors.py, cache_key.py,
annotations.py for strict typing, compiler.py is on gradual
typing but has much more structure, in particular where it
connects with the outside world.
The work within compiler.py also reached back out to
engine/cursor.py , default.py quite a bit.
Mike Bayer [Thu, 10 Mar 2022 16:57:00 +0000 (11:57 -0500)]
additional mypy strictness
enable type checking within untyped defs. This allowed
some more internals to be fixed up with assertions etc.
some internals that were unnecessary or not even used
at all were removed. BaseCursorResult was no longer
necessary since we only have one kind of CursorResult
now. The different ResultProxy subclasses that had
alternate "strategies" dont appear to be used at all
even in 1.4.x, as there's no code that accesses the
_cursor_strategy_cls attribute, which is also removed.
As these were mostly private constructs that weren't
even functioning correctly in any case,
it's fine to remove these over the 2.0 boundary.
Mike Bayer [Fri, 11 Mar 2022 21:23:14 +0000 (16:23 -0500)]
restore quickstart to toc
this was coming out wrong due to the wrong header level
on the page. it needs to be in a visible toc so that
the inner elements show up on the sidebar / top mobile nav.
Mike Bayer [Fri, 11 Mar 2022 21:01:09 +0000 (16:01 -0500)]
further simplify pool-sharing doc
1. the event based approach doesn't require dispose() to be
called at all, and the note that the "pool will hang" makes
no sense. I have no idea what that refers towards
2. the subsequent paragraph about connections and sessions
is unintelligible. old paragraphs like these are likely
why people complain about the docs so much. try to just
say "don't do this", as that is easier than trying to
explain to use connection.invalidate() etc.
Mike Bayer [Fri, 11 Mar 2022 18:40:16 +0000 (13:40 -0500)]
add copybutton
works great, including for plain code plus prompt code
with SQL (copies only the prompt code when prompts are present).
added some styling to zzzeeksphinx
Mike Bayer [Fri, 11 Mar 2022 15:08:36 +0000 (10:08 -0500)]
ORM quickstart
This is done in 1.4 style so it can be backported to 1.4.
Will put this up as is, we can work on it. For 2.0, the
ORM mapping will be updated to mapped_column() style when
we do the full pass.
Mike Bayer [Tue, 8 Mar 2022 18:40:12 +0000 (13:40 -0500)]
pop the stack that we pushed
Fixed regression caused by :ticket:`7760` where the new capabilities of
:class:`.TextualSelect` were not fully implemented within the compiler
properly, leading to issues with composed INSERT constructs such as "INSERT
FROM SELECT" and "INSERT...ON CONFLICT" when combined with CTE and textual
statements.
Mike Bayer [Tue, 8 Mar 2022 14:34:09 +0000 (09:34 -0500)]
support adapt_on_names for with_polymorphic
Added :paramref:`_orm.with_polymorphic.adapt_on_names` to the
:func:`_orm.with_polymorphic` function, which allows a polymorphic load
(typically with concrete mapping) to be stated against an alternative
selectable that will adapt to the original mapped selectable on column
names alone.
Mike Bayer [Mon, 7 Mar 2022 20:11:29 +0000 (15:11 -0500)]
support selectin_polymorphic w/ no fixed polymorphic_on
Fixed issue where the :func:`_orm.polymorphic_selectin` loader option would
not work with joined inheritance mappers that don't have a fixed
"polymorphic_on" column. Additionally added test support for a wider
variety of usage patterns with this construct.
Fixed bug where :func:`_orm.composite` attributes would not work in
conjunction with the :func:`_orm.selectin_polymorphic` loader strategy for
joined table inheritance.
Mike Bayer [Mon, 7 Mar 2022 17:01:42 +0000 (12:01 -0500)]
adapt create_engine from sqlalchemy2-stubs
this is much simplified, will try to see if _IsolationLevel
can work out, technically some driver can have custom values
here but in practice this might not be a thing
Mike Bayer [Fri, 4 Mar 2022 23:27:24 +0000 (18:27 -0500)]
warn for enum length silently ignored
the "length" parameter is silently ignored when native_enum
is not passed as False. if native_enum is True, a non-native
VARCHAR can still be generated. Warn for this silent ignore
right now, consider having "length" used in all cases where
non-native enum is rendered likely in 2.0.
Mike Bayer [Fri, 4 Mar 2022 22:17:53 +0000 (17:17 -0500)]
fix type string formatting calls
Fixed type-related error messages that would fail for values that were
tuples, due to string formatting syntax, including compile of unsupported
literal values and invalid boolean values.
Mike Bayer [Fri, 4 Mar 2022 21:28:31 +0000 (16:28 -0500)]
correct for pytest discovery
Made corrections to the default pytest configuration so that test discovery
runs correctly; previously, a configuration error had the effect of
discovery locating the wrong files if a super-directory of the current
directory were named "test".
petit87 [Sat, 26 Feb 2022 21:46:32 +0000 (16:46 -0500)]
Fix repr for MySQL SET, generic Enum
Fixed issues in :class:`_mysql.SET` datatype as well as :class:`.Enum`
where the ``__repr__()`` method would not render all optional parameters in
the string output, impacting the use of these types in Alembic
autogenerate. Pull request for MySQL courtesy Yuki Nishimine.
Mike Bayer [Thu, 3 Mar 2022 02:43:53 +0000 (21:43 -0500)]
improve error raise for dialect/pool events w/ async engine
Fixed issues where a descriptive error message was not raised for some
classes of event listening with an async engine, which should instead be a
sync engine instance.
Mike Bayer [Wed, 2 Mar 2022 16:51:57 +0000 (11:51 -0500)]
prevent Mapped[] auto-column logic w/ fixed table
When a class has __table__, people will still want to
annotate the attributes on classes, so make sure a
Mapped annotation without a right side is only interpreted
as a column if there is no __table__
Mike Bayer [Mon, 28 Feb 2022 04:05:46 +0000 (23:05 -0500)]
pep484 + abc bases for assocaitionproxy
went to this one next as it was going to be hard,
and also exercises the ORM expression hierarchy a bit.
made some adjustments to SQLCoreOperations etc.
Mike Bayer [Thu, 17 Feb 2022 18:43:04 +0000 (13:43 -0500)]
pep-484 for engine
All modules in sqlalchemy.engine are strictly
typed with the exception of cursor, default, and
reflection. cursor and default pass with non-strict
typing, reflection is waiting on the multi-reflection
refactor.
Behavioral changes:
* create_connect_args() methods return a tuple of list,
dict, rather than a list of list, dict
* removed allow_chars parameter from
pyodbc connector ._get_server_version_info()
method
* the parameter list passed to do_executemany is now
a list in all cases. previously, this was being run
through dialect.execute_sequence_format, which
defaults to tuple and was only intended for individual
tuple params.
* broke up dialect.dbapi into dialect.import_dbapi
class method and dialect.dbapi module object. added
a deprecation path for legacy dialects. it's not
really feasible to type a single attr as a classmethod
vs. module type. The "type_compiler" attribute also
has this problem with greater ability to work around,
left that one for now.
* lots of constants changing to be Enum, so that we can
type them. for fixed tuple-position constants in
cursor.py / compiler.py (which are used to avoid the
speed overhead of namedtuple), using Literal[value]
which seems to work well
* some tightening up in Row regarding __getitem__, which
we can do since we are on full 2.0 style result use
* altered the set_connection_execution_options and
set_engine_execution_options event flows so that the
dictionary of options may be mutated within the event
hook, where it will then take effect as the actual
options used. Previously, changing the dict would
be silently ignored which seems counter-intuitive
and not very useful.
* A lot of DefaultDialect/DefaultExecutionContext
methods and attributes, including underscored ones, move
to interfaces. This is not fully ideal as it means
the Dialect/ExecutionContext interfaces aren't publicly
subclassable directly, but their current purpose
is more of documentation for dialect authors who should
(and certainly are) still be subclassing the DefaultXYZ
versions in all cases
Overall, Result was the most extremely difficult class
hierarchy to type here as this hierarchy passes through
largely amorphous "row" datatypes throughout, which
can in fact by all kinds of different things, like
raw DBAPI rows, or Row objects, or "scalar"/Any, but
at the same time these types have meaning so I tried still
maintaining some level of semantic markings for these,
it highlights how complex Result is now, as it's trying
to be extremely efficient and inlined while also being
very open-ended and extensible.
Mike Bayer [Fri, 25 Feb 2022 17:40:21 +0000 (12:40 -0500)]
block pypy for now
the most recent pypy 7.3.8 series (3.7 and 3.9 included, likely
3.8 as well) have installed a 9 year old version of SQLite,
and additionally seem to have some other behavioral changes
like formatting of exception messages that is breaking
some tests. as we are waiting on a response at [1]
remove pypy testing for now.
zeeeeeb [Sat, 12 Feb 2022 19:00:02 +0000 (14:00 -0500)]
Implement generic Double and related fixed types
Added :class:`.Double`, :class:`.DOUBLE`, :class:`.DOUBLE_PRECISION`
datatypes to the base ``sqlalchemy.`` module namespace, for explicit use of
double/double precision as well as generic "double" datatypes. Use
:class:`.Double` for generic support that will resolve to DOUBLE/DOUBLE
PRECISION/FLOAT as needed for different backends.
Implemented DDL and reflection support for ``FLOAT`` datatypes which
include an explicit "binary_precision" value. Using the Oracle-specific
:class:`_oracle.FLOAT` datatype, the new parameter
:paramref:`_oracle.FLOAT.binary_precision` may be specified which will
render Oracle's precision for floating point types directly. This value is
interpreted during reflection. Upon reflecting back a ``FLOAT`` datatype,
the datatype returned is one of :class:`_types.DOUBLE_PRECISION` for a
``FLOAT`` for a precision of 126 (this is also Oracle's default precision
for ``FLOAT``), :class:`_types.REAL` for a precision of 63, and
:class:`_oracle.FLOAT` for a custom precision, as per Oracle documentation.
As part of this change, the generic :paramref:`_sqltypes.Float.precision`
value is explicitly rejected when generating DDL for Oracle, as this
precision cannot be accurately converted to "binary precision"; instead, an
error message encourages the use of
:meth:`_sqltypes.TypeEngine.with_variant` so that Oracle's specific form of
precision may be chosen exactly. This is a backwards-incompatible change in
behavior, as the previous "precision" value was silently ignored for
Oracle.
Mike Bayer [Wed, 23 Feb 2022 17:24:31 +0000 (12:24 -0500)]
Add more nesting features to add_cte()
Added new parameter :paramref:`.HasCTE.add_cte.nest_here` to
:meth:`.HasCTE.add_cte` which will "nest" a given :class:`.CTE` at the
level of the parent statement. This parameter is equivalent to using the
:paramref:`.HasCTE.cte.nesting` parameter, but may be more intuitive in
some scenarios as it allows the nesting attribute to be set simultaneously
along with the explicit level of the CTE.
The :meth:`.HasCTE.add_cte` method also accepts multiple CTE objects.
Mike Bayer [Wed, 23 Feb 2022 18:43:03 +0000 (13:43 -0500)]
support cx_Oracle DPI disconnect codes
Added support to parse "DPI" error codes from cx_Oracle exception objects
such as ``DPI-1080`` and ``DPI-1010``, both of which now indicate a
disconnect scenario as of cx_Oracle 8.3.
Mike Bayer [Wed, 23 Feb 2022 17:50:36 +0000 (12:50 -0500)]
support add_cte() for TextualSelect
Fixed issue where the :meth:`.HasCTE.add_cte` method as called upon a
:class:`.TextualSelect` instance was not being accommodated by the SQL
compiler. The fix additionally adds more "SELECT"-like compiler behavior to
:class:`.TextualSelect` including that DML CTEs such as UPDATE and INSERT
may be accommodated.
Mike Bayer [Wed, 23 Feb 2022 19:02:07 +0000 (14:02 -0500)]
disable pyright from pep 484
new version of pyright is suddenly emitting an
error that makes no sense for a particular line of code,
breaking builds. As we already have mypy which is generally
more conservative in its checking, disable pyright from CI
for now until it changes its behaviors less often.
Mike Bayer [Mon, 21 Feb 2022 21:56:16 +0000 (16:56 -0500)]
undefer column name sooner to accommodate composites
This was from adding composite class introspection as a
means of determining column name, cols need to have their
normal attribute name set up already.
Federico Caselli [Sun, 20 Feb 2022 11:03:19 +0000 (12:03 +0100)]
Revert SQLAlchemy warnings to warnings.py
Configuring the warning filters in pyproject breaks tests if
no sqlalchemy is installed in the env, since the filters are
processed before loading conftest. It also may interfere
with coverage.
Mike Bayer [Sun, 20 Feb 2022 14:51:22 +0000 (09:51 -0500)]
improve reflection of inline UNIQUE constraints
Fixed issue where SQLite unique constraint reflection would not work
for an inline UNIQUE constraint where the column name had an underscore
in its name.
Added support for reflecting SQLite inline unique constraints where
the column names are formatted with SQLite "escape quotes" ``[]``
or `` ` ``, which are discarded by the database when producing the
column name.
This method I would assume got committed during the
1.4 engine refactor, where we moved from different kinds of
ResultProxy implementations to different strategy
classes instead. These strategies are set up by
dialects by setting "self.cursor_fetch_strategy"
in the execution context. The method here was
likely a previous iteration of that which got merged
but was never used.
Mike Bayer [Sat, 19 Feb 2022 19:11:19 +0000 (14:11 -0500)]
updates for mariadb connector 1.0.10
Fixed regression in mariadbconnector dialect as of mariadb connector 1.0.10
where the DBAPI no longer pre-buffers cursor.lastrowid. The dialect now
fetches this value proactively for situations where it applies.
test_invalidate_on_results seems to pass for mariadbconnector now.
the driver has likely changed how it buffers result sets. This is
a major change for them to make in a point release so we might
want to watch this in case they reverse course again.
Mike Bayer [Wed, 16 Feb 2022 04:43:51 +0000 (23:43 -0500)]
pep-484 for pool
also extends into some areas of utils, events and others
as needed.
Formalizes a public hierarchy for pool API,
with ManagesConnection -> PoolProxiedConnection /
ConnectionPoolEntry for connectionfairy / connectionrecord,
which are now what's exposed in the event API and other
APIs. all public API docs moved to the new objects.
Corrects the mypy plugin's check for sqlalchemy-stubs
not being insatlled, which has to be imported using the
dash in the name to be effective.
Mike Bayer [Sun, 13 Feb 2022 21:45:18 +0000 (16:45 -0500)]
pep-484 for sqlalchemy.event; use future annotations
__future__.annotations mode allows us to use non-string
annotations for argument and return types in most cases,
but more importantly it removes a large amount of runtime
overhead that would be spent in evaluating the annotations.
Mike Bayer [Tue, 15 Feb 2022 15:53:07 +0000 (10:53 -0500)]
correct for non-deterministic gc artifacts
Observed the tests here have different profiling
counts when run individually vs. as a group, and
this seems to be due to whether or not results of
each query are garbage collected or not. for
all but one test, ensuring results stay between
query runs seems to meet the current profiling
counts.
Mike Bayer [Mon, 14 Feb 2022 14:01:56 +0000 (09:01 -0500)]
remove **kw from AsyncSession.begin() / AsyncSession.begin_nested()
Removed the unused ``**kw`` arguments from
:class:`_asyncio.AsyncSession.begin` and
:class:`_asyncio.AsyncSession.begin_nested`. These kw aren't used and
appear to have been added to the API in error.
Mike Bayer [Mon, 24 Jan 2022 22:04:27 +0000 (17:04 -0500)]
establish mypy / typing approach for v2.0
large patch to get ORM / typing efforts started.
this is to support adding new test cases to mypy,
support dropping sqlalchemy2-stubs entirely from the
test suite, validate major ORM typing reorganization
to eliminate the need for the mypy plugin.
* New declarative approach which uses annotation
introspection, fixes: #7535
* Mapped[] is now at the base of all ORM constructs
that find themselves in classes, to support direct
typing without plugins
* Mypy plugin updated for new typing structures
* Mypy test suite broken out into "plugin" tests vs.
"plain" tests, and enhanced to better support test
structures where we assert that various objects are
introspected by the type checker as we expect.
as we go forward with typing, we will
add new use cases to "plain" where we can assert that
types are introspected as we expect.
* For typing support, users will be much more exposed to the
class names of things. Add these all to "sqlalchemy" import
space.
* Column(ForeignKey()) no longer needs to be `@declared_attr`
if the FK refers to a remote table
* composite() attributes mapped to a dataclass no longer
need to implement a `__composite_values__()` method
* with_variant() accepts multiple dialect names
Mike Bayer [Sat, 12 Feb 2022 15:50:45 +0000 (10:50 -0500)]
Raise and re-catch NameError from _ModNS
Fixed issue where using a fully qualified path for the classname in
:func:`_orm.relationship` that nonetheless contained an incorrect name for
path tokens that were not the first token, would fail to raise an
informative error and would instead fail randomly at a later step.
Mike Bayer [Wed, 22 Dec 2021 13:34:15 +0000 (08:34 -0500)]
use QueuePool for sqlite file databases
The SQLite dialect now defaults to :class:`_pool.QueuePool` when a file
based database is used. This is set along with setting the
``check_same_thread`` parameter to ``False``. It has been observed that the
previous approach of defaulting to :class:`_pool.NullPool`, which does not
hold onto database connections after they are released, did in fact have a
measurable negative performance impact. As always, the pool class is always
customizable via the :paramref:`_sa.create_engine.poolclass` parameter.
Mike Bayer [Tue, 8 Feb 2022 15:12:33 +0000 (10:12 -0500)]
Accommodate escaped_bind_names for defaults/insert params
Fixed issue in Oracle dialect where using a column name that requires
quoting when written as a bound parameter, such as ``"_id"``, would not
correctly track a Python generated default value due to the bound-parameter
rewriting missing this value, causing an Oracle error to be raised.
Arie Bovenberg [Sun, 6 Feb 2022 21:37:02 +0000 (16:37 -0500)]
add slotscheck to CI
As discussed in #7589, `slotscheck` can prevent slots-related mistakes from creeping back in.
Plan for now is to have slotscheck part of the "lint" tests
(renamed from pep8) that will run for CI and github actions.
To support slotscheck's runtime nature, slotscheck is
run twice, first with cython exts enabled and then
with them disabled via new environment variable.
Also added sqlalchemy[mypy] dependency to support slots
checking the mypy plugin.
Found and fixed one more `__slots__` issue by disabling C
exts.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
