Mike Bayer [Thu, 11 Feb 2021 19:05:49 +0000 (14:05 -0500)]
Apply consistent labeling for all future style ORM queries
Fixed issue in new 1.4/2.0 style ORM queries where a statement-level label
style would not be preserved in the keys used by result rows; this has been
applied to all combinations of Core/ORM columns / session vs. connection
etc. so that the linkage from statement to result row is the same in all
cases.
also repairs a cache key bug where query.from_statement()
vs. select().from_statement() would not be disambiguated; the
compile options were not included in the cache key for
FromStatement.
Mike Bayer [Wed, 10 Feb 2021 15:36:52 +0000 (10:36 -0500)]
Don't pre-calc inserted primary key if no getter
the _setup_ins_pk_from_empty() method provides a placeholder
result for inserted_primary_key_rows that is not typically
going to be invoked. For an executemany (or even execute)
that isn't already stating it wants primary key values
up front, defer this computation until explicitly requested.
Mike Bayer [Mon, 8 Feb 2021 16:58:15 +0000 (11:58 -0500)]
Add identifier_preparer per-execution context for schema translates
Fixed bug where the "schema_translate_map" feature failed to be taken into
account for the use case of direct execution of
:class:`_schema.DefaultGenerator` objects such as sequences, which included
the case where they were "pre-executed" in order to generate primary key
values when implicit_returning was disabled.
Mike Bayer [Sun, 7 Feb 2021 14:51:48 +0000 (09:51 -0500)]
Document implicit IO points in ORM
I purposely didn't spend much documentation writing
about implicit IO when I first pushed out the asyncio
extension because I wanted to get a sense on what kinds
of issues people had. Now we know and the answer is
predictably "all of them". List out all the known
implicit IO points and how to avoid them. Also rename
the "adapting lazy loads" section, so that the title is less
suggestive that this is a necessary technique.
Mike Bayer [Fri, 5 Feb 2021 18:38:28 +0000 (13:38 -0500)]
Track a second from_linter for lateral subqueries
Fixed bug where the "cartesian product" assertion was not correctly
accommodating for joins between tables that relied upon the use of LATERAL
to connect from a subquery to another subquery in the enclosing context.
Additionally, enabled from_linting for the base assert_compile(),
however it remains off by default; to enable by default we would
have to make sure it isn't set for DDL compiles and there's also
a lot of tests that would also need to turn it off, so leaving
this off for expediency.
Mike Bayer [Thu, 4 Feb 2021 17:09:54 +0000 (12:09 -0500)]
Accommodate column-based naming conventions for pk constraint
Repaired / implemented support for primary key constraint naming
conventions that use column names/keys/etc as part of the convention. In
particular, this includes that the :class:`.PrimaryKeyConstraint` object
that's automatically associated with a :class:`.schema.Table` will update
its name as new primary key :class:`_schema.Column` objects are added to
the table and then to the constraint. Internal failure modes related to
this constraint construction process including no columns present, no name
present or blank name present are now accommodated.
Mike Bayer [Wed, 27 Jan 2021 20:29:29 +0000 (15:29 -0500)]
un-deprecate Oracle 2pc
Oracle two-phase transactions at a rudimentary level are now no longer
deprecated. After receiving support from cx_Oracle devs we can provide for
basic xid + begin/prepare support with some limitations, which will work
more fully in an upcoming release of cx_Oracle. Two phase "recovery" is not
currently supported.
Mike Bayer [Sun, 17 Jan 2021 18:35:02 +0000 (13:35 -0500)]
Implement support for functions as FROM with columns clause support
Implemented support for "table valued functions" along with additional
syntaxes supported by PostgreSQL, one of the most commonly requested
features. Table valued functions are SQL functions that return lists of
values or rows, and are prevalent in PostgreSQL in the area of JSON
functions, where the "table value" is commonly referred towards as the
"record" datatype. Table valued functions are also supported by Oracle and
SQL Server.
Moved from I5b093b72533ef695293e737eb75850b9713e5e03 due
to accidental push
Mike Bayer [Wed, 3 Feb 2021 16:53:18 +0000 (11:53 -0500)]
Implement per-connection logging token
Added new execution option
:paramref:`_engine.Connection.execution_options.logging_token`. This option
will add an additional per-message token to log messages generated by the
:class:`_engine.Connection` as it executes statements. This token is not
part of the logger name itself (that part can be affected using the
existing :paramref:`_sa.create_engine.logging_name` parameter), so is
appropriate for ad-hoc connection use without the side effect of creating
many new loggers. The option can be set at the level of
:class:`_engine.Connection` or :class:`_engine.Engine`.
Mike Bayer [Wed, 3 Feb 2021 15:42:08 +0000 (10:42 -0500)]
Add coercions to literal()
To prevent literal() from receiving a ClauseElement which
then leads to confusing results, add a new LiteralValueRole
coercion that does an _is_literal() check and implement
for literal().
Mike Bayer [Sat, 30 Jan 2021 16:55:22 +0000 (11:55 -0500)]
reorganize mapper compile/teardown under registry
Mapper "configuration", which occurs within the
:func:`_orm.configure_mappers` function, is now organized to be on a
per-registry basis. This allows for example the mappers within a certain
declarative base to be configured, but not those of another base that is
also present in memory. The goal is to provide a means of reducing
application startup time by only running the "configure" process for sets
of mappers that are needed. This also adds the
:meth:`_orm.registry.configure` method that will run configure for the
mappers local in a particular registry only.
Mike Bayer [Sat, 30 Jan 2021 21:57:50 +0000 (16:57 -0500)]
set identifier length for MySQL constraints to 64
The rule to limit index names to 64 also applies to all
DDL names, such as those coming from naming conventions.
Add another limiting variable for constraint names and
create test cases against all constraint types.
Additionally, codified in the test suite MySQL's lack of
support for naming of a FOREIGN KEY constraint after
the name was given, which apparently assigns the name to an
associated KEY but not the constraint itself, until MySQL 8
and MariaDB 10.5 which appear to have resolved the
behavior. However it's not clear how Alembic hasn't had
issues reported with this so far.
Fixed long-lived bug in MySQL dialect where the maximum identifier length
of 255 was too long for names of all types of constraints, not just
indexes, all of which have a size limit of 64. As metadata naming
conventions can create too-long names in this area, apply the limit to the
identifier generator within the DDL compiler.
Gord Thompson [Mon, 25 Jan 2021 18:24:25 +0000 (11:24 -0700)]
Use schema._copy_expression() fully in column collection constraints
Fixed issue where using :meth:`_schema.Table.to_metadata` (called
:meth:`_schema.Table.tometadata` in 1.3) in conjunction with a PostgreSQL
:class:`_postgresql.ExcludeConstraint` that made use of ad-hoc column
expressions would fail to copy correctly.
Mike Bayer [Fri, 29 Jan 2021 18:43:38 +0000 (13:43 -0500)]
Clarify Column.index / Column.unique parameters
These parameters need to be more clear that they cause a
constraint / index object to be generated. Clarify the rules
by which this occurs and include contextual information about
naming conventions as well.
Mike Bayer [Thu, 28 Jan 2021 19:53:02 +0000 (14:53 -0500)]
Render NULL for bindparam w/ None value/literal_binds, warn
Adjusted the "literal_binds" feature of :class:`_sql.Compiler` to render
NULL for a bound parameter that has ``None`` as the value, either
explicitly passed or omitted. The previous error message "bind parameter
without a renderable value" is removed, and a missing or ``None`` value
will now render NULL in all cases. Previously, rendering of NULL was
starting to happen for DML statements due to internal refactorings, but was
not explicitly part of test coverage, which it now is.
While no error is raised, when the context is within that of a column
comparison, and the operator is not "IS"/"IS NOT", a warning is emitted
that this is not generally useful from a SQL perspective.
Mike Bayer [Thu, 28 Jan 2021 16:04:29 +0000 (11:04 -0500)]
Allow Oracle CLOB/NCLOB/BLOB in returning
Fixed bug in Oracle dialect where retriving a CLOB/BLOB column via
:meth:`_dml.Insert.returning` would fail as the LOB value would need to be
read when returned; additionally, repaired support for retrieval of Unicode
values via RETURNING under Python 2.
As of yet, we still don't know how to reproduce the
ORA-24813 error indicated in the issue.
Gord Thompson [Sat, 9 Jan 2021 21:56:38 +0000 (14:56 -0700)]
Replace with_labels() and apply_labels() in ORM/Core
Replace :meth:`_orm.Query.with_labels` and
:meth:`_sql.GenerativeSelect.apply_labels` with explicit getters and
setters ``get_label_style`` and ``set_label_style`` to accommodate the
three supported label styles: ``LABEL_STYLE_DISAMBIGUATE_ONLY`` (default),
``LABEL_STYLE_TABLENAME_PLUS_COL``, and ``LABEL_STYLE_NONE``.
In addition, for Core and "future style" ORM queries,
``LABEL_STYLE_DISAMBIGUATE_ONLY`` is now the default label style. This
style differs from the existing "no labels" style in that labeling is
applied in the case of column name conflicts; with ``LABEL_STYLE_NONE``, a
duplicate column name is not accessible via name in any case.
For legacy ORM queries using :class:`_query.Query`, the table-plus-column
names labeling style applied by ``LABEL_STYLE_TABLENAME_PLUS_COL``
continues to be used so that existing test suites and logging facilities
see no change in behavior by default, however this style of labeling is no
longer required for SQLAlchemy queries to function, as result sets are
commonly matched to columns using a positional approach since SQLAlchemy
1.0.
Within test suites, all use of apply_labels() / use_labels
now uses the new methods. New tests added to
test/sql/test_deprecations.py nad test/orm/test_deprecations.py
to cover just the old apply_labels() method call. Tests
in ORM that made explicit use apply_labels()/ etc. where it isn't needed
for the ORM to work correctly use default label style now.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #4757
Change-Id: I5fdcd2ed4ae8c7fe62f8be2b6d0e8f66409b6a54
Mike Bayer [Mon, 25 Jan 2021 18:22:34 +0000 (13:22 -0500)]
Add more docs on engine connecting w/ URL parameters
We have a special section for connect_args but didn't appropriately
introduce that URL keywords may be used as well, which otherwise
does not seem to be documented in a generalized way. Add
this additional detail and contrast between URL query string
and connect_args including providing some transparency for
how to see what URL query string parameters will do for a given
dialect, as we do not handle these parameters consistently
right now.
Mike Bayer [Sun, 24 Jan 2021 22:26:36 +0000 (17:26 -0500)]
un-ignore mike's favorite testing filenames
As im using vscode I'd like these filenames to show up
in filesearch but I'd like to keep "ignore .gitignore files"
turned on. I've moved these names to my own local
.git/info/exclude instead.
Mike Bayer [Sat, 23 Jan 2021 23:02:17 +0000 (18:02 -0500)]
Remove errant assertion from unit of work
Fixed ORM unit of work regression where an errant "assert primary_key"
statement interferes with primary key generation sequences that don't
actually consider the columns in the table to use a real primary key
constraint, instead using :paramref:`_orm.mapper.primary_key` to establish
certain columns as "primary".
Also remove errant "identity" requirement which does not seem to
represent any current backend and is applied to
test/sql/test_defaults.py->AutoIncrementTest, but these tests work
on all backends.
Federico Caselli [Sun, 24 Jan 2021 19:49:06 +0000 (14:49 -0500)]
Limit AsyncAdaptedQueue to Python 3.7
Tests here are failing for python 3.6 due to the lack
of asyncio.run(). It seems to be non-trivial to vendor
a working version of this in Python 3.6 as the tests here
are running it in alternate threads.
The python documentation imports everything directly from the
asyncio package, and it seems that py < 3.8 does not have the
asyncio.exception module
Fix a couple of bugs in the asyncio implementation
Log an informative message if a connection is not closed
and the gc is reclaiming it when using an async dpapi, that
does not support running IO at that stage.
The ``AsyncAdaptedQueue`` used by default on async dpapis
should instantiate a queue only when it's first used
to avoid binding it to a possibly wrong event loop.
Mike Bayer [Wed, 20 Jan 2021 22:06:14 +0000 (17:06 -0500)]
chain joins from SelectState context, not Select
Fixed issue in new :meth:`_sql.Select.join` method where chaining from the
current JOIN wasn't looking at the right state, causing an expression like
"FROM a JOIN b <onclause>, b JOIN c <onclause>" rather than
"FROM a JOIN b <onclause> JOIN c <onclause>".
Added :meth:`_sql.Select.outerjoin_from` method to complement
:meth:`_sql.Select.join_from`.
Mike Bayer [Tue, 19 Jan 2021 20:14:46 +0000 (15:14 -0500)]
Document Table/Column accessors
As Sphinx will not allow us to add attributes to the
.rst file while maintaining order, these have to be added
as class-level attributes.
Inlcude notes that "index" and "unique" parameters, while
indicated by Column.index / Column.unique, do not actually
indicate if the column is part of an index.
Federico Caselli [Tue, 12 Jan 2021 21:14:38 +0000 (22:14 +0100)]
Disallow non-native psycopg2 Unicode in Python 3; update docs
Fixed issue where the psycopg2 dialect would silently pass the
``use_native_unicode=False`` flag without actually having any effect under
Python 3, as the psycopg2 DBAPI uses Unicode unconditionally under Python
3. This usage now raises an :class:`_exc.ArgumentError` when used under
Python 3. Added test support for Python 2.
Additionally, added documentation for client_encoding parameter
that may be passed to libpq directly via psycopg2.
Mike Bayer [Mon, 18 Jan 2021 21:11:21 +0000 (16:11 -0500)]
restore greenlet req
greenlet 1.0 is now released, so as openstack builds are up again and the upstream pip bug
is fixed, let's now try to restore the requirements we had as we'd
rather not have users confused about why asyncio doesn't work without
additional packages.
Mike Bayer [Sat, 16 Jan 2021 17:39:51 +0000 (12:39 -0500)]
introduce generalized decorator to prevent invalid method calls
This introduces the ``_exclusive_against()`` utility decorator
that can be used to prevent repeated invocations of methods that
typically should only be called once.
An informative error message is now raised for a selected set of DML
methods (currently all part of :class:`_dml.Insert` constructs) if they are
called a second time, which would implicitly cancel out the previous
setting. The methods altered include:
:class:`_sqlite.Insert.on_conflict_do_update`,
:class:`_sqlite.Insert.on_conflict_do_nothing` (SQLite),
:class:`_postgresql.Insert.on_conflict_do_update`,
:class:`_postgresql.Insert.on_conflict_do_nothing` (PostgreSQL),
:class:`_mysql.Insert.on_duplicate_key_update` (MySQL)
Altered the behavior of the :class:`_schema.Identity` construct such that
when applied to a :class:`_schema.Column`, it will automatically imply that
the value of :paramref:`_sql.Column.nullable` should default to ``False``,
in a similar manner as when the :paramref:`_sql.Column.primary_key`
parameter is set to ``True``. This matches the default behavior of all
supporting databases where ``IDENTITY`` implies ``NOT NULL``. The
PostgreSQL backend is the only one that supports adding ``NULL`` to an
``IDENTITY`` column, which is here supported by passing a ``True`` value
for the :paramref:`_sql.Column.nullable` parameter at the same time.
Mike Bayer [Fri, 15 Jan 2021 23:24:56 +0000 (18:24 -0500)]
Guard against re-entrant autobegin in Core, ORM
Fixed bug in "future" version of :class:`.Engine` where emitting SQL during
the :meth:`.EngineEvents.do_begin` event hook would cause a re-entrant
condition due to autobegin, including the recipe documented for SQLite to
allow for savepoints and serializable isolation support.
Fixed issue in new :class:`_orm.Session` similar to that of the
:class:`_engine.Connection` where the new "autobegin" logic could be
tripped into a re-entrant state if SQL were executed within the
:meth:`.SessionEvents.after_transaction_create` event hook.
Also repair the new "testing_engine" pytest fixture to
set up for "future" engine appropriately, which wasn't working
leading to the test_execute.py tests not using the future
engine since recent f1e96cb0874927a475d0c11139.
Mike Bayer [Fri, 15 Jan 2021 22:23:52 +0000 (17:23 -0500)]
run handle error for commit/rollback fail and cancel transaction
Fixed bug in asyncpg dialect where a failure during a "commit" or less
likely a "rollback" should cancel the entire transaction; it's no longer
possible to emit rollback. Previously the connection would continue to
await a rollback that could not succeed as asyncpg would reject it.
Mike Bayer [Wed, 6 Jan 2021 15:43:19 +0000 (10:43 -0500)]
update execute() arg formats in modules and tests
continuing with producing a SQLAlchemy 1.4.0b2 that internally
does not emit any of its own 2.0 deprecation warnings,
migrate the *args and **kwargs passed to execute() methods
that now must be a single list or dictionary.
Alembic 1.5 is again waiting on this internal consistency to
be present so that it can pass all tests with no 2.0
deprecation warnings.
Mike Bayer [Thu, 14 Jan 2021 23:07:14 +0000 (18:07 -0500)]
Create explicit GC ordering between ConnectionFairy/ConnectionRecord
Fixed issue where connection pool would not return connections to the pool
or otherwise be finalized upon garbage collection under pypy if the checked
out connection fell out of scope without being closed. This is a long
standing issue due to pypy's difference in GC behavior that does not call
weakref finalizers if they are relative to another object that is also
being garbage collected. A strong reference to the related record is now
maintained so that the weakref has a strong-referenced "base" to trigger
off of.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
