I completely misread https://www.python.org/dev/peps/pep-0418/#rationale
and the accuracy of monotonic() is *worse* on windows than time.time(),
which is bizarre.
Mike Bayer [Wed, 7 Oct 2020 13:47:46 +0000 (09:47 -0400)]
force a sleep for test_reconnect
as I dont have a windows machine to test I don't really know
how to get a millisecond-accurate timer for windows,
Python documentation claimed time.monotonic() did this however
the continued failure of test_reconnect indicates this is not the case
and that the timer is still bumping up by multi-millisecond
granularity. force a delay instead.
Mike Bayer [Wed, 7 Oct 2020 12:42:48 +0000 (08:42 -0400)]
Use monotonic time for pool age measurement
The internal clock used by the :class:`_pool.Pool` object is now
time.monotonic_time() under Python 3. Under Python 2, time.time() is still
used, which is legacy. This clock is used to measure the age of a
connection against its starttime, and used in comparisons against the
pool_timeout setting as well as the last time the pool was marked as
invalid to determine if the connection should be recycled. Previously,
time.time() was used which was subject to inaccuracies as a result of
system clock changes as well as poor time resolution on windows.
Mike Bayer [Wed, 7 Oct 2020 04:50:53 +0000 (00:50 -0400)]
limit "no identity" test to a hardcoded dialect
this test can't require "skip identity_columns" because
older Postgresql and Oracle report false for "identity_columns",
but their dialects won't skip actually rendering.
for now the only option is to hardcode to a non-identity
dialect.
Mike Bayer [Wed, 7 Oct 2020 03:48:47 +0000 (23:48 -0400)]
add --notimingintensive; block from github jobs
this provides a front-end option to disable tests marked
as timing_intensive, all of which are in test_pool, which are more
fragile and aren't consistent on the
github runners. also remove /reduce unnecessary time.sleep()
from two other pool tests that are not timing intensive.
note that this removes test_hanging_connect_within_overflow
from the github runs via the timing_intensive requirement.
I've also removed MockReconnectTest from exclusions as those are
really important tests and they use mocks so should not have
platform dependent issues. Need to see what the
windows failures are.
Fixed bug where a call ``InstanceState.obj()`` could return None when
synchronizing the instance states of the objects in the session in case
they become out of scope but are not yet finalized by the gc. This
case does not happen in cPython, but it may present itself in pypy.
The approach is to allow None to be gracefully handled by the
evaluator itself, ensuring it returns None in all cases when None
is passed in.
Mike Bayer [Wed, 7 Oct 2020 03:46:03 +0000 (23:46 -0400)]
modernize requirements for boolean constraint test
this test was not actually working correctly against
mariadb and was "failing" (i.e. "passing") inappropriately
for pymysql, mysqlclient. it also started "passing" (i.e. "failing")
for mariadb as of 1.0.3.
modernize reqs here including for mssql bit behavior.
Add support to ``FETCH {FIRST | NEXT} [ count ] {ROW | ROWS}
{ONLY | WITH TIES}`` in the select for the supported backends,
currently PostgreSQL, Oracle and MSSQL.
Mike Bayer [Wed, 30 Sep 2020 12:37:57 +0000 (08:37 -0400)]
raise on lower-case column shared to multiple tables
Fixed bug where an error was not raised for lower-case
:func:`_column` added to lower-case :func:`_table` object. This now raises
:class:`_exc.ArgumentError` which has always been the case for
upper-case :class:`_schema.Column` and :class:`_schema.Table`.
Mike Bayer [Tue, 29 Sep 2020 18:17:42 +0000 (14:17 -0400)]
Scan for tables without relying upon whereclause
Fixed bug where an UPDATE statement against a JOIN using MySQL multi-table
format would fail to include the table prefix for the target table if the
statement had no WHERE clause, as only the WHERE clause were scanned to
detect a "multi table update" at that particular point. The target
is now also scanned if it's a JOIN to get the leftmost table as the
primary table and the additional entries as additional FROM entries.
Mike Bayer [Tue, 29 Sep 2020 19:44:33 +0000 (15:44 -0400)]
bump variance on test_string, test_unicode
a recent rerun of profiles added more profiling data that's
failing over small differences. 15% variance is fine for these
tests that are looking for thousands of encode calls.
Added support for reflecting "identity" columns, which are now returned
as part of the structure returned by :meth:`_reflection.Inspector.get_columns`.
When reflecting full :class:`_schema.Table` objects, identity columns will
be represented using the :class:`_schema.Identity` construct.
Fixed compilation error on oracle for sequence and identity column
``nominvalue`` and ``nomaxvalue`` options that require no space in them.
Improved test compatibility with oracle 18.
As part of the support for reflecting :class:`_schema.Identity` objects,
the method :meth:`_reflection.Inspector.get_columns` no longer returns
``mssql_identity_start`` and ``mssql_identity_increment`` as part of the
``dialect_options``. Use the information in the ``identity`` key instead.
The mssql dialect will assume that at least MSSQL 2005 is used.
There is no hard exception raised if a previous version is detected,
but operations may fail for older versions.
The :meth:`_sql.Join.alias` method is deprecated and will be removed in
SQLAlchemy 2.0. An explicit select + subquery, or aliasing of the inner
tables, should be used instead.
Mike Bayer [Mon, 28 Sep 2020 02:40:09 +0000 (22:40 -0400)]
dont use uninstrument event to dispose registry entry
since 450f5c0d6519a439f4025c3892fe4c we've been seeing
errors during the uninstrument_class event where first
we saw an internal weakref being collected earlier than seen,
then fixing that we saw the listener collection changing during
iteration for similar reasons.
we would assume the issue is because of the interaction between
mapper / instrumentation/ registry during a test teardown
and the usage of the uninstrument_class event within this
interaction. this interaction is too fundamental to be
relying upon this event in any case and when I wrote this
new code i was planning on changing that part in any case,
I just forgot.
Mike Bayer [Mon, 28 Sep 2020 03:18:57 +0000 (23:18 -0400)]
build the full compilestate every time
the ORMSelectCompileState was trying to get away with
not building out the "froms" list of the state, but we need
this for select.froms. Build this out and add some tests
for select(), including some other state-oriented use cases.
Mike Bayer [Sun, 27 Sep 2020 15:06:48 +0000 (11:06 -0400)]
Repair erroneous "future" symbol
the change in 1e800285508ecd869c6874fed failed to fully
remove the "future" symbol which then got confused against the
import of the "future" package itself, which is also not needed.
remove it entirely.
pin pytest < 6.1 to see if new error condition may be avoided
Mike Bayer [Thu, 17 Sep 2020 22:15:42 +0000 (18:15 -0400)]
new docs WIP
This WIP is part of the final push for 1.4's docs
to fully "2.0-ize" what we can, and have it all ready.
So far this includes a rewrite of the 2.0 migration,
set up for the 1.4 /2.0 docs style, and a total redesign
of the index page using a new flex layout in zzzeeksphinx.
It also reworks some of the API reference sections
to have more subheaders. zzzeeksphinx is also enhanced
to provide automatic summaries for all api doc section.
Set default type codec for ``json`` and ``jsonb`` types when using
the asyncpg driver. By default asyncpg will not decode them and return
strings instead.
Support for multiple hosts in PostgreSQL connection string
Provide support for multiple hosts in the PostgreSQL connection string.
A user requested for SQLAlchemy to support multiple hosts within a PostgreSQL URL string. The proposed fix allows this. In the event that the url contains multiple hosts the proposed code will convert the query["hosts"] tuple into a single string. This allows the hosts to then get converted into a valid dsn variable in the psycopg2 connect function.
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 22 Sep 2020 14:26:35 +0000 (10:26 -0400)]
Deprecate negative slice indexes
The "slice index" feature used by :class:`_orm.Query` as well as by the
dynamic relationship loader will no longer accept negative indexes in
SQLAlchemy 2.0. These operations do not work efficiently and load the
entire collection in, which is both surprising and undesirable. These
will warn in 1.4 unless the :paramref:`_orm.Session.future` flag is set in
which case they will raise IndexError.
Mike Bayer [Mon, 21 Sep 2020 21:28:03 +0000 (17:28 -0400)]
Raise if unique() not applied to 2.0 joined eager load results
The automatic uniquing of rows on the client side is turned off for the new
:term:`2.0 style` of ORM querying. This improves both clarity and
performance. However, uniquing of rows on the client side is generally
necessary when using joined eager loading for collections, as there
will be duplicates of the primary entity for each element in the
collection because a join was used. This uniquing must now be manually
enabled and can be achieved using the new
:meth:`_engine.Result.unique` modifier. To avoid silent failure, the ORM
explicitly requires the method be called when the result of an ORM
query in 2.0 style makes use of joined load collections. The newer
:func:`_orm.selectinload` strategy is likely preferable for eager loading
of collections in any case.
This changeset also fixes an issue where ORM-style "single entity"
results would not apply unique() correctly if results were returned
as tuples.
Stringify correctly for non-str exception argument
Fixed issue where a non-string object sent to
:class:`_exc.SQLAlchemyError` or a subclass, as occurs with some third
party dialects, would fail to stringify correctly. Pull request
courtesy Andrzej Bartosiński.
Mike Bayer [Fri, 18 Sep 2020 17:29:42 +0000 (13:29 -0400)]
Complete deprecation of from_self()
For most from_self() tests, move them into
test/orm/test_deprecated.py and replace the existing
test with one that uses aliased() plus a subquery.
This then revealed a few more issues.
Related items:
* Added slice() method to GenerativeSelect, to match that
of orm.Query and to make possible migration of one of the
from_self() tests. moved the utility functions used for this
from orm/util into sql/util.
* repairs a caching issue related to subqueryload
where information being derived from the cached path info
was mixing up with query information based on the per-query
state, specifically an AliasedClass that is per query.
* for the above issue, it seemed like path_registry maybe
had to change so that it represents AliasedClass objects
as their cache key rather than on identity, but it wasn't
needed. still seems like it would be more correct.
* enhances the error message raised by coercions for a case
such as when an AliasedClass holds onto a select() object
and not a subquery(); will name the original and resolved
object for clarity (although how is AliasedClass able to
accept a Select() object in the first place?)
* Added _set_propagate_attrs() to Query so that again if
it's passed to AliasedClass, it doesn't raise an error
during coercion, but again maybe that should also be
rejected up front
Mike Bayer [Mon, 17 Aug 2020 21:24:27 +0000 (17:24 -0400)]
Create a framework to allow all SQLALCHEMY_WARN_20 to pass
As the test suite has widespread use of many patterns
that are deprecated, enable SQLALCHEMY_WARN_20 globally
for the test suite but then break the warnings filter
out into a whole list of all the individual warnings
we are looking for. this way individual changesets
can target a specific class of warning, as many of these
warnings will indivdidually affect dozens of files
and potentially hundreds of lines of code.
Many warnings are also resolved here as this
patch started out that way. From this point
forward there should be changesets that target a
subset of the warnings at a time.
For expediency, updates some migration 2.0 docs
for ORM as well.
Mike Bayer [Wed, 16 Sep 2020 12:26:14 +0000 (08:26 -0400)]
Don't change asyncpg's "char" codec
This codec was used to ensure the "pg_attribute.generated"
column comes back as a string and not bytes, matching how
other PG drivers treat this datatype. However, this breaks
asyncpg's internal implementation of set_type_codec going forward
and the "char" datatype is actually a bytes in any case.
So at the moment it appears psycopg2/pg8000 are broken for mis-treatment
of the datatype and asyncpg is broken in that it was allowing
us to change a codec that it appears to rely upon internally.
Mike Bayer [Tue, 15 Sep 2020 22:48:36 +0000 (18:48 -0400)]
Correct for SQL Server temp table owner
on my machine, the owner for a temp table comes out as
dbo, and i am testing against a CI machine. im not sure
what happens on a CI machine except perhaps that it provisions
new databases is changing things. in any case, since we
are searching the tempdb for the name, get the schema/owner also.
Also refines the test to use a single connection and a transaction
that rolls back, doesn't hang here but let's see what CI does.
Remove silent ignore for skip_locked w/ unsupported backends
For SQLAlchemy 1.4:
The "skip_locked" keyword used with ``with_for_update()`` will render "SKIP
LOCKED" on all MySQL backends, meaning it will fail for MySQL less than
version 8 and on current MariaDB backends. This is because those backends
do not support "SKIP LOCKED" or any equivalent, so this error should not be
silently ignored. This is upgraded from a warning in the 1.3 series.
For SQLAlchemy 1.3:
The "skip_locked" keyword used with ``with_for_update()`` will emit a
warning when used on MariaDB backends, and will then be ignored. This is
a deprecated behavior that will raise in SQLAlchemy 1.4, as an application
that requests "skip locked" is looking for a non-blocking operation which
is not available on those backends.
Mike Bayer [Mon, 14 Sep 2020 14:14:48 +0000 (10:14 -0400)]
Pass all pool parameters in recreate()
The following pool parameters were not being propagated to the new pool
created when :meth:`_engine.Engine.dispose` were called: ``pre_ping``,
``use_lifo``. Additionally the ``recycle`` and ``reset_on_return``
parameters were not propagated for the :class:`_engine.AssertionPool`
class. These issues have been fixed.
Several operators are renamed to achieve more consistent naming across
SQLAlchemy.
The operator changes are:
* `isnot` is now `is_not`
* `not_in_` is now `not_in`
Because these are core operators, the internal migration strategy for this
change is to support legacy terms for an extended period of time -- if not
indefinitely -- but update all documentation, tutorials, and internal usage
to the new terms. The new terms are used to define the functions, and
the legacy terms have been deprecated into aliases of the new terms.
Mike Bayer [Sun, 13 Sep 2020 17:15:29 +0000 (13:15 -0400)]
Update session.execute() and related documentation
The docs here were completely out of date and referred
to behaviors that are no longer true, behaviors that are
deprecated, etc. For the moment, take out all the verbiage
so that nothing incorrect is present. New ORM documentation
will need to be constructed to support this statement.
Mike Bayer [Sun, 13 Sep 2020 14:36:16 +0000 (10:36 -0400)]
Deprecate engine-wise ss cursors; repair mariadbconnector
The server_side_cursors engine-wide feature relies upon
regexp parsing of statements a well as general guessing as
to when the feature should be used. This is not within the
2.0 way of doing things and should be removed.
Additionally, mariadbconnector defaults to unbuffered cursors;
add new cursor hooks so that mariadbconnector can specify
buffered or unbuffered cursors without too much difficulty.
This will also correctly default mariadbconnector to buffered
cursors which should repair the segfaults we've been getting.
Try to restore the assert_raises that was removed in 5b6dfc0c38bf1f01da4b8 to see if mariadbconnector segfaults
are resolved.
Mike Bayer [Sat, 12 Sep 2020 20:07:50 +0000 (16:07 -0400)]
Ensure cursor is closed for scalar() if make_row fails
As we have some tests that are against enums which
can raise on fetch, if we call scalar() and it fails,
we need to close the cursor. mariadb segfaults
etc. seem to have been caused by this.
Improved support for covering indexes (with INCLUDE columns). Added the
ability for postgresql to render CREATE INDEX statements with an INCLUDE
clause from Core. Index reflection also report INCLUDE columns separately
for both mssql and postgresql (11+).
RamonWill [Thu, 20 Aug 2020 19:05:39 +0000 (15:05 -0400)]
Reflect mssql/postgresql filtered/partial indexes
Added support for inspection / reflection of partial indexes / filtered
indexes, i.e. those which use the ``mssql_where`` or ``postgresql_where``
parameters, with :class:`_schema.Index`. The entry is both part of the
dictionary returned by :meth:`.Inspector.get_indexes` as well as part of a
reflected :class:`_schema.Index` construct that was reflected. Pull
request courtesy Ramon Williams.
Mike Bayer [Fri, 11 Sep 2020 14:50:58 +0000 (10:50 -0400)]
Emit deprecation warning for **kw passed to session.execute()
Passing keyword arguments to methods such as :meth:`_orm.Session.execute`
to be passed into the :meth:`_orm.Session.get_bind` method is deprecated;
the new :paramref:`_orm.Session.execute.bind_arguments` dictionary should
be passed instead.
Mike Bayer [Mon, 31 Aug 2020 15:46:55 +0000 (11:46 -0400)]
Build out new declarative systems; deprecate mapper()
The ORM Declarative system is now unified into the ORM itself, with new
import spaces under ``sqlalchemy.orm`` and new kinds of mappings. Support
for decorator-based mappings without using a base class, support for
classical style-mapper() calls that have access to the declarative class
registry for relationships, and full integration of Declarative with 3rd
party class attribute systems like ``dataclasses`` and ``attrs`` is now
supported.
Mike Bayer [Thu, 10 Sep 2020 15:56:34 +0000 (11:56 -0400)]
Add more docs for populate_existing(); link with_for_update
The populate_existing() method is actually changing
to be an execution option, however it has almost no
mention in the narrative docs so add docs in terms of the
1.x version first, including that we mention you almost
definitely want to use this method if you are also using
with_for_update().
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
