]>
git.ipfire.org Git - thirdparty/psycopg.git/log
Daniele Varrazzo [Thu, 11 Mar 2021 14:31:55 +0000 (15:31 +0100)]
Don't use a worker in pool.putconn() if not needed
If _reset is not set there is little work to do so the overhead of a
context switch is measurable and not necessary.
Daniele Varrazzo [Wed, 10 Mar 2021 23:40:33 +0000 (00:40 +0100)]
pool: fix off-by-one error in shrink log message
Daniele Varrazzo [Wed, 10 Mar 2021 18:34:00 +0000 (19:34 +0100)]
Add async pool stats
Daniele Varrazzo [Wed, 10 Mar 2021 02:39:46 +0000 (03:39 +0100)]
Add pool connection_class parameter
Daniele Varrazzo [Wed, 10 Mar 2021 02:32:02 +0000 (03:32 +0100)]
Make pool params keyword only in subclasses too
Daniele Varrazzo [Wed, 10 Mar 2021 02:22:07 +0000 (03:22 +0100)]
Add pool stats
Daniele Varrazzo [Mon, 8 Mar 2021 03:43:33 +0000 (04:43 +0100)]
Add reset callback to connection pool
Daniele Varrazzo [Mon, 8 Mar 2021 03:18:25 +0000 (04:18 +0100)]
Make sure that the pool config function leaves connections in idle state
Daniele Varrazzo [Mon, 8 Mar 2021 02:32:34 +0000 (03:32 +0100)]
Rename pool.wait_ready() to pool.wait()
Daniele Varrazzo [Mon, 8 Mar 2021 02:26:12 +0000 (03:26 +0100)]
Drop especially flaky async test dealing with del behaviour
Daniele Varrazzo [Mon, 8 Mar 2021 02:25:57 +0000 (03:25 +0100)]
Add AsyncConnectionPool.check()
Daniele Varrazzo [Mon, 8 Mar 2021 01:59:05 +0000 (02:59 +0100)]
Simpler, more approximative, more reliable pool max lifetime test
Daniele Varrazzo [Mon, 8 Mar 2021 01:41:53 +0000 (02:41 +0100)]
More tolerance in schedule timing tests
Daniele Varrazzo [Mon, 8 Mar 2021 01:28:14 +0000 (02:28 +0100)]
Add test for pool config
Daniele Varrazzo [Mon, 8 Mar 2021 01:24:09 +0000 (02:24 +0100)]
Don't throw an error on context exit if the connection is closed
Daniele Varrazzo [Mon, 8 Mar 2021 01:05:09 +0000 (02:05 +0100)]
Make the async pool unavailable on Python 3.6
Because of https://bugs.python.org/issue42600 or thereabout.
Daniele Varrazzo [Sun, 7 Mar 2021 23:52:58 +0000 (00:52 +0100)]
Don't use threads to manage the scheduler and workers of async tasks
Daniele Varrazzo [Sat, 27 Feb 2021 14:28:18 +0000 (15:28 +0100)]
Add an async scheduler
Daniele Varrazzo [Mon, 8 Mar 2021 00:07:53 +0000 (01:07 +0100)]
Drop get_running_loop call on async pool close
Daniele Varrazzo [Sat, 6 Mar 2021 04:02:29 +0000 (05:02 +0100)]
Add compat module for python versions compatibility
Daniele Varrazzo [Sat, 6 Mar 2021 02:38:45 +0000 (03:38 +0100)]
Add pool.check()
Daniele Varrazzo [Sat, 6 Mar 2021 02:13:37 +0000 (03:13 +0100)]
Set a timeout for the test run
Daniele Varrazzo [Sat, 6 Mar 2021 01:21:30 +0000 (02:21 +0100)]
retry flaky max lifetime test
Daniele Varrazzo [Sun, 28 Feb 2021 15:19:49 +0000 (16:19 +0100)]
Add Connection.broken
Daniele Varrazzo [Sun, 28 Feb 2021 03:35:56 +0000 (04:35 +0100)]
Add max_lifetime to pool connections
Daniele Varrazzo [Sun, 28 Feb 2021 02:55:29 +0000 (03:55 +0100)]
Add jitter method to pool
Daniele Varrazzo [Sat, 27 Feb 2021 22:47:26 +0000 (23:47 +0100)]
Use more of the pool context manager in the tests
Daniele Varrazzo [Sat, 27 Feb 2021 21:32:34 +0000 (22:32 +0100)]
Retry flaky tests a few times
Daniele Varrazzo [Sat, 27 Feb 2021 20:44:07 +0000 (21:44 +0100)]
Add pool context managers
Daniele Varrazzo [Sat, 27 Feb 2021 20:07:38 +0000 (21:07 +0100)]
Add pool.resize()
Daniele Varrazzo [Sat, 27 Feb 2021 11:47:36 +0000 (12:47 +0100)]
Make cleanup tests more reliable
Unfortunately there is some race condition on delete and for now just
using a timer.
In order to limit the problem don't actually run the StopWorker task but
bail out from the workers as soon as seen.
Daniele Varrazzo [Sat, 27 Feb 2021 11:13:17 +0000 (12:13 +0100)]
Fix Python 3.6 compatibility issues in async pool
Daniele Varrazzo [Sat, 27 Feb 2021 01:09:44 +0000 (02:09 +0100)]
Add async connection pool
Daniele Varrazzo [Thu, 25 Feb 2021 21:38:33 +0000 (22:38 +0100)]
Move non-blocking pool functionalities to a generic base class
Note that running maintenance threads is "non-blocking" and will be
thread-based in the async pool.
Daniele Varrazzo [Thu, 25 Feb 2021 20:57:20 +0000 (21:57 +0100)]
Add pool.wait_ready, drop connection_timeout on init
This way init should never block. Always fill the connection in
background but a client can decide if they want to wait for it to be
ready.
Daniele Varrazzo [Thu, 25 Feb 2021 19:36:48 +0000 (20:36 +0100)]
Split the pool package objects in modules
Daniele Varrazzo [Thu, 25 Feb 2021 19:11:56 +0000 (20:11 +0100)]
Move task implementations to the pool object
Daniele Varrazzo [Thu, 25 Feb 2021 17:31:16 +0000 (18:31 +0100)]
Make psycopg3.pool a package
Daniele Varrazzo [Tue, 23 Feb 2021 22:39:44 +0000 (23:39 +0100)]
Use the daemon param at constructor rather than the attribute
Daniele Varrazzo [Mon, 22 Feb 2021 03:20:33 +0000 (04:20 +0100)]
Use the connections in the pool uniformly
I feel this is a better use than using some more than other (e.g. in
term of bloat of the connections associated with the resources) and
gives a more predictable performance of the connection (there won't be
some cold, some hot).
Now there aren't really "unused connections" to single out in order to
shrink the pool. So keep a tally of the number of connections unused and
use a worker thread to close some if there are above minconn unused in a
period.
Daniele Varrazzo [Mon, 22 Feb 2021 01:49:16 +0000 (02:49 +0100)]
Change state in critical section on pool.close()
Nothing bad should happen, because what maintains the state respect the
_closed state. However because the logic is complex make it more
defensive.
Daniele Varrazzo [Mon, 22 Feb 2021 01:33:36 +0000 (02:33 +0100)]
Make any pool task schedulable
Added interface to the pool to schedule a task in the future.
Daniele Varrazzo [Mon, 22 Feb 2021 01:05:14 +0000 (02:05 +0100)]
Don't lose pool connections giving them to a clients already timed out
Daniele Varrazzo [Sun, 21 Feb 2021 13:04:10 +0000 (14:04 +0100)]
Allow proxy tests to fail on Travis
Don't know why it fails, it requires interactive investigation there.
Make sure tests can run connecting on TCP socket and avoid SSH in the
proxy anyway. Not that any of this worked...
Daniele Varrazzo [Sun, 21 Feb 2021 12:41:02 +0000 (13:41 +0100)]
Wait worker thread to stop on pool close
Daniele Varrazzo [Sun, 21 Feb 2021 12:07:58 +0000 (13:07 +0100)]
Don't print warning gc'ing a pool in incomplete state
Daniele Varrazzo [Sun, 21 Feb 2021 03:48:03 +0000 (04:48 +0100)]
pool.close() called on del, idemptotent
Daniele Varrazzo [Sun, 21 Feb 2021 03:47:20 +0000 (04:47 +0100)]
Add separate setup_timeout param for pool
Allow 0 too, which would fill the pool in background.
Daniele Varrazzo [Sun, 21 Feb 2021 03:28:01 +0000 (04:28 +0100)]
Drop _sec suffix from pool timeouts
Python is consistent enough to use seconds everywhere timeouts are
involved.
Daniele Varrazzo [Sun, 21 Feb 2021 03:26:00 +0000 (04:26 +0100)]
Add pool reconnection retry
Daniele Varrazzo [Sun, 21 Feb 2021 03:20:38 +0000 (04:20 +0100)]
Don't try to rollback on exit if the connection is bad
Daniele Varrazzo [Sun, 21 Feb 2021 01:17:41 +0000 (02:17 +0100)]
Use a weak reference to avoid loops between pool and maintenance tasks
Daniele Varrazzo [Sat, 20 Feb 2021 23:07:10 +0000 (00:07 +0100)]
Add scheduler class
To be used in the connection pool to implement reconnection with
backoff, maybe periodic tasks.
Daniele Varrazzo [Sat, 20 Feb 2021 03:36:13 +0000 (04:36 +0100)]
Add test for pool kwargs
Daniele Varrazzo [Sat, 20 Feb 2021 03:31:02 +0000 (04:31 +0100)]
Default number of pool workers bumped to 3
Daniele Varrazzo [Sat, 20 Feb 2021 03:16:27 +0000 (04:16 +0100)]
Shrink the pool when connections have been idle long enough
Pool behaviour on start changed: block on __init__ until minconn
connections have been obtained or raise PoolTimeout if timeout_sec have
passed. Not doing so makes complicated to understand, when a connection
is requested, if it's done during initialisation, and avoid an unneeded
grow request.
Daniele Varrazzo [Sat, 20 Feb 2021 00:47:00 +0000 (01:47 +0100)]
Don't waste a worker thread adding a connection to the pool
Daniele Varrazzo [Sat, 20 Feb 2021 00:19:35 +0000 (01:19 +0100)]
Force pool num_workers to be > 0
it might work with num_workers = 0, however it's not trivial, there is
some deadlock to avoid at creation time.
Daniele Varrazzo [Fri, 19 Feb 2021 23:35:27 +0000 (00:35 +0100)]
Allow the pool to grow
Daniele Varrazzo [Sun, 14 Feb 2021 02:33:19 +0000 (03:33 +0100)]
Make the pool maintenance tasks base class abstract
Daniele Varrazzo [Sun, 14 Feb 2021 02:27:30 +0000 (03:27 +0100)]
Delete an eventual task-pool loop in StopWorker too
Daniele Varrazzo [Sun, 14 Feb 2021 02:25:56 +0000 (03:25 +0100)]
Add pool docstrings
Daniele Varrazzo [Sun, 14 Feb 2021 01:22:20 +0000 (02:22 +0100)]
Add connection pool close()
When the pool is closed, raise an exception in the thread of the clients
already waiting and refuse new requests. Let the current request finish
anyway.
Daniele Varrazzo [Sat, 13 Feb 2021 23:45:50 +0000 (00:45 +0100)]
Make sure the pool can be deleted with no warning
Make sure to delete reference loops between the pool and the maintenance
tasks after they have run.
Do not raise a warning if a connection in a pool is deleted without
being closed as this is a normal condition (imagining a pool being
created as a global object).
Daniele Varrazzo [Sat, 13 Feb 2021 22:51:02 +0000 (23:51 +0100)]
Add pool context test
Daniele Varrazzo [Sat, 13 Feb 2021 22:42:00 +0000 (23:42 +0100)]
Awake the pool client outside the critical section
Daniele Varrazzo [Sat, 13 Feb 2021 22:07:54 +0000 (23:07 +0100)]
Reset the connection status returning it to the pool
Daniele Varrazzo [Sat, 13 Feb 2021 17:02:08 +0000 (18:02 +0100)]
Add basic pool functionality test for queuing and timeout
Daniele Varrazzo [Sat, 13 Feb 2021 16:17:28 +0000 (17:17 +0100)]
Display the name in the connection pool repr
Daniele Varrazzo [Sat, 13 Feb 2021 03:06:54 +0000 (04:06 +0100)]
Add ConnectionPool class sketch
It has a pool of connections alright, workers, clients waiting list.
Only minconn implemented, no maintenance thread etc. It's a start.
Daniele Varrazzo [Mon, 8 Mar 2021 21:37:50 +0000 (22:37 +0100)]
@logilab is backing psycopg3!
Merci beaucoup!
Daniele Varrazzo [Mon, 8 Mar 2021 21:29:45 +0000 (22:29 +0100)]
Welcoming @dalibo as backer
Vive la France!
Daniele Varrazzo [Sat, 6 Mar 2021 02:08:54 +0000 (03:08 +0100)]
Fix concurrency test
It was blocking because the committer was a too busy loop never yielding
to the other worker.
Daniele Varrazzo [Sat, 6 Mar 2021 02:08:33 +0000 (03:08 +0100)]
Use gather instead of wait in tests to be coro friendly
Daniele Varrazzo [Sat, 6 Mar 2021 01:09:39 +0000 (02:09 +0100)]
Bump mypy version
Daniele Varrazzo [Sat, 6 Mar 2021 00:26:55 +0000 (01:26 +0100)]
ci: get token to rebuild docs from secrets
Daniele Varrazzo [Sat, 6 Mar 2021 00:13:59 +0000 (01:13 +0100)]
Merge branch 'ci-gh-actions'
Daniele Varrazzo [Tue, 2 Mar 2021 11:38:05 +0000 (12:38 +0100)]
Yaml lint
Daniele Varrazzo [Fri, 5 Mar 2021 17:46:30 +0000 (18:46 +0100)]
Run leak tests with both text and binary output
At the moment the output text is actually skipped but it will run once
text input is no more xfailed.
Denis Laxalde [Tue, 2 Mar 2021 11:12:31 +0000 (12:12 +0100)]
Fix binary argument of conn.cursor() in tests
Argument 'binary' of connection.cursor() should be a bool.
This follows up on
467d20bd0315a62b4c7377d8a0b170ece08ef99c (change from
'format' to 'binary').
Daniele Varrazzo [Tue, 2 Mar 2021 02:10:07 +0000 (03:10 +0100)]
ci: consistent uppercase in steps titles
Daniele Varrazzo [Tue, 2 Mar 2021 01:49:30 +0000 (02:49 +0100)]
ci: Test with Python 3.9 and PostgreSQL 13
Test different combination of Python and Postgres in the different
implementation to cover more cases.
Daniele Varrazzo [Tue, 2 Mar 2021 01:41:39 +0000 (02:41 +0100)]
ci: Allow lint and test on any branch
Daniele Varrazzo [Tue, 2 Mar 2021 01:30:53 +0000 (02:30 +0100)]
ci: colour tests output
Daniele Varrazzo [Tue, 2 Mar 2021 01:28:50 +0000 (02:28 +0100)]
tox: use the github action "python" value as toxenv name
Daniele Varrazzo [Tue, 2 Mar 2021 00:55:16 +0000 (01:55 +0100)]
ci; Use PG* env vars to define the connection string
Daniele Varrazzo [Tue, 2 Mar 2021 00:23:11 +0000 (01:23 +0100)]
ci: Fix test dsn
The github runner /etc/host file is pretty broken and localhost resolves
to ipv6, so use 127.0.0.1 to connect.
Riccardo Magliocchetti [Mon, 1 Mar 2021 08:56:13 +0000 (09:56 +0100)]
ci: sort out connection to postgresql service
Riccardo Magliocchetti [Mon, 1 Mar 2021 08:37:33 +0000 (09:37 +0100)]
ci: split tests in one job for each implementation
To simplify the matrix
Riccardo Magliocchetti [Sun, 28 Feb 2021 19:05:24 +0000 (20:05 +0100)]
ci: move tests to gh actions
Riccardo Magliocchetti [Sun, 28 Feb 2021 18:26:56 +0000 (19:26 +0100)]
ci: move docs building to gh actions
Riccardo Magliocchetti [Sun, 28 Feb 2021 18:26:27 +0000 (19:26 +0100)]
ci: move lint task to gh actions
Daniele Varrazzo [Mon, 1 Mar 2021 00:38:37 +0000 (01:38 +0100)]
Update script to trigger docs rebuild to use travis-ci.com
Daniele Varrazzo [Sun, 28 Feb 2021 14:12:18 +0000 (15:12 +0100)]
Retry server-side cursor tests on del
Warnings seem unreliably to receive. Also print any eventual unexpected
warning: pytest doesn't do it automatically.
Daniele Varrazzo [Sun, 28 Feb 2021 14:11:13 +0000 (15:11 +0100)]
Add fixture to help retrying flaky tests
Tried to use a couple of pytest plugins for retrying but they don't work
well with asyncio.
Daniele Varrazzo [Sun, 28 Feb 2021 02:17:54 +0000 (03:17 +0100)]
Make the DBAPI errors appear defined in the psycopg3 module
Daniele Varrazzo [Sun, 28 Feb 2021 02:15:30 +0000 (03:15 +0100)]
Drop psycopg3.pq.PQerror
Use an OperationalError instead (of which PQerror was a subclass), which
is better documented.
Daniele Varrazzo [Sun, 28 Feb 2021 01:35:23 +0000 (02:35 +0100)]
Add doc example about how to detect a disconnection
Daniele Varrazzo [Sun, 28 Feb 2021 00:36:34 +0000 (01:36 +0100)]
Add Connection.fileno()
Daniele Varrazzo [Sun, 21 Feb 2021 01:16:28 +0000 (02:16 +0100)]
Fix return error without exception on PQsocket call of broken connection
Still some weirdness here around: the method raises an exception on my
box (both with unix and tcp socket) but it seems to raise still a valid
number on certain databases on Travis. Make sure, in the test, at least
that it is a reasonable value.