]> git.ipfire.org Git - thirdparty/psycopg.git/log
thirdparty/psycopg.git
4 years agoRe-export names from psycopg3.types 44/head
Denis Laxalde [Wed, 7 Apr 2021 14:08:04 +0000 (16:08 +0200)] 
Re-export names from psycopg3.types

This is useful to silent mypy errors when implicit reexport is disabled
(e.g. in strict mode) like:

    error: Module 'psycopg3.types' does not explicitly export attribute
    'Jsonb'; implicit reexport disabled  [attr-defined]

https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-no-implicit-reexport

Accordingly, we import Int4 from psycopg3.types directly in
test_copy.py.

4 years agoShorten tracebacks executing on connection too
Daniele Varrazzo [Fri, 26 Mar 2021 10:49:46 +0000 (10:49 +0000)] 
Shorten tracebacks executing on connection too

4 years agoCut traceback on execute short
Daniele Varrazzo [Thu, 18 Mar 2021 02:27:33 +0000 (03:27 +0100)] 
Cut traceback on execute short

Users don't care where in psycopg the error comes.

4 years agoDump all string subclasses as text by default
Daniele Varrazzo [Wed, 17 Mar 2021 17:39:44 +0000 (18:39 +0100)] 
Dump all string subclasses as text by default

Useful for enums.

4 years agoFix dump of enum subtypes
Daniele Varrazzo [Wed, 17 Mar 2021 17:10:04 +0000 (18:10 +0100)] 
Fix dump of enum subtypes

4 years agoDrop outdated change behaviour from docs
Daniele Varrazzo [Wed, 17 Mar 2021 00:08:08 +0000 (01:08 +0100)] 
Drop outdated change behaviour from docs

4 years agoGrazie tante per il vostro aiuto, @20tab!
Daniele Varrazzo [Tue, 16 Mar 2021 13:38:40 +0000 (14:38 +0100)] 
Grazie tante per il vostro aiuto, @20tab!

4 years agoAdded sentry (@getsentry) as a backer
Daniele Varrazzo [Tue, 16 Mar 2021 01:01:09 +0000 (02:01 +0100)] 
Added sentry (@getsentry) as a backer

Thank you very much :)

4 years agoAdded bit.io (@bitdotioinc) as top sponsor
Daniele Varrazzo [Tue, 16 Mar 2021 00:59:17 +0000 (01:59 +0100)] 
Added bit.io (@bitdotioinc) as top sponsor

Amazing, thank you!

4 years agoTypo
Daniele Varrazzo [Fri, 12 Mar 2021 18:21:35 +0000 (19:21 +0100)] 
Typo

4 years agoFix typo in pool with usage
Daniele Varrazzo [Fri, 12 Mar 2021 17:43:12 +0000 (18:43 +0100)] 
Fix typo in pool with usage

4 years agoMerge branch 'connection-pool'
Daniele Varrazzo [Fri, 12 Mar 2021 15:57:17 +0000 (16:57 +0100)] 
Merge branch 'connection-pool'

4 years agoBetter looking default params for pool in the docs
Daniele Varrazzo [Fri, 12 Mar 2021 15:47:37 +0000 (16:47 +0100)] 
Better looking default params for pool in the docs

4 years agoRename pool minconn, maxconn params to min_size, max_size
Daniele Varrazzo [Fri, 12 Mar 2021 15:30:35 +0000 (16:30 +0100)] 
Rename pool minconn, maxconn params to min_size, max_size

4 years agoChange pool stat requests_timeout to requests_errors, include queue full
Daniele Varrazzo [Fri, 12 Mar 2021 14:40:49 +0000 (15:40 +0100)] 
Change pool stat requests_timeout to requests_errors, include queue full

4 years agopool 'queue_length' stats renamed to 'requests_waiting'
Daniele Varrazzo [Fri, 12 Mar 2021 14:27:15 +0000 (15:27 +0100)] 
pool 'queue_length' stats renamed to 'requests_waiting'

4 years agoAdd pool.max_waiting
Daniele Varrazzo [Fri, 12 Mar 2021 14:26:18 +0000 (15:26 +0100)] 
Add pool.max_waiting

4 years agoPool docs corrections
Daniele Varrazzo [Fri, 12 Mar 2021 04:21:39 +0000 (05:21 +0100)] 
Pool docs corrections

4 years agoDivide docs file into chapter directories
Daniele Varrazzo [Fri, 12 Mar 2021 04:06:15 +0000 (05:06 +0100)] 
Divide docs file into chapter directories

4 years agoAdd connection pool docs
Daniele Varrazzo [Fri, 12 Mar 2021 03:55:03 +0000 (04:55 +0100)] 
Add connection pool docs

4 years agoGrow the pool size one connection at time
Daniele Varrazzo [Thu, 11 Mar 2021 20:11:48 +0000 (21:11 +0100)] 
Grow the pool size one connection at time

Not doing so saturates the worker processes which may not be able to do
other maintenance, including returning connections to the pool. It also
helps to limit the amount of growth in a sudden spike.
o

4 years agoAdd logging to pool.wait()
Daniele Varrazzo [Thu, 11 Mar 2021 20:11:05 +0000 (21:11 +0100)] 
Add logging to pool.wait()

4 years agoDon't use a worker in pool.putconn() if not needed
Daniele Varrazzo [Thu, 11 Mar 2021 14:31:55 +0000 (15:31 +0100)] 
Don't use a worker in pool.putconn() if not needed

If _reset is not set there is little work to do so the overhead of a
context switch is measurable and not necessary.

4 years agopool: fix off-by-one error in shrink log message
Daniele Varrazzo [Wed, 10 Mar 2021 23:40:33 +0000 (00:40 +0100)] 
pool: fix off-by-one error in shrink log message

4 years agoAdd async pool stats
Daniele Varrazzo [Wed, 10 Mar 2021 18:34:00 +0000 (19:34 +0100)] 
Add async pool stats

4 years agoAdd pool connection_class parameter
Daniele Varrazzo [Wed, 10 Mar 2021 02:39:46 +0000 (03:39 +0100)] 
Add pool connection_class parameter

4 years agoMake pool params keyword only in subclasses too
Daniele Varrazzo [Wed, 10 Mar 2021 02:32:02 +0000 (03:32 +0100)] 
Make pool params keyword only in subclasses too

4 years agoAdd pool stats
Daniele Varrazzo [Wed, 10 Mar 2021 02:22:07 +0000 (03:22 +0100)] 
Add pool stats

4 years agoAdd reset callback to connection pool
Daniele Varrazzo [Mon, 8 Mar 2021 03:43:33 +0000 (04:43 +0100)] 
Add reset callback to connection pool

4 years agoMake sure that the pool config function leaves connections in idle state
Daniele Varrazzo [Mon, 8 Mar 2021 03:18:25 +0000 (04:18 +0100)] 
Make sure that the pool config function leaves connections in idle state

4 years agoRename pool.wait_ready() to pool.wait()
Daniele Varrazzo [Mon, 8 Mar 2021 02:32:34 +0000 (03:32 +0100)] 
Rename pool.wait_ready() to pool.wait()

4 years agoDrop especially flaky async test dealing with del behaviour
Daniele Varrazzo [Mon, 8 Mar 2021 02:26:12 +0000 (03:26 +0100)] 
Drop especially flaky async test dealing with del behaviour

4 years agoAdd AsyncConnectionPool.check()
Daniele Varrazzo [Mon, 8 Mar 2021 02:25:57 +0000 (03:25 +0100)] 
Add AsyncConnectionPool.check()

4 years agoSimpler, more approximative, more reliable pool max lifetime test
Daniele Varrazzo [Mon, 8 Mar 2021 01:59:05 +0000 (02:59 +0100)] 
Simpler, more approximative, more reliable pool max lifetime test

4 years agoMore tolerance in schedule timing tests
Daniele Varrazzo [Mon, 8 Mar 2021 01:41:53 +0000 (02:41 +0100)] 
More tolerance in schedule timing tests

4 years agoAdd test for pool config
Daniele Varrazzo [Mon, 8 Mar 2021 01:28:14 +0000 (02:28 +0100)] 
Add test for pool config

4 years agoDon't throw an error on context exit if the connection is closed
Daniele Varrazzo [Mon, 8 Mar 2021 01:24:09 +0000 (02:24 +0100)] 
Don't throw an error on context exit if the connection is closed

4 years agoMake the async pool unavailable on Python 3.6
Daniele Varrazzo [Mon, 8 Mar 2021 01:05:09 +0000 (02:05 +0100)] 
Make the async pool unavailable on Python 3.6

Because of https://bugs.python.org/issue42600 or thereabout.

4 years agoDon't use threads to manage the scheduler and workers of async tasks
Daniele Varrazzo [Sun, 7 Mar 2021 23:52:58 +0000 (00:52 +0100)] 
Don't use threads to manage the scheduler and workers of async tasks

4 years agoAdd an async scheduler
Daniele Varrazzo [Sat, 27 Feb 2021 14:28:18 +0000 (15:28 +0100)] 
Add an async scheduler

4 years agoDrop get_running_loop call on async pool close
Daniele Varrazzo [Mon, 8 Mar 2021 00:07:53 +0000 (01:07 +0100)] 
Drop get_running_loop call on async pool close

4 years agoAdd compat module for python versions compatibility
Daniele Varrazzo [Sat, 6 Mar 2021 04:02:29 +0000 (05:02 +0100)] 
Add compat module for python versions compatibility

4 years agoAdd pool.check()
Daniele Varrazzo [Sat, 6 Mar 2021 02:38:45 +0000 (03:38 +0100)] 
Add pool.check()

4 years agoSet a timeout for the test run
Daniele Varrazzo [Sat, 6 Mar 2021 02:13:37 +0000 (03:13 +0100)] 
Set a timeout for the test run

4 years agoretry flaky max lifetime test
Daniele Varrazzo [Sat, 6 Mar 2021 01:21:30 +0000 (02:21 +0100)] 
retry flaky max lifetime test

4 years agoAdd Connection.broken
Daniele Varrazzo [Sun, 28 Feb 2021 15:19:49 +0000 (16:19 +0100)] 
Add Connection.broken

4 years agoAdd max_lifetime to pool connections
Daniele Varrazzo [Sun, 28 Feb 2021 03:35:56 +0000 (04:35 +0100)] 
Add max_lifetime to pool connections

4 years agoAdd jitter method to pool
Daniele Varrazzo [Sun, 28 Feb 2021 02:55:29 +0000 (03:55 +0100)] 
Add jitter method to pool

4 years agoUse more of the pool context manager in the tests
Daniele Varrazzo [Sat, 27 Feb 2021 22:47:26 +0000 (23:47 +0100)] 
Use more of the pool context manager in the tests

4 years agoRetry flaky tests a few times
Daniele Varrazzo [Sat, 27 Feb 2021 21:32:34 +0000 (22:32 +0100)] 
Retry flaky tests a few times

4 years agoAdd pool context managers
Daniele Varrazzo [Sat, 27 Feb 2021 20:44:07 +0000 (21:44 +0100)] 
Add pool context managers

4 years agoAdd pool.resize()
Daniele Varrazzo [Sat, 27 Feb 2021 20:07:38 +0000 (21:07 +0100)] 
Add pool.resize()

4 years agoMake cleanup tests more reliable
Daniele Varrazzo [Sat, 27 Feb 2021 11:47:36 +0000 (12:47 +0100)] 
Make cleanup tests more reliable

Unfortunately there is some race condition on delete and for now just
using a timer.

In order to limit the problem don't actually run the StopWorker task but
bail out from the workers as soon as seen.

4 years agoFix Python 3.6 compatibility issues in async pool
Daniele Varrazzo [Sat, 27 Feb 2021 11:13:17 +0000 (12:13 +0100)] 
Fix Python 3.6 compatibility issues in async pool

4 years agoAdd async connection pool
Daniele Varrazzo [Sat, 27 Feb 2021 01:09:44 +0000 (02:09 +0100)] 
Add async connection pool

4 years agoMove non-blocking pool functionalities to a generic base class
Daniele Varrazzo [Thu, 25 Feb 2021 21:38:33 +0000 (22:38 +0100)] 
Move non-blocking pool functionalities to a generic base class

Note that running maintenance threads is "non-blocking" and will be
thread-based in the async pool.

4 years agoAdd pool.wait_ready, drop connection_timeout on init
Daniele Varrazzo [Thu, 25 Feb 2021 20:57:20 +0000 (21:57 +0100)] 
Add pool.wait_ready, drop connection_timeout on init

This way init should never block. Always fill the connection in
background but a client can decide if they want to wait for it to be
ready.

4 years agoSplit the pool package objects in modules
Daniele Varrazzo [Thu, 25 Feb 2021 19:36:48 +0000 (20:36 +0100)] 
Split the pool package objects in modules

4 years agoMove task implementations to the pool object
Daniele Varrazzo [Thu, 25 Feb 2021 19:11:56 +0000 (20:11 +0100)] 
Move task implementations to the pool object

4 years agoMake psycopg3.pool a package
Daniele Varrazzo [Thu, 25 Feb 2021 17:31:16 +0000 (18:31 +0100)] 
Make psycopg3.pool a package

4 years agoUse the daemon param at constructor rather than the attribute
Daniele Varrazzo [Tue, 23 Feb 2021 22:39:44 +0000 (23:39 +0100)] 
Use the daemon param at constructor rather than the attribute

4 years agoUse the connections in the pool uniformly
Daniele Varrazzo [Mon, 22 Feb 2021 03:20:33 +0000 (04:20 +0100)] 
Use the connections in the pool uniformly

I feel this is a better use than using some more than other (e.g. in
term of bloat of the connections associated with the resources) and
gives a more predictable performance of the connection (there won't be
some cold, some hot).

Now there aren't really "unused connections" to single out in order to
shrink the pool. So keep a tally of the number of connections unused and
use a worker thread to close some if there are above minconn unused in a
period.

4 years agoChange state in critical section on pool.close()
Daniele Varrazzo [Mon, 22 Feb 2021 01:49:16 +0000 (02:49 +0100)] 
Change state in critical section on pool.close()

Nothing bad should happen, because what maintains the state respect the
_closed state. However because the logic is complex make it more
defensive.

4 years agoMake any pool task schedulable
Daniele Varrazzo [Mon, 22 Feb 2021 01:33:36 +0000 (02:33 +0100)] 
Make any pool task schedulable

Added interface to the pool to schedule a task in the future.

4 years agoDon't lose pool connections giving them to a clients already timed out
Daniele Varrazzo [Mon, 22 Feb 2021 01:05:14 +0000 (02:05 +0100)] 
Don't lose pool connections giving them to a clients already timed out

4 years agoAllow proxy tests to fail on Travis
Daniele Varrazzo [Sun, 21 Feb 2021 13:04:10 +0000 (14:04 +0100)] 
Allow proxy tests to fail on Travis

Don't know why it fails, it requires interactive investigation there.
Make sure tests can run connecting on TCP socket and avoid SSH in the
proxy anyway. Not that any of this worked...

4 years agoWait worker thread to stop on pool close
Daniele Varrazzo [Sun, 21 Feb 2021 12:41:02 +0000 (13:41 +0100)] 
Wait worker thread to stop on pool close

4 years agoDon't print warning gc'ing a pool in incomplete state
Daniele Varrazzo [Sun, 21 Feb 2021 12:07:58 +0000 (13:07 +0100)] 
Don't print warning gc'ing a pool in incomplete state

4 years agopool.close() called on del, idemptotent
Daniele Varrazzo [Sun, 21 Feb 2021 03:48:03 +0000 (04:48 +0100)] 
pool.close() called on del, idemptotent

4 years agoAdd separate setup_timeout param for pool
Daniele Varrazzo [Sun, 21 Feb 2021 03:47:20 +0000 (04:47 +0100)] 
Add separate setup_timeout param for pool

Allow 0 too, which would fill the pool in background.

4 years agoDrop _sec suffix from pool timeouts
Daniele Varrazzo [Sun, 21 Feb 2021 03:28:01 +0000 (04:28 +0100)] 
Drop _sec suffix from pool timeouts

Python is consistent enough to use seconds everywhere timeouts are
involved.

4 years agoAdd pool reconnection retry
Daniele Varrazzo [Sun, 21 Feb 2021 03:26:00 +0000 (04:26 +0100)] 
Add pool reconnection retry

4 years agoDon't try to rollback on exit if the connection is bad
Daniele Varrazzo [Sun, 21 Feb 2021 03:20:38 +0000 (04:20 +0100)] 
Don't try to rollback on exit if the connection is bad

4 years agoUse a weak reference to avoid loops between pool and maintenance tasks
Daniele Varrazzo [Sun, 21 Feb 2021 01:17:41 +0000 (02:17 +0100)] 
Use a weak reference to avoid loops between pool and maintenance tasks

4 years agoAdd scheduler class
Daniele Varrazzo [Sat, 20 Feb 2021 23:07:10 +0000 (00:07 +0100)] 
Add scheduler class

To be used in the connection pool to implement reconnection with
backoff, maybe periodic tasks.

4 years agoAdd test for pool kwargs
Daniele Varrazzo [Sat, 20 Feb 2021 03:36:13 +0000 (04:36 +0100)] 
Add test for pool kwargs

4 years agoDefault number of pool workers bumped to 3
Daniele Varrazzo [Sat, 20 Feb 2021 03:31:02 +0000 (04:31 +0100)] 
Default number of pool workers bumped to 3

4 years agoShrink the pool when connections have been idle long enough
Daniele Varrazzo [Sat, 20 Feb 2021 03:16:27 +0000 (04:16 +0100)] 
Shrink the pool when connections have been idle long enough

Pool behaviour on start changed: block on __init__ until minconn
connections have been obtained or raise PoolTimeout if timeout_sec have
passed. Not doing so makes complicated to understand, when a connection
is requested, if it's done during initialisation, and avoid an unneeded
grow request.

4 years agoDon't waste a worker thread adding a connection to the pool
Daniele Varrazzo [Sat, 20 Feb 2021 00:47:00 +0000 (01:47 +0100)] 
Don't waste a worker thread adding a connection to the pool

4 years agoForce pool num_workers to be > 0
Daniele Varrazzo [Sat, 20 Feb 2021 00:19:35 +0000 (01:19 +0100)] 
Force pool num_workers to be > 0

it might work with num_workers = 0, however it's not trivial, there is
some deadlock to avoid at creation time.

4 years agoAllow the pool to grow
Daniele Varrazzo [Fri, 19 Feb 2021 23:35:27 +0000 (00:35 +0100)] 
Allow the pool to grow

4 years agoMake the pool maintenance tasks base class abstract
Daniele Varrazzo [Sun, 14 Feb 2021 02:33:19 +0000 (03:33 +0100)] 
Make the pool maintenance tasks base class abstract

4 years agoDelete an eventual task-pool loop in StopWorker too
Daniele Varrazzo [Sun, 14 Feb 2021 02:27:30 +0000 (03:27 +0100)] 
Delete an eventual task-pool loop in StopWorker too

4 years agoAdd pool docstrings
Daniele Varrazzo [Sun, 14 Feb 2021 02:25:56 +0000 (03:25 +0100)] 
Add pool docstrings

4 years agoAdd connection pool close()
Daniele Varrazzo [Sun, 14 Feb 2021 01:22:20 +0000 (02:22 +0100)] 
Add connection pool close()

When the pool is closed, raise an exception in the thread of the clients
already waiting and refuse new requests. Let the current request finish
anyway.

4 years agoMake sure the pool can be deleted with no warning
Daniele Varrazzo [Sat, 13 Feb 2021 23:45:50 +0000 (00:45 +0100)] 
Make sure the pool can be deleted with no warning

Make sure to delete reference loops between the pool and the maintenance
tasks after they have run.

Do not raise a warning if a connection in a pool is deleted without
being closed as this is a normal condition (imagining a pool being
created as a global object).

4 years agoAdd pool context test
Daniele Varrazzo [Sat, 13 Feb 2021 22:51:02 +0000 (23:51 +0100)] 
Add pool context test

4 years agoAwake the pool client outside the critical section
Daniele Varrazzo [Sat, 13 Feb 2021 22:42:00 +0000 (23:42 +0100)] 
Awake the pool client outside the critical section

4 years agoReset the connection status returning it to the pool
Daniele Varrazzo [Sat, 13 Feb 2021 22:07:54 +0000 (23:07 +0100)] 
Reset the connection status returning it to the pool

4 years agoAdd basic pool functionality test for queuing and timeout
Daniele Varrazzo [Sat, 13 Feb 2021 17:02:08 +0000 (18:02 +0100)] 
Add basic pool functionality test for queuing and timeout

4 years agoDisplay the name in the connection pool repr
Daniele Varrazzo [Sat, 13 Feb 2021 16:17:28 +0000 (17:17 +0100)] 
Display the name in the connection pool repr

4 years agoAdd ConnectionPool class sketch
Daniele Varrazzo [Sat, 13 Feb 2021 03:06:54 +0000 (04:06 +0100)] 
Add ConnectionPool class sketch

It has a pool of connections alright, workers, clients waiting list.

Only minconn implemented, no maintenance thread etc. It's a start.

4 years ago@logilab is backing psycopg3!
Daniele Varrazzo [Mon, 8 Mar 2021 21:37:50 +0000 (22:37 +0100)] 
@logilab is backing psycopg3!

Merci beaucoup!

4 years agoWelcoming @dalibo as backer
Daniele Varrazzo [Mon, 8 Mar 2021 21:29:45 +0000 (22:29 +0100)] 
Welcoming @dalibo as backer

Vive la France!

4 years agoFix concurrency test
Daniele Varrazzo [Sat, 6 Mar 2021 02:08:54 +0000 (03:08 +0100)] 
Fix concurrency test

It was blocking because the committer was a too busy loop never yielding
to the other worker.

4 years agoUse gather instead of wait in tests to be coro friendly
Daniele Varrazzo [Sat, 6 Mar 2021 02:08:33 +0000 (03:08 +0100)] 
Use gather instead of wait in tests to be coro friendly

4 years agoBump mypy version
Daniele Varrazzo [Sat, 6 Mar 2021 01:09:39 +0000 (02:09 +0100)] 
Bump mypy version

4 years agoci: get token to rebuild docs from secrets
Daniele Varrazzo [Sat, 6 Mar 2021 00:26:55 +0000 (01:26 +0100)] 
ci: get token to rebuild docs from secrets

4 years agoMerge branch 'ci-gh-actions'
Daniele Varrazzo [Sat, 6 Mar 2021 00:13:59 +0000 (01:13 +0100)] 
Merge branch 'ci-gh-actions'

4 years agoYaml lint
Daniele Varrazzo [Tue, 2 Mar 2021 11:38:05 +0000 (12:38 +0100)] 
Yaml lint