Daniele Varrazzo [Sat, 18 Oct 2025 15:48:32 +0000 (17:48 +0200)]
fix(pool): more robust placement of reconnection attempts
The problem reported in #1176 is the close recursion already fixed.
However, because we try to reconnect before closing, we end up scheduling
a reconnection attempt for every recursion loop.
The recursion shouldn't happen anymore, but it seems more robust to
reconnect after the connection has been successfully closed, otherwise
we might end up requesting more connections than we ought (hopefully not
as dramatically as here anyway).
Daniele Varrazzo [Sun, 19 Oct 2025 00:20:02 +0000 (02:20 +0200)]
Merge branch 'maint-pool-3.2-backport' into maint-3.2
This branch contained the changes in the maint-pool-3.2 that hadn't been
applied to the maint-3.2 branch.
The psycopg-pool code and tests in the maint-3.2 branch is now at the
same level of the maint-pool-3.2 branch but is much more modern, so
further psycopg-pool 3.2 will continue on this branch.
Daniele Varrazzo [Fri, 21 Feb 2025 16:27:29 +0000 (17:27 +0100)]
fix(pool): check that there is some connection in the pool before shrinking
There is a race condition between shrinking the pool and closing an
expired connection which might result in attempting to pop from the
empty _pool deque. Add a check to prevent that.
The bug is not serious as it doesn't compromise the pool working but it causes
noise in the logging.
Jörg Breitbart [Thu, 16 Oct 2025 00:01:32 +0000 (02:01 +0200)]
perf: load results by row rather than by column
In the past we were converting results to Python tuples proceeding
column by column. The rationale was that certain overhead such as
selecting the loader was to be paid only once per column, not once per
datum.
However this analysis was dismantled in #1163, see especially the comment at
https://github.com/psycopg/psycopg/pull/1163#issuecomment-3288921422
for some benchmark comparing various conversion strategies. In the end,
the simple row-by-row loading in a single function ends up being more
performing. Performance now surpasses the one of psycopg2.
See also #1155 for an initial analysis of performance regression.
A big thank you to Jörg Breitbart (@jerch) for this improvement!
Daniele Varrazzo [Sat, 11 Oct 2025 01:26:40 +0000 (03:26 +0200)]
fix: consider a connection closed in poll() only if it's not ready too
When closing the proxy and trying to communicate, poll returns state 25
on the connection, i.e. POLLIN | POLLERR | POLLHUP. The connection is
not closed though, so we might end up in weird state ahead (e.g. connection
stuck in ACTIVE).
Because it's suggested that the connection is readable, do read from it.
Likely a proper error will be raised downstream, but we will remain in
consistent state.
Daniele Varrazzo [Sun, 12 Oct 2025 02:38:24 +0000 (04:38 +0200)]
ci: use working test server on macOS
macos-14 can use PostgreSQL 18. I saw a failure trying to use 17; not
sure if it's still there but 18 works fine, so bump.
macos-13 better says on 17. Trying to "brew update" it will pull a
version that will try to compile docbook and takes forever to install. I
understand that avoid an upgrade we get a version that has a binary
package ("bottled", in this deranged naming convention).
refactor: check for closed connection in wait functions
Catch and display errors in a more homogeneous way: if a wait function
finds a connection closed it will raise an OperationalError chained to
the OSError obtained from stat'ing the socket. Previously control would
have gone back to the generator with a read-ready state and it would
have failed on whatever libpq function would have touched the socket.
Fix the problem reported in #608 affecting the epoll wait function, for
which we opted to use the poll function instead (which, more simply,
made the closed fd easier to spot).
note that on Windows it's not possible to use os.fstat() to ckeck a
socket state, therefore make it no-op.
ubifred [Fri, 10 Oct 2025 11:35:35 +0000 (13:35 +0200)]
ci: fix bad expansion of an array variable in build_libpq.sh
* ci: fix bad expansion of an array variable in build_libpq.sh
The `options` array variable have been introduced in
`adb8336a392a71a3cb2ddf18bd9b17238a01a06e` but is used with `$options`
which only expand the 1st value (for bash).
The full expansion have to be done with `${options[*]}`
Note that `-shared` and `-fPIC` options are still used as they are
enabled by default by `./Configure`, but the `zlib` option is not taken
into account (disabled by default)
fix: drop warning for objects passed as JSON lods/dump function
Just optimistically assume that they are ok and will not cause leaks.
The possibility of there being something else which might cause a leak
is lower than the possibility that we didn't think about something else
and raise a warning, refusing to cache the loader, and really cause a
leak.
We can't bump the min mypy on this maintenance branch because it doesn't
support Python 3.8, therefore ignore warnings in this file to
accommodate the min supported mypy version 1.14.
The server version reported is 250300. Tests fail with the error:
psycopg.errors.OperatorIntervention: this schema change is
disallowed because table "execmany" is locked and this operation cannot
automatically unlock the table
DETAIL: To unlock the table, execute `ALTER TABLE execmany SET (schema_locked = false);`
After the schema change completes, we recommend setting it back to
true with `ALTER TABLE execmany SET (schema_locked = true);`.
HINT: Locking the table improves changefeed performance; see
https://www.cockroachlabs.com/docs/dev/changefeed-best-practices.html#lock-the-schema-on-changefeed-watched-tables
Daniele Varrazzo [Wed, 14 May 2025 15:16:03 +0000 (17:16 +0200)]
fix: collect notifies only if no handler was registered
If someone is listening to notifications by using an handler, the
notifies backlog would fill without ever being emptied.
This change has the risk of breaking something if someone is relying on
notifies being received both via callback and via generator, but I don't
know how to satisfy it without creating a leak to users who don't use the
generator. Will ask around...
fix: keep a lock for the entire duration of executemany
Before this change we had the lock context inside the pipeline context,
because conn.pipepline() and Pipeline.__enter/exit__ take a lock. But
this created windows of opportunities for other threads to execute
concurrent operations on the connection, resulting in "another command
is already in progress" errors.
Daniele Varrazzo [Mon, 25 Aug 2025 16:35:30 +0000 (18:35 +0200)]
feat: add support for libpq PQconnectionUsedGSSAPI()
We will not advertise its presence as PGconn.used_gssapi until Psycopg
3.3. However we can use it internally to implement a warning to check
the implicit usage of gssencmode=prefer.
Not happy to drop them in a bugfix release, it seems possible to still
produce them with current cibuildwheel, but the documentation doesn't
make obvious how to restore them.
Daniele Varrazzo [Fri, 29 Aug 2025 15:11:54 +0000 (17:11 +0200)]
ci(alpine): install missing libraris required to test and package wheels
Since some recent alpine version the krb5-libs package is not installed by
default anymore. When building the libpq the package got installed as a
side effect of installing krb5-dev, but, in case libpq build was cached,
the libraries would have been missing when packaging the wheel or
importing psycopg for test.
Daniele Varrazzo [Wed, 21 May 2025 16:54:43 +0000 (18:54 +0200)]
chore: bump Cython version
Skip version 3.1.0 because of https://github.com/cython/cython/issues/6850
I understand that >= 3.1.0 is required for the free threading support (#1096).
projects / thirdparty / psycopg.git / log
