Jörg Breitbart [Thu, 16 Oct 2025 00:01:32 +0000 (02:01 +0200)]
perf: load results by row rather than by column
In the past we were converting results to Python tuples proceeding
column by column. The rationale was that certain overhead such as
selecting the loader was to be paid only once per column, not once per
datum.
However this analysis was dismantled in #1163, see especially the comment at
https://github.com/psycopg/psycopg/pull/1163#issuecomment-3288921422
for some benchmark comparing various conversion strategies. In the end,
the simple row-by-row loading in a single function ends up being more
performing. Performance now surpasses the one of psycopg2.
See also #1155 for an initial analysis of performance regression.
A big thank you to Jörg Breitbart (@jerch) for this improvement!
Daniele Varrazzo [Sat, 11 Oct 2025 01:26:40 +0000 (03:26 +0200)]
fix: consider a connection closed in poll() only if it's not ready too
When closing the proxy and trying to communicate, poll returns state 25
on the connection, i.e. POLLIN | POLLERR | POLLHUP. The connection is
not closed though, so we might end up in weird state ahead (e.g. connection
stuck in ACTIVE).
Because it's suggested that the connection is readable, do read from it.
Likely a proper error will be raised downstream, but we will remain in
consistent state.
Daniele Varrazzo [Sun, 12 Oct 2025 02:38:24 +0000 (04:38 +0200)]
ci: use working test server on macOS
macos-14 can use PostgreSQL 18. I saw a failure trying to use 17; not
sure if it's still there but 18 works fine, so bump.
macos-13 better says on 17. Trying to "brew update" it will pull a
version that will try to compile docbook and takes forever to install. I
understand that avoid an upgrade we get a version that has a binary
package ("bottled", in this deranged naming convention).
refactor: check for closed connection in wait functions
Catch and display errors in a more homogeneous way: if a wait function
finds a connection closed it will raise an OperationalError chained to
the OSError obtained from stat'ing the socket. Previously control would
have gone back to the generator with a read-ready state and it would
have failed on whatever libpq function would have touched the socket.
Fix the problem reported in #608 affecting the epoll wait function, for
which we opted to use the poll function instead (which, more simply,
made the closed fd easier to spot).
note that on Windows it's not possible to use os.fstat() to ckeck a
socket state, therefore make it no-op.
ubifred [Fri, 10 Oct 2025 11:35:35 +0000 (13:35 +0200)]
ci: fix bad expansion of an array variable in build_libpq.sh
* ci: fix bad expansion of an array variable in build_libpq.sh
The `options` array variable have been introduced in
`adb8336a392a71a3cb2ddf18bd9b17238a01a06e` but is used with `$options`
which only expand the 1st value (for bash).
The full expansion have to be done with `${options[*]}`
Note that `-shared` and `-fPIC` options are still used as they are
enabled by default by `./Configure`, but the `zlib` option is not taken
into account (disabled by default)
fix: drop warning for objects passed as JSON lods/dump function
Just optimistically assume that they are ok and will not cause leaks.
The possibility of there being something else which might cause a leak
is lower than the possibility that we didn't think about something else
and raise a warning, refusing to cache the loader, and really cause a
leak.
We can't bump the min mypy on this maintenance branch because it doesn't
support Python 3.8, therefore ignore warnings in this file to
accommodate the min supported mypy version 1.14.
The server version reported is 250300. Tests fail with the error:
psycopg.errors.OperatorIntervention: this schema change is
disallowed because table "execmany" is locked and this operation cannot
automatically unlock the table
DETAIL: To unlock the table, execute `ALTER TABLE execmany SET (schema_locked = false);`
After the schema change completes, we recommend setting it back to
true with `ALTER TABLE execmany SET (schema_locked = true);`.
HINT: Locking the table improves changefeed performance; see
https://www.cockroachlabs.com/docs/dev/changefeed-best-practices.html#lock-the-schema-on-changefeed-watched-tables
Daniele Varrazzo [Wed, 14 May 2025 15:16:03 +0000 (17:16 +0200)]
fix: collect notifies only if no handler was registered
If someone is listening to notifications by using an handler, the
notifies backlog would fill without ever being emptied.
This change has the risk of breaking something if someone is relying on
notifies being received both via callback and via generator, but I don't
know how to satisfy it without creating a leak to users who don't use the
generator. Will ask around...
fix: keep a lock for the entire duration of executemany
Before this change we had the lock context inside the pipeline context,
because conn.pipepline() and Pipeline.__enter/exit__ take a lock. But
this created windows of opportunities for other threads to execute
concurrent operations on the connection, resulting in "another command
is already in progress" errors.
Daniele Varrazzo [Mon, 25 Aug 2025 16:35:30 +0000 (18:35 +0200)]
feat: add support for libpq PQconnectionUsedGSSAPI()
We will not advertise its presence as PGconn.used_gssapi until Psycopg
3.3. However we can use it internally to implement a warning to check
the implicit usage of gssencmode=prefer.
Not happy to drop them in a bugfix release, it seems possible to still
produce them with current cibuildwheel, but the documentation doesn't
make obvious how to restore them.
Daniele Varrazzo [Fri, 29 Aug 2025 15:11:54 +0000 (17:11 +0200)]
ci(alpine): install missing libraris required to test and package wheels
Since some recent alpine version the krb5-libs package is not installed by
default anymore. When building the libpq the package got installed as a
side effect of installing krb5-dev, but, in case libpq build was cached,
the libraries would have been missing when packaging the wheel or
importing psycopg for test.
Daniele Varrazzo [Wed, 21 May 2025 16:54:43 +0000 (18:54 +0200)]
chore: bump Cython version
Skip version 3.1.0 because of https://github.com/cython/cython/issues/6850
I understand that >= 3.1.0 is required for the free threading support (#1096).
Create all the connections beforehand otherwise some may reuse the same
address, resulting in a duplicated warning message, which would be
omitted and fail the test.
fix: don't call str(self) or imported objects in __del__
We easily end up trying to access resources already unavailable in the
objects representation. In Python 3.14 this seems to happen aggressively
(or maybe it's just more visible because testing a dev version).
Daniele Varrazzo [Tue, 20 May 2025 15:06:06 +0000 (17:06 +0200)]
fix: change signature of SQL.join() to allow a sequence of Any
We declared accepting a sequence of Composable, but, because we passed
the joined values to `Composite`, actually non-composable were
interpreted as `Literal`.
Make this behaviour explicit by testing it and fixing the signature.
fix(json): don't leak when lambdas are used as dumps/loads function
Cache functions on the code, not on the function identity itself:
different lambdas or local functions are different objects but he
underlying code object is the same.
Avoid caching if the function is a closure because it then becomes a can
of worms (if the argument is not hashable etc). Throw a warning in that
case.
Fix #1108
fix(json): don't leak when lambdas are used as loads function
projects / thirdparty / psycopg.git / log
