]> git.ipfire.org Git - thirdparty/psycopg.git/log
thirdparty/psycopg.git
4 years agoDropped unneeded assignment
Daniele Varrazzo [Wed, 30 Dec 2020 21:22:27 +0000 (22:22 +0100)] 
Dropped unneeded assignment

4 years agoDropped format parameter from register
Daniele Varrazzo [Wed, 30 Dec 2020 21:02:02 +0000 (22:02 +0100)] 
Dropped format parameter from register

Now the loaders/dumper declare their format themselves.

4 years agoAdded guard for lazy import to avoid the import statement
Daniele Varrazzo [Wed, 30 Dec 2020 19:30:59 +0000 (20:30 +0100)] 
Added guard for lazy import to avoid the import statement

I'm mot so sure it's always so fast to re-import.

4 years agoAdded Loader.format and Dumper.format
Daniele Varrazzo [Wed, 30 Dec 2020 19:27:56 +0000 (20:27 +0100)] 
Added Loader.format and Dumper.format

4 years agoDon't convert fformat and binary_tuples to Format in C pq wrapper
Daniele Varrazzo [Wed, 30 Dec 2020 17:53:19 +0000 (18:53 +0100)] 
Don't convert fformat and binary_tuples to Format in C pq wrapper

4 years agoFixed binary copy with empty content
Daniele Varrazzo [Wed, 30 Dec 2020 17:23:06 +0000 (18:23 +0100)] 
Fixed binary copy with empty content

4 years agoDropped async/sync code duplication using high level generators
Daniele Varrazzo [Wed, 30 Dec 2020 15:31:13 +0000 (16:31 +0100)] 
Dropped async/sync code duplication using high level generators

4 years agoUse slots on PostgresQuery object
Daniele Varrazzo [Wed, 30 Dec 2020 03:03:06 +0000 (04:03 +0100)] 
Use slots on PostgresQuery object

4 years agoFast-path in query preparation to skip expensive isinstance
Daniele Varrazzo [Wed, 30 Dec 2020 02:53:23 +0000 (03:53 +0100)] 
Fast-path in query preparation to skip expensive isinstance

4 years agoAdded slots on cursor
Daniele Varrazzo [Wed, 30 Dec 2020 02:30:10 +0000 (03:30 +0100)] 
Added slots on cursor

4 years agoAdded backer @MaterializeInc. Thank you :)
Daniele Varrazzo [Wed, 6 Jan 2021 23:55:56 +0000 (00:55 +0100)] 
Added backer @MaterializeInc. Thank you :)

4 years agoMerge branch 'c-optimise'
Daniele Varrazzo [Tue, 29 Dec 2020 17:47:45 +0000 (18:47 +0100)] 
Merge branch 'c-optimise'

4 years agoMake testable, and test, the different waiting implementation
Daniele Varrazzo [Tue, 29 Dec 2020 17:09:02 +0000 (18:09 +0100)] 
Make testable, and test, the different waiting implementation

Also fixed segfault calling flush on a closed connection.

4 years agoAdded epoll-based wait() implementation
Daniele Varrazzo [Tue, 29 Dec 2020 03:03:34 +0000 (04:03 +0100)] 
Added epoll-based wait() implementation

Only used if the EpollSelector would (i.e. epoll is available and it's
the best thing on the menu). Saves noticeable time over the
EpollSelector as there's less Python stuff going on and can use ONESHOT
+ MOD_CTL.

4 years agoMore tweaks to the Transformer C code
Daniele Varrazzo [Tue, 29 Dec 2020 04:16:44 +0000 (05:16 +0100)] 
More tweaks to the Transformer C code

4 years agoTweaking generators.execute C code
Daniele Varrazzo [Mon, 28 Dec 2020 04:28:01 +0000 (05:28 +0100)] 
Tweaking generators.execute C code

4 years agoUse the C api to create memoryview objects
Daniele Varrazzo [Sun, 27 Dec 2020 23:20:47 +0000 (00:20 +0100)] 
Use the C api to create memoryview objects

4 years agoUse cython type declarations for bytes too
Daniele Varrazzo [Sun, 27 Dec 2020 22:55:18 +0000 (23:55 +0100)] 
Use cython type declarations for bytes too

4 years agoUse cython type declaration rather than python annotation for int
Daniele Varrazzo [Sun, 27 Dec 2020 22:25:48 +0000 (23:25 +0100)] 
Use cython type declaration rather than python annotation for int

It gives directly a c variable.

4 years agoDon't use enums for libpq arguments
Daniele Varrazzo [Sun, 27 Dec 2020 21:26:20 +0000 (22:26 +0100)] 
Don't use enums for libpq arguments

We might want to have speed there. Enums can be returned from high-level
objects such as the Connection.info object to be implemented.

4 years agoAdded guard to avoid failure on __del__ of unborn connection
Daniele Varrazzo [Mon, 28 Dec 2020 03:38:28 +0000 (04:38 +0100)] 
Added guard to avoid failure on __del__ of unborn connection

4 years agoSet public module for PQerror
Daniele Varrazzo [Mon, 28 Dec 2020 03:25:40 +0000 (04:25 +0100)] 
Set public module for PQerror

4 years agoModule psycopg3_c.pq_cython renamed to psycopg3_c.pq
Daniele Varrazzo [Sun, 27 Dec 2020 16:56:18 +0000 (17:56 +0100)] 
Module psycopg3_c.pq_cython renamed to psycopg3_c.pq

psycopg3_c files reorganised in directories. Still unable to split and
localise pxd files, but now it's already cleaner.

Travis keeps failing in funny ways, so skipping fork tests (they pass ok
locally).

4 years agoFixed comment after adapters refactoring
Daniele Varrazzo [Sun, 27 Dec 2020 02:18:37 +0000 (03:18 +0100)] 
Fixed comment after adapters refactoring

4 years agoAvoid catching exceptions for dict lookup in C
Daniele Varrazzo [Sun, 27 Dec 2020 02:09:05 +0000 (03:09 +0100)] 
Avoid catching exceptions for dict lookup in C

4 years agoFurther use of the C api to reduce some cython checking
Daniele Varrazzo [Sun, 27 Dec 2020 01:45:58 +0000 (02:45 +0100)] 
Further use of the C api to reduce some cython checking

4 years agoMore efficient rows loading
Daniele Varrazzo [Sun, 27 Dec 2020 00:46:52 +0000 (01:46 +0100)] 
More efficient rows loading

Iterate by columns first, so that choices regarding the loader can be
done only once instead of once per row.

4 years agoAdded Transformer.load_rows()
Daniele Varrazzo [Sat, 26 Dec 2020 23:28:29 +0000 (00:28 +0100)] 
Added Transformer.load_rows()

There is some good optimisation to be had moving in C the two loops.

4 years agoSome C optimisation of the Transformer object
Daniele Varrazzo [Sat, 26 Dec 2020 22:12:02 +0000 (23:12 +0100)] 
Some C optimisation of the Transformer object

Don't use a list of 2-tuples to set types, use two lists of scalars.

4 years agoOptimise access to PGresult data
Daniele Varrazzo [Sat, 26 Dec 2020 20:55:32 +0000 (21:55 +0100)] 
Optimise access to PGresult data

Access the underlying data structure instead of using the libpq
functions.

We shouldn't, but the speed increase is noticeable.

4 years agoInternal functions moved out of cython class
Daniele Varrazzo [Sat, 26 Dec 2020 18:40:08 +0000 (19:40 +0100)] 
Internal functions moved out of cython class

Avoid a vpointer lookup for them

4 years agoRelease the gil around the exec/send libpq functions
Daniele Varrazzo [Sat, 26 Dec 2020 18:09:17 +0000 (19:09 +0100)] 
Release the gil around the exec/send libpq functions

4 years agoAdded guard for negative numbers in load_row
Daniele Varrazzo [Sat, 26 Dec 2020 18:07:38 +0000 (19:07 +0100)] 
Added guard for negative numbers in load_row

Also added a reminder to review the C implementation for performance.

4 years agoAdded repr of several objects
Daniele Varrazzo [Sun, 27 Dec 2020 16:12:51 +0000 (17:12 +0100)] 
Added repr of several objects

Show the underlying connection basic identification and status.

4 years agoRaise a resource warning if a connection is deleted while open
Daniele Varrazzo [Sun, 27 Dec 2020 04:18:27 +0000 (05:18 +0100)] 
Raise a resource warning if a connection is deleted while open

4 years agoAdaptation context reworked
Daniele Varrazzo [Sat, 26 Dec 2020 00:08:32 +0000 (01:08 +0100)] 
Adaptation context reworked

Don't use a sequence of dictionaries, but a single copy-on-write
structure, which is cheaper to explore.

Added a protocol representing all the objects that can be used as
adaptation context in order to avoid cascades of isinstance.

The difference in behaviour is that changing global adapters doesn't
affect already created connections which had been customised, and likely
customised cursors are not affected by global and connection changes.
The utility of the previous behaviour doesn't seem anything on which
people would build an empire on, the new behaviour is good as well.

4 years agoDropped unused Transformer attribute
Daniele Varrazzo [Fri, 25 Dec 2020 18:40:37 +0000 (19:40 +0100)] 
Dropped unused Transformer attribute

4 years agoDropped tox from dev requirements
Daniele Varrazzo [Fri, 25 Dec 2020 18:17:22 +0000 (19:17 +0100)] 
Dropped tox from dev requirements

4 years agoPrepared statements management moved to a purpose class
Daniele Varrazzo [Fri, 25 Dec 2020 17:43:56 +0000 (18:43 +0100)] 
Prepared statements management moved to a purpose class

4 years agoThe function to maintain prepared statements is no more a generator
Daniele Varrazzo [Fri, 25 Dec 2020 14:20:23 +0000 (15:20 +0100)] 
The function to maintain prepared statements is no more a generator

Just generate the necessary statements, yield will happen only if
necessary.

4 years agoDropped "unknown loader", use the text loader for everything
Daniele Varrazzo [Fri, 25 Dec 2020 14:05:59 +0000 (15:05 +0100)] 
Dropped "unknown loader", use the text loader for everything

Register the C implementation on all the types where it is useful.

4 years agoImplement fetch* not in term of iter
Daniele Varrazzo [Fri, 25 Dec 2020 12:14:34 +0000 (13:14 +0100)] 
Implement fetch* not in term of iter

Shave some time on lists reallocation, but especially avoid going
through __aiter__ on async, because profiling showed some 20% of the
time spent there.

This makes me rethink the entire giving async interface for things that are
already on the client side. Feels like that, in order to implement server-side
cursor, we are shedding speed on the common use of async connections.

4 years agoExclude .pyx files from sdist package
Daniele Varrazzo [Fri, 25 Dec 2020 03:13:23 +0000 (04:13 +0100)] 
Exclude .pyx files from sdist package

This way install only depends on the .c files compiled by us.

4 years agoMerge branch 'ctypes-reorganisation'
Daniele Varrazzo [Fri, 25 Dec 2020 03:01:13 +0000 (04:01 +0100)] 
Merge branch 'ctypes-reorganisation'

4 years agoFixed typos in cython
Daniele Varrazzo [Mon, 21 Dec 2020 01:39:09 +0000 (02:39 +0100)] 
Fixed typos in cython

4 years agoDropped 'impl' alias for libpq in cython
Daniele Varrazzo [Mon, 21 Dec 2020 00:59:57 +0000 (01:59 +0100)] 
Dropped 'impl' alias for libpq in cython

4 years agopsycopg3_c packages moved closer to where they are used
Daniele Varrazzo [Mon, 21 Dec 2020 00:49:06 +0000 (01:49 +0100)] 
psycopg3_c packages moved closer to where they are used

Things were compiling well because everything goes into the same namespace.

4 years agoOther psycopg3_c.pq_cython objects split in modules
Daniele Varrazzo [Mon, 21 Dec 2020 00:27:10 +0000 (01:27 +0100)] 
Other psycopg3_c.pq_cython objects split in modules

4 years agoFurther PGconn code moving and stricter declarations
Daniele Varrazzo [Mon, 21 Dec 2020 00:21:19 +0000 (01:21 +0100)] 
Further PGconn code moving and stricter declarations

4 years agoPGconn code moved to pq directory
Daniele Varrazzo [Sun, 20 Dec 2020 23:58:11 +0000 (00:58 +0100)] 
PGconn code moved to pq directory

4 years agoDropped __init__ file from types, which is not a package
Daniele Varrazzo [Sun, 20 Dec 2020 23:49:34 +0000 (00:49 +0100)] 
Dropped __init__ file from types, which is not a package

4 years agoUse a strict PGconn type on Escaping.__init__ definition definition
Daniele Varrazzo [Sun, 20 Dec 2020 23:48:43 +0000 (00:48 +0100)] 
Use a strict PGconn type on Escaping.__init__ definition definition

4 years agoEscaping code moved to psycopg3_c/pq dir
Daniele Varrazzo [Sun, 20 Dec 2020 23:36:19 +0000 (00:36 +0100)] 
Escaping code moved to psycopg3_c/pq dir

4 years agolibpq cython declarations moved to psycopg3_c/pq package
Daniele Varrazzo [Sun, 20 Dec 2020 18:51:02 +0000 (19:51 +0100)] 
libpq cython declarations moved to psycopg3_c/pq package

4 years agoMerge branch 'prepared-statements'
Daniele Varrazzo [Fri, 25 Dec 2020 02:46:16 +0000 (03:46 +0100)] 
Merge branch 'prepared-statements'

4 years agoAdded note to remember to review preparation in executemany
Daniele Varrazzo [Fri, 25 Dec 2020 02:44:56 +0000 (03:44 +0100)] 
Added note to remember to review preparation in executemany

4 years agoAdded documentation for prepared statements
Daniele Varrazzo [Fri, 25 Dec 2020 02:10:47 +0000 (03:10 +0100)] 
Added documentation for prepared statements

4 years agoPostgresQuery const initialisation moved to the class
Daniele Varrazzo [Thu, 24 Dec 2020 04:00:27 +0000 (05:00 +0100)] 
PostgresQuery const initialisation moved to the class

4 years agoUse text oid for text/unknown types on PG 9.6
Daniele Varrazzo [Thu, 24 Dec 2020 03:42:41 +0000 (04:42 +0100)] 
Use text oid for text/unknown types on PG 9.6

This creates a difference between PG 96 and >= 10 as in the latter less
casts are required. However unknown oid cannot be used to prepare queries
in PG 9.6.

4 years agoAdded first implementation of prepared statements support
Daniele Varrazzo [Wed, 23 Dec 2020 02:35:00 +0000 (03:35 +0100)] 
Added first implementation of prepared statements support

4 years agoImplement connection/cursor procedures as generators
Daniele Varrazzo [Tue, 22 Dec 2020 17:09:34 +0000 (18:09 +0100)] 
Implement connection/cursor procedures as generators

Generators are common to both sync and async code so there is much less
code duplication, which will be useful in the light of doing more
complex things such as prepared transactions.

As a side effect of the refactoring, operations on closed connections
are raised as OperationalError rather than InterfaceError because
connection closure may depend on external factors.

Dropped fd from PQGen stream: the having the fd in the middle of the
`Ready` values made more difficult to chain more than one operation in
the same generator. The fd is not passed as extra parameter.

4 years agoRegister automatically the array loader on all the builtins
Daniele Varrazzo [Wed, 23 Dec 2020 21:02:19 +0000 (22:02 +0100)] 
Register automatically the array loader on all the builtins

Also added reg* types to the known builtins. This way regtype[] is
loaded as a list of strings.

4 years agoBetter use of Cython public properties
Daniele Varrazzo [Sun, 20 Dec 2020 18:23:20 +0000 (19:23 +0100)] 
Better use of Cython public properties

4 years agoEnable cython faster call convention
Daniele Varrazzo [Mon, 21 Dec 2020 01:39:23 +0000 (02:39 +0100)] 
Enable cython faster call convention

4 years agoRelease the gil around functions known to be slow
Daniele Varrazzo [Mon, 21 Dec 2020 02:58:24 +0000 (03:58 +0100)] 
Release the gil around functions known to be slow

Profiling some time ago showed that these functions take relevant time.
Other paths will be reviewed.

https://www.varrazzo.com/blog/2020/05/19/a-trip-into-optimisation/

4 years agoAdded website to sponsors file
Daniele Varrazzo [Fri, 18 Dec 2020 20:08:05 +0000 (21:08 +0100)] 
Added website to sponsors file

4 years agoAdded @postgrespro sponsor!
Daniele Varrazzo [Fri, 18 Dec 2020 13:55:46 +0000 (14:55 +0100)] 
Added @postgrespro sponsor!

Thank you Oleg!

4 years agoAdded _oid C attribute to CDumper
Daniele Varrazzo [Thu, 17 Dec 2020 16:29:48 +0000 (17:29 +0100)] 
Added _oid C attribute to CDumper

4 years agoReduce re-allocations in objects quoting
Daniele Varrazzo [Thu, 17 Dec 2020 12:55:56 +0000 (13:55 +0100)] 
Reduce re-allocations in objects quoting

4 years agoFixed broken previous commit
Daniele Varrazzo [Thu, 17 Dec 2020 11:50:48 +0000 (12:50 +0100)] 
Fixed broken previous commit

4 years agoescape_string, escape_identifier use buffers too
Daniele Varrazzo [Thu, 17 Dec 2020 04:55:56 +0000 (05:55 +0100)] 
escape_string, escape_identifier use buffers too

4 years agoMake wider use of Composite.as_bytes
Daniele Varrazzo [Thu, 17 Dec 2020 04:13:25 +0000 (05:13 +0100)] 
Make wider use of Composite.as_bytes

4 years agoAdded buffer object to wrap libpq memory
Daniele Varrazzo [Thu, 17 Dec 2020 04:02:35 +0000 (05:02 +0100)] 
Added buffer object to wrap libpq memory

Several code paths extended to deal with objects implementing the buffer
protocol instead of just bytes: this allows to perform less memory copy.

Added sql.Composable.as_bytes method to use in preference of as_string
(you may do without decoding altogether).

4 years agoAdded bytearray and memoryview dumpers
Daniele Varrazzo [Thu, 17 Dec 2020 01:29:09 +0000 (02:29 +0100)] 
Added bytearray and memoryview dumpers

Also manage these objects when they rich the libp param passing.

4 years agoDropped use of a couple of constants in json
Daniele Varrazzo [Wed, 16 Dec 2020 21:06:17 +0000 (22:06 +0100)] 
Dropped use of a couple of constants in json

4 years agoAdded docs about string/binary adaptation
Daniele Varrazzo [Wed, 16 Dec 2020 21:05:02 +0000 (22:05 +0100)] 
Added docs about string/binary adaptation

4 years agoSponsorship by @commandprompt - Thank you very much @jdatcmd
Daniele Varrazzo [Thu, 10 Dec 2020 12:41:59 +0000 (13:41 +0100)] 
Sponsorship by @commandprompt - Thank you very much @jdatcmd

4 years agoAdded @mailupinc sponsor. Thank you!
Daniele Varrazzo [Thu, 10 Dec 2020 12:19:22 +0000 (13:19 +0100)] 
Added @mailupinc sponsor. Thank you!

4 years agoSponsorship by @phenopolis from @pontikos
Daniele Varrazzo [Thu, 10 Dec 2020 12:08:19 +0000 (13:08 +0100)] 
Sponsorship by @phenopolis from @pontikos

Thank you very much, Niko!

4 years agoUpdated sponsorship by @gporcari @genropy
Daniele Varrazzo [Thu, 10 Dec 2020 12:01:17 +0000 (13:01 +0100)] 
Updated sponsorship by @gporcari @genropy

Grazie tantissime, Giovanni!

4 years agoBackers file converted to yaml
Daniele Varrazzo [Thu, 10 Dec 2020 11:04:46 +0000 (12:04 +0100)] 
Backers file converted to yaml

It will be displayed at https://www.psycopg.org/sponsors/

4 years agoAdded several range tests and fixed a pasto error
Daniele Varrazzo [Thu, 10 Dec 2020 02:51:36 +0000 (03:51 +0100)] 
Added several range tests and fixed a pasto error

4 years agoMerge branch 'range-adaptation'
Daniele Varrazzo [Thu, 10 Dec 2020 01:43:55 +0000 (02:43 +0100)] 
Merge branch 'range-adaptation'

4 years agoAdded range type fetching and registration
Daniele Varrazzo [Sun, 6 Dec 2020 02:50:50 +0000 (02:50 +0000)] 
Added range type fetching and registration

Fixed quoting of [ ] chars for range types: unlike for composite they
need quoting.

4 years agoFetch/register composite made methods of a CompositeInfo class.
Daniele Varrazzo [Sun, 6 Dec 2020 01:35:16 +0000 (01:35 +0000)] 
Fetch/register composite made methods of a CompositeInfo class.

4 years agoAdded builtin ranges adaptation
Daniele Varrazzo [Sat, 5 Dec 2020 18:41:26 +0000 (18:41 +0000)] 
Added builtin ranges adaptation

4 years agoAdded `conection.execute()` method
Daniele Varrazzo [Tue, 8 Dec 2020 08:55:41 +0000 (09:55 +0100)] 
Added `conection.execute()` method

4 years agoComposite record parsing made independent from the "()" terminators
Daniele Varrazzo [Sat, 5 Dec 2020 15:51:39 +0000 (15:51 +0000)] 
Composite record parsing made independent from the "()" terminators

This should make this code useful to parse range representations too,
which may use a mix of () and [] terminators.

4 years ago@Intevation is sponsoring psycopg3, thank you :)
Daniele Varrazzo [Sat, 5 Dec 2020 14:38:39 +0000 (14:38 +0000)] 
@Intevation is sponsoring psycopg3, thank you :)

4 years agoFixed escaping of a few chars in composite dumping
Daniele Varrazzo [Sat, 5 Dec 2020 04:05:34 +0000 (04:05 +0000)] 
Fixed escaping of a few chars in composite dumping

4 years agoFixed composite info fetch with homonymous types
Daniele Varrazzo [Sat, 5 Dec 2020 02:48:40 +0000 (02:48 +0000)] 
Fixed composite info fetch with homonymous types

Use a fully qualified name, optionally expressed as sql.Identifier, to
find the type in the right schema.

4 years agoInstallation and readme improvements
Daniele Varrazzo [Sat, 5 Dec 2020 01:58:19 +0000 (01:58 +0000)] 
Installation and readme improvements

4 years agoDropped erroneous statement that psycopg3 quotes values
Daniele Varrazzo [Sat, 5 Dec 2020 00:52:05 +0000 (00:52 +0000)] 
Dropped erroneous statement that psycopg3 quotes values

4 years agoAdded backer @bernhardreiter, thank you!
Daniele Varrazzo [Sat, 5 Dec 2020 00:28:18 +0000 (00:28 +0000)] 
Added backer @bernhardreiter, thank you!

4 years agoSmall tweaks to add_backer script
Daniele Varrazzo [Fri, 4 Dec 2020 04:48:02 +0000 (04:48 +0000)] 
Small tweaks to add_backer script

4 years agoBackers file updated
Daniele Varrazzo [Fri, 4 Dec 2020 04:33:28 +0000 (04:33 +0000)] 
Backers file updated

Added scripts to help maintaining it.

4 years agoFixed COPY statement suggesting it accepts parameters
Daniele Varrazzo [Wed, 2 Dec 2020 19:03:15 +0000 (19:03 +0000)] 
Fixed COPY statement suggesting it accepts parameters

It was, it just didn't work... :P

4 years agoFixed testing for diag properties on Pg < 9.6
Daniele Varrazzo [Wed, 2 Dec 2020 17:26:26 +0000 (17:26 +0000)] 
Fixed testing for diag properties on Pg < 9.6

4 years agoSeparate generators where the fd can change from the ones where not
Daniele Varrazzo [Wed, 2 Dec 2020 17:23:14 +0000 (17:23 +0000)] 
Separate generators where the fd can change from the ones where not

This should make the query operation marginally faster.

4 years agoMake sure Notify is hashable
Daniele Varrazzo [Wed, 2 Dec 2020 04:01:48 +0000 (04:01 +0000)] 
Make sure Notify is hashable