]>
git.ipfire.org Git - thirdparty/psycopg.git/log
Daniele Varrazzo [Fri, 22 Jan 2021 03:05:12 +0000 (04:05 +0100)]
Added Cursor.stream() to support fetch in single-row mode
Daniele Varrazzo [Thu, 21 Jan 2021 03:04:09 +0000 (04:04 +0100)]
Added a few docstrings on types registry
Daniele Varrazzo [Thu, 21 Jan 2021 03:02:03 +0000 (04:02 +0100)]
Run black on the builtin oids
Daniele Varrazzo [Thu, 21 Jan 2021 02:57:08 +0000 (03:57 +0100)]
Array dumper ligtened up
Don't fetch a string dumper on init, dropped sub_oid attribute.
Daniele Varrazzo [Wed, 20 Jan 2021 17:25:05 +0000 (18:25 +0100)]
Added two-steps dumpers implementation in C too
Daniele Varrazzo [Wed, 20 Jan 2021 12:23:06 +0000 (13:23 +0100)]
Don't require a Transformer roundtrip to choose the specific int dumper
They are static object, they require no customisation with the
connection. I don't expect anyone wanting to subclass them; if they
do they will deal with the whole 5 objects.
Daniele Varrazzo [Wed, 20 Jan 2021 01:56:52 +0000 (02:56 +0100)]
Added test to verify queries failing if int is dumped as int8 or numeric
I wanted these so badly.
Daniele Varrazzo [Wed, 20 Jan 2021 01:46:45 +0000 (02:46 +0100)]
Adapt the dumper used to the value of the objects
Added a second dispatch to allow a dumper to upgrade to a specialised
version. Currently used to dump int to the smallest Postgres type
holding that value and to dump lists of object into typed arrays.
This change allows to write queries more naturally as no ``::int`` cast
should be needed anymore e.g. in date + int or jsonb ->> int.
Only Python implementation; C version to be implemented yet.
Daniele Varrazzo [Tue, 19 Jan 2021 17:29:12 +0000 (18:29 +0100)]
Use prepared statements machinery for executemany
Don't use the anonymous prepared statement anymore, as it doesn't allow
to manage different types.
Daniele Varrazzo [Sun, 17 Jan 2021 06:49:38 +0000 (07:49 +0100)]
Copyright year bumped
egg-info dirs ignored from the root gitignore file to help ag to skpt
them.
Command used:
ag -l copyright | xargs sed -i "s/\(.*copyright[^0-9]*\)\([0-9]\+\)\(-[0-9]\+\)\?\(.*Psycopg.*\)/\1\2-$(date +%Y)\4/I"
Daniele Varrazzo [Sun, 17 Jan 2021 06:19:40 +0000 (07:19 +0100)]
More uniform names for internal objects
Daniele Varrazzo [Sat, 16 Jan 2021 02:51:19 +0000 (03:51 +0100)]
Added tests to detect leaks on copy from
Daniele Varrazzo [Sat, 16 Jan 2021 02:35:20 +0000 (03:35 +0100)]
Added leak test for async cursor
Leak and correctness tests separated.
Daniele Varrazzo [Sat, 16 Jan 2021 02:34:32 +0000 (03:34 +0100)]
Added random tests for data copy
Daniele Varrazzo [Sat, 16 Jan 2021 02:06:56 +0000 (03:06 +0100)]
Declare all loaders to receive an object supporting the buffer interface
There isn't in mypy such an object, so just use `Union[bytes, bytearray,
memoryview]`
Avoid a memory copy passing data from the Transformer to the loaders.
I had started exercising this code path by adding copy tests, but after
dropping the extra copy the same path is now exercised by any select, so
I've stopped doing that.
Daniele Varrazzo [Fri, 15 Jan 2021 23:35:56 +0000 (00:35 +0100)]
Allow passing a list of names to copy.set_types()
Also improved TypeRegistry by recognising a type name even if includes
the array marker '[]' and added get_oid() function.
No test for this object as it's considered internal and leveraged by
several other tests, but I'm sure I will regret it.
Daniele Varrazzo [Fri, 15 Jan 2021 22:23:35 +0000 (23:23 +0100)]
Don't clobber the exception on rollback error in connection exit
Daniele Varrazzo [Fri, 15 Jan 2021 17:26:04 +0000 (18:26 +0100)]
Better integration between C copy functions and C transformer
Daniele Varrazzo [Fri, 15 Jan 2021 17:09:45 +0000 (18:09 +0100)]
Added copy Format helpers to handle text/binary differences
Daniele Varrazzo [Fri, 15 Jan 2021 13:55:29 +0000 (14:55 +0100)]
Added parallel worker to COPY FROM
Have a background writer to push data down the connection while the main
thread is CPU-bound to encode data.
Use a thread to implement the sync worker, a task to implement the
asyncio one.
Daniele Varrazzo [Fri, 15 Jan 2021 01:54:50 +0000 (02:54 +0100)]
End-of-copy logic moved from exit to finish gen
Daniele Varrazzo [Thu, 14 Jan 2021 23:31:21 +0000 (00:31 +0100)]
Added some docs about binary parameters/values.
Daniele Varrazzo [Thu, 14 Jan 2021 22:40:03 +0000 (23:40 +0100)]
Dropped format argument to cursor: use binary= instead
This should make pretty unneeded to import the Format object in client
code.
Daniele Varrazzo [Thu, 14 Jan 2021 21:31:15 +0000 (22:31 +0100)]
Define MAXINT8LEN as a constant
Daniele Varrazzo [Thu, 14 Jan 2021 19:45:00 +0000 (20:45 +0100)]
Dropped intermediate dumpers cache in C transformer too
Daniele Varrazzo [Thu, 14 Jan 2021 17:41:46 +0000 (18:41 +0100)]
Dropped intermediate C Transformer cache creating loaders
Keep in the instance cache the RowLoader instance.
Daniele Varrazzo [Thu, 14 Jan 2021 16:40:05 +0000 (17:40 +0100)]
Less work to create a C Transformer
Daniele Varrazzo [Thu, 14 Jan 2021 16:39:15 +0000 (17:39 +0100)]
Don't create a new transformer if the query is repeated on a cursor
Daniele Varrazzo [Tue, 12 Jan 2021 22:47:39 +0000 (23:47 +0100)]
Added reference leak test for other fetch methods
Daniele Varrazzo [Thu, 14 Jan 2021 15:04:19 +0000 (16:04 +0100)]
Fixed fake string creation: never create \x00 byte
Daniele Varrazzo [Wed, 13 Jan 2021 14:46:49 +0000 (15:46 +0100)]
Choose automatically text or binary format using %s placeholders
A format can be enforced using %b or %t placeholders.
Use binary formats dumpers if available, otherwise use text dumpers.
Python str are an exception: prefer text dumper, because it will dump
with format 0, so Postgres can cast the value to any format.
Special-case empty lists to be always dumped as textual unknown '{}' so
that Postgres can cast them into whatever array it fancies.
Daniele Varrazzo [Wed, 13 Jan 2021 14:28:06 +0000 (15:28 +0100)]
Transformer.dump_sequence returns a list of formats too
Daniele Varrazzo [Wed, 13 Jan 2021 13:55:59 +0000 (14:55 +0100)]
Added xfail to a binary empty array test
Fixing it requires reconsidering how to deal with parameters formats.
Also fixed xpass to make sure to spot the tests to fix after fixing the
above.
Daniele Varrazzo [Wed, 13 Jan 2021 13:38:00 +0000 (14:38 +0100)]
Dropped explicit support for Postgres 9.6
People can still use it, but they should likely disable stored procedure
on a connection and avoid to use executemany, on top of adding plenty of
cast.
Daniele Varrazzo [Wed, 13 Jan 2021 00:02:13 +0000 (01:02 +0100)]
Don't return None as text in Python 9.6
This makes to require a cast on pretty much any placeholder that might
receive a None, which is really asking too much.
Exploratory commit to test what can be done with PG 9.6. The result
shows that in PG 9.6 it's pretty much impossible to use prepared
statements: neither for executemany nor for preparation. Tests failing:
- tests/test_cursor.py::test_executemany_null_first
- tests/test_cursor_async.py::test_executemany_null_first
- tests/test_prepared.py::test_different_types
- tests/test_prepared_async.py::test_different_types
all with: psycopg3.errors.IndeterminateDatatype: could not determine
data type of parameter $x
Daniele Varrazzo [Tue, 12 Jan 2021 22:18:01 +0000 (23:18 +0100)]
Added random data generator and test for memory leaks
Only implemented for binary types, as they are less than text...
Daniele Varrazzo [Tue, 12 Jan 2021 21:47:57 +0000 (22:47 +0100)]
Use cdll instead of pydll in ctypes pq wrapper
cdll releases the GIL. Even for callback functions it doesn't seem a
problem.
Daniele Varrazzo [Tue, 12 Jan 2021 21:47:19 +0000 (22:47 +0100)]
Added [] in the representation of array columns
Daniele Varrazzo [Tue, 12 Jan 2021 21:44:28 +0000 (22:44 +0100)]
Added dumpers row
Speed up repeated dumps on the same query (e.g. in executemany, in copy,
for composite types).
Things are a bit iffy around empty list dumping. As a tradeoff, prefer
that empty list don't behave too differently from non-empty list, but
accept that text and binary work differently (because it doesn't seem
possible to pass an array as unknown in binary, whereas in text '{}'
works ok).
Daniele Varrazzo [Mon, 11 Jan 2021 22:09:19 +0000 (23:09 +0100)]
Set sub-dumper on array dumper on creation in Transformer
Fixes binary dump of empty array and other unforeseen problems such as
the same array dumper dumping lists with two different elements.
Daniele Varrazzo [Mon, 11 Jan 2021 15:04:11 +0000 (16:04 +0100)]
Added test to reproduce array bug found with random data
Daniele Varrazzo [Sun, 10 Jan 2021 18:27:22 +0000 (19:27 +0100)]
Add a reference on the ViewBuffer to the object owning the memory
Daniele Varrazzo [Sun, 10 Jan 2021 04:15:27 +0000 (05:15 +0100)]
Added randomisation of tests run
Fixed a test dealing with an unpredictable libpq behaviour
Daniele Varrazzo [Tue, 12 Jan 2021 15:06:23 +0000 (16:06 +0100)]
Finish correctly a COPY TO operation reading copy.rows()
Close #23
Daniele Varrazzo [Tue, 12 Jan 2021 15:08:17 +0000 (16:08 +0100)]
Added note to remember to document multiple statements + binary
Daniele Varrazzo [Tue, 12 Jan 2021 14:43:12 +0000 (15:43 +0100)]
Reporting pq import error improved
List all the reasons why the implementations failed to import, e.g.
```
Attempts made:
- error importing 'c' wrapper: No module named 'psycopg3_c'
- error importing 'binary' wrapper: No module named 'psycopg3_binary'
- error importing 'python' wrapper: module 'ctypes' has no attribute 'pydll'
```
Daniele Varrazzo [Tue, 12 Jan 2021 01:00:59 +0000 (02:00 +0100)]
Fixed raising unicode encode errors in C
Daniele Varrazzo [Mon, 11 Jan 2021 14:54:31 +0000 (15:54 +0100)]
Added freelist to PGresult
Daniele Varrazzo [Mon, 11 Jan 2021 14:53:26 +0000 (15:53 +0100)]
Added repr and slots to Json wrappers
Daniele Varrazzo [Mon, 11 Jan 2021 14:52:55 +0000 (15:52 +0100)]
Added comment to remember what the array structure is
Daniele Varrazzo [Sun, 10 Jan 2021 19:10:05 +0000 (20:10 +0100)]
Don't use Python enums in C PGconn.status
Leftover from previous cleaning.
Daniele Varrazzo [Tue, 12 Jan 2021 15:09:48 +0000 (16:09 +0100)]
Make cursor.query/params available after closing too
Useful to debug what went wrong with a statement after a cursor context
is exited.
Daniele Varrazzo [Sun, 10 Jan 2021 03:53:24 +0000 (04:53 +0100)]
Avoid local import raising exception in text encoder
Daniele Varrazzo [Sun, 10 Jan 2021 03:46:26 +0000 (04:46 +0100)]
Added C implementation of binary copy load
Return value of the copy read function redefined as memoryview to avoid
unneeded data copying.
Daniele Varrazzo [Sun, 10 Jan 2021 01:56:10 +0000 (02:56 +0100)]
Added C implementation of text copy load
Daniele Varrazzo [Sat, 9 Jan 2021 03:45:08 +0000 (04:45 +0100)]
Added reading row-by-row from Copy
Daniele Varrazzo [Sat, 9 Jan 2021 03:46:09 +0000 (04:46 +0100)]
Added result status in PGresult and Cursor repr
Daniele Varrazzo [Fri, 8 Jan 2021 03:57:22 +0000 (04:57 +0100)]
Further C code streamlining
Daniele Varrazzo [Fri, 8 Jan 2021 02:54:36 +0000 (03:54 +0100)]
Some refcount care in tight loops
Daniele Varrazzo [Fri, 8 Jan 2021 01:57:35 +0000 (02:57 +0100)]
Added test to verify that the C adapters are registered correctly
Daniele Varrazzo [Fri, 8 Jan 2021 01:24:33 +0000 (02:24 +0100)]
Return type of fetch methods improved
Daniele Varrazzo [Fri, 8 Jan 2021 01:03:53 +0000 (02:03 +0100)]
Fixed executemany
D'oh
Daniele Varrazzo [Fri, 8 Jan 2021 00:55:26 +0000 (01:55 +0100)]
Added Transformer.dump_sequence()
Daniele Varrazzo [Thu, 7 Jan 2021 23:07:56 +0000 (00:07 +0100)]
Don't deal with the python-level connection in C adapters
Use the libpq-level connection to shed some overhead.
Daniele Varrazzo [Thu, 7 Jan 2021 21:58:10 +0000 (22:58 +0100)]
Added C interface to Transformer.get_dumper
The Python churning around that call was visible.
Daniele Varrazzo [Thu, 7 Jan 2021 17:51:31 +0000 (18:51 +0100)]
Optimise dumping int to text
Code lifted from Postgres sources. However there is no config phase here
so it will require tweaking to work with other compilers or architectures)
Daniele Varrazzo [Thu, 7 Jan 2021 04:04:39 +0000 (05:04 +0100)]
Added text copy implementation in C
Daniele Varrazzo [Wed, 6 Jan 2021 23:49:05 +0000 (00:49 +0100)]
Accumulate lines into a buffer to send copy data in larger chunks
Daniele Varrazzo [Sat, 2 Jan 2021 21:45:06 +0000 (22:45 +0100)]
Work around __weakref__ + __slots__ + Generics bug on Python 3.6
See bug https://bugs.python.org/issue41451
Daniele Varrazzo [Sat, 2 Jan 2021 13:52:27 +0000 (14:52 +0100)]
Define better registering, importing, subclassing adapters.
- The C dumpers cannot be subclassed. Subclassing from Python makes
difficult to use the c fast path, unless doing relatively expensive
checks.
- C optimised classes are chosen on registration. The types module only
exposes the Python objects, so they can be documented and subclassed.
Override adapter in the respective type modules
This way they are properly exposed to the rest of the Python code.
Note that it will need some trickery to be able to subclass them in
Python: overridden adapter will need the C fast path bypassed if
subclassing is detected.
Export C int subclasses dumpers to Python
Added guards to allow subclassing C adapters
Not sure this is the way to go though. Subclassing might be a rare
enough case to just make the classes final.
The check is implemented at instance level: it would be more efficient
to implement it at class level, however I don't know how to store class
attribute or implement metaclasses in Cython.
Don't make C types subclassable
C types are no more exposed in types, so they won't get subclassed by
final users.
Checking for subclass in adaptation is expensive, so I'd rather not do
it.
Check if a class has a matching optimised class (with the same name) at
registration time.
Daniele Varrazzo [Sat, 2 Jan 2021 02:47:06 +0000 (03:47 +0100)]
Dropped use of decorator to register builtin adapters
Although the registration function is unwieldy already, it's good to get
more flexibility in when it should happen.
It also convinced me that flat is better than nested so the
psycopg3.types can be the facade for all the type-related objects.
More important, it should allow to override certain objects with C
equivalents.
Daniele Varrazzo [Fri, 1 Jan 2021 20:58:01 +0000 (21:58 +0100)]
Added int4 C dumper
Daniele Varrazzo [Fri, 1 Jan 2021 19:19:39 +0000 (20:19 +0100)]
Added C float dumpers
Daniele Varrazzo [Fri, 1 Jan 2021 17:51:32 +0000 (18:51 +0100)]
Added C helper for binary copy formatting
Daniele Varrazzo [Fri, 1 Jan 2021 17:46:22 +0000 (18:46 +0100)]
Create the waiting selector only after the first wait
Sometimes there is no need to wait at all, if the generator exits on the
first result. It happens especially in copy.
Daniele Varrazzo [Thu, 31 Dec 2020 19:25:57 +0000 (20:25 +0100)]
Copy objects refactoring to isolate the row format function
Daniele Varrazzo [Thu, 31 Dec 2020 18:59:22 +0000 (19:59 +0100)]
Added c interface to dumpers
Daniele Varrazzo [Thu, 31 Dec 2020 03:53:19 +0000 (04:53 +0100)]
Global replace of src -> cls params and attribs
Daniele Varrazzo [Thu, 31 Dec 2020 03:27:49 +0000 (04:27 +0100)]
Added AdaptersMap.get_loader(), get_dumper() methods
Internal refactoring to keep separate maps for different formats rather
than using tuples as keys, both in the AdaptersMap and Transformer.
Dumper.src renamed to cls.
Daniele Varrazzo [Wed, 30 Dec 2020 22:34:23 +0000 (23:34 +0100)]
Using new types registration system on the C implementation too
Daniele Varrazzo [Wed, 30 Dec 2020 21:52:18 +0000 (22:52 +0100)]
Loder and Dumper text/binary decorator unified as "builtin"
Allow registration by builtin name too and on several arguments for
terseness.
Daniele Varrazzo [Wed, 30 Dec 2020 21:22:27 +0000 (22:22 +0100)]
Dropped unneeded assignment
Daniele Varrazzo [Wed, 30 Dec 2020 21:02:02 +0000 (22:02 +0100)]
Dropped format parameter from register
Now the loaders/dumper declare their format themselves.
Daniele Varrazzo [Wed, 30 Dec 2020 19:30:59 +0000 (20:30 +0100)]
Added guard for lazy import to avoid the import statement
I'm mot so sure it's always so fast to re-import.
Daniele Varrazzo [Wed, 30 Dec 2020 19:27:56 +0000 (20:27 +0100)]
Added Loader.format and Dumper.format
Daniele Varrazzo [Wed, 30 Dec 2020 17:53:19 +0000 (18:53 +0100)]
Don't convert fformat and binary_tuples to Format in C pq wrapper
Daniele Varrazzo [Wed, 30 Dec 2020 17:23:06 +0000 (18:23 +0100)]
Fixed binary copy with empty content
Daniele Varrazzo [Wed, 30 Dec 2020 15:31:13 +0000 (16:31 +0100)]
Dropped async/sync code duplication using high level generators
Daniele Varrazzo [Wed, 30 Dec 2020 03:03:06 +0000 (04:03 +0100)]
Use slots on PostgresQuery object
Daniele Varrazzo [Wed, 30 Dec 2020 02:53:23 +0000 (03:53 +0100)]
Fast-path in query preparation to skip expensive isinstance
Daniele Varrazzo [Wed, 30 Dec 2020 02:30:10 +0000 (03:30 +0100)]
Added slots on cursor
Daniele Varrazzo [Wed, 6 Jan 2021 23:55:56 +0000 (00:55 +0100)]
Added backer @MaterializeInc. Thank you :)
Daniele Varrazzo [Tue, 29 Dec 2020 17:47:45 +0000 (18:47 +0100)]
Merge branch 'c-optimise'
Daniele Varrazzo [Tue, 29 Dec 2020 17:09:02 +0000 (18:09 +0100)]
Make testable, and test, the different waiting implementation
Also fixed segfault calling flush on a closed connection.
Daniele Varrazzo [Tue, 29 Dec 2020 03:03:34 +0000 (04:03 +0100)]
Added epoll-based wait() implementation
Only used if the EpollSelector would (i.e. epoll is available and it's
the best thing on the menu). Saves noticeable time over the
EpollSelector as there's less Python stuff going on and can use ONESHOT
+ MOD_CTL.
Daniele Varrazzo [Tue, 29 Dec 2020 04:16:44 +0000 (05:16 +0100)]
More tweaks to the Transformer C code
Daniele Varrazzo [Mon, 28 Dec 2020 04:28:01 +0000 (05:28 +0100)]
Tweaking generators.execute C code
Daniele Varrazzo [Sun, 27 Dec 2020 23:20:47 +0000 (00:20 +0100)]
Use the C api to create memoryview objects
Daniele Varrazzo [Sun, 27 Dec 2020 22:55:18 +0000 (23:55 +0100)]
Use cython type declarations for bytes too
Daniele Varrazzo [Sun, 27 Dec 2020 22:25:48 +0000 (23:25 +0100)]
Use cython type declaration rather than python annotation for int
It gives directly a c variable.