From: Daniele Varrazzo Date: Sat, 2 Apr 2022 22:55:18 +0000 (+0200) Subject: docs: add first cut of pipeline mode docs X-Git-Tag: 3.1~120^2~10 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=f62a36c9490c88c17c1e9db31f0a1c8b0601ed06;p=thirdparty%2Fpsycopg.git docs: add first cut of pipeline mode docs --- diff --git a/docs/advanced/index.rst b/docs/advanced/index.rst index a554ae239..6920bd74f 100644 --- a/docs/advanced/index.rst +++ b/docs/advanced/index.rst @@ -18,3 +18,4 @@ usages. cursors adapt prepare + pipeline diff --git a/docs/advanced/pipeline.rst b/docs/advanced/pipeline.rst new file mode 100644 index 000000000..6a50f4cd5 --- /dev/null +++ b/docs/advanced/pipeline.rst @@ -0,0 +1,73 @@ +.. currentmodule:: psycopg + +.. _pipeline-mode: + +Pipeline mode support +===================== + +.. versionadded:: 3.1 + +The *pipeline mode* allows PostgreSQL client applications to send a query +without having to read the result of the previously sent query. Taking +advantage of the pipeline mode, a client will wait less for the server, since +multiple queries/results can be sent/received in a single network transaction. +Pipeline mode can provide a significant performance boost to the application. + +The server executes statements, and returns results, in the order the client +sends them. The server will begin executing the commands in the pipeline +immediately, not waiting for the end of the pipeline. Note that results are +buffered on the server side; the server flushes that buffer when a +*synchronization point* is established. If any statement encounters an error, +the server aborts the current transaction and does not execute any subsequent +command in the queue until the next synchronization point; a +`~errors.PipelineAborted` exception is raised for each such command (including +a `~Connection.rollback()`). Query processing resumes after the +synchronization point. + +Pipeline mode is most useful when the server is distant, i.e., network latency +(“ping time”) is high, and also when many small operations are being performed +in rapid succession. There is usually less benefit in using pipelined commands +when each query takes many multiples of the client/server round-trip time to +execute. A 100-statement operation run on a server 300 ms round-trip-time away +would take 30 seconds in network latency alone without pipelining; with +pipelining it may spend as little as 0.3 s waiting for results from the +server. + +The pipeline mode is available on any currently supported PostgreSQL version +but, in order to make use of it, the client must use a libpq from PostgreSQL +14 or higher. You can use `Pipeline.is_supported()` to make sure your client +has the right library. + +.. seealso:: The `PostgreSQL pipeline mode documentation`__ contains many + details around when it is most useful to use the pipeline mode and about + errors management and interaction with transactions. + + .. __: https://www.postgresql.org/docs/14/libpq-pipeline-mode.html + +Psycopg supports the pipeline mode via the `Connection.pipeline()` method. The +method is a context manager: at the end of the ``with`` block, the connection +resumes the normal operation mode. + +Within the pipeline block, you can use one or more cursors to execute several +operations, using `~Cursor.execute()` and `~Cursor.executemany()`. Unlike in +normal mode, Psycopg will not wait for the server to receive the result of +each query, which will be received in batches when a synchronization point is +established. + +Psycopg can establish a synchronization points: + +- using the `Pipeline.sync()` method; +- at the end of a `!Pipeline` block; +- using a fetch method such as `Cursor.fetchone()`. + +The server might perform a sync on its own initiative, for instance when the +query buffer is full. + +When a sync is performed, all the pending results are sent back to the cursors +which executed them. If a cursor had run more than one query, it will receive +more than one result; results after the first will be available, in their +execution order, using `~Cursor.nextset()`. + +.. note:: + Starting from Psycopg 3.1, `Cursor.executemany()` is optimised to make use + of pipeline mode. diff --git a/docs/api/connections.rst b/docs/api/connections.rst index 666464473..8679271c4 100644 --- a/docs/api/connections.rst +++ b/docs/api/connections.rst @@ -160,6 +160,25 @@ The `!Connection` class See :ref:`query-parameters` for all the details about executing queries. + .. automethod:: pipeline + + The method is a context manager: you should call it using:: + + with conn.pipeline() as p: + ... + + At the end of the block, a synchronization point is established and + the connection returns in normal mode. + + You can call the method recursively from within a pipeline block. + Innermost blocks will establish a synchronization point on exit, but + pipeline mode will be kept until the outermost block exits. + + See :ref:`pipeline-mode` for details. + + .. versionadded:: 3.1 + + .. rubric:: Transaction management methods For details see :ref:`transactions`. @@ -266,8 +285,6 @@ The `!Connection` class .. automethod:: fileno - .. automethod:: pipeline - .. _tpc-methods: @@ -431,6 +448,16 @@ The `!AsyncConnection` class .. autoattribute:: row_factory .. automethod:: execute + + .. automethod:: pipeline + + .. note:: + + It must be called as:: + + async with conn.pipeline() as p: + ... + .. automethod:: commit .. automethod:: rollback @@ -449,15 +476,6 @@ The `!AsyncConnection` class .. automethod:: set_read_only .. automethod:: set_deferrable - .. automethod:: pipeline - - .. note:: - - It must be called as:: - - async with conn.pipeline(): - ... - .. automethod:: tpc_prepare .. automethod:: tpc_commit .. automethod:: tpc_rollback diff --git a/docs/api/cursors.rst b/docs/api/cursors.rst index 282436c88..d7c75bb69 100644 --- a/docs/api/cursors.rst +++ b/docs/api/cursors.rst @@ -102,7 +102,9 @@ The `!Cursor` class .. versionchanged:: 3.1 - added ``returning`` parameter to receive query results. + - Added ``returning`` parameter to receive query results. + - Preformance optimised by making use of the pipeline mode, when + using libpq 14 or newer. .. automethod:: copy diff --git a/docs/api/errors.rst b/docs/api/errors.rst index ca567be26..cd1f3c6b3 100644 --- a/docs/api/errors.rst +++ b/docs/api/errors.rst @@ -83,6 +83,7 @@ In addition to the standard DB-API errors, Psycopg defines a few more specific ones. .. autoexception:: ConnectionTimeout() +.. autoexception:: PipelineAborted() diff --git a/docs/api/objects.rst b/docs/api/objects.rst index 419606c24..14a4b08f2 100644 --- a/docs/api/objects.rst +++ b/docs/api/objects.rst @@ -193,6 +193,26 @@ Notifications The PID of the backend process which sent the notification. +Pipeline-related objects +------------------------ + +See :ref:`pipeline-mode` for details. + +.. autoclass:: Pipeline + + This objects is returned by `Connection.pipeline()`. + + .. automethod:: sync + .. automethod:: is_supported + + +.. autoclass:: AsyncPipeline + + This objects is returned by `AsyncConnection.pipeline()`. + + .. automethod:: sync + + Transaction-related objects --------------------------- diff --git a/docs/news.rst b/docs/news.rst index 08ad5151f..ac33b9a34 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -13,6 +13,7 @@ Future releases Psycopg 3.1 (unreleased) ^^^^^^^^^^^^^^^^^^^^^^^^ +- Add :ref:`Pipeline mode ` (:ticket:`#74`). - Add :ref:`Two-Phase Commit ` support (:ticket:`#72`). - Add :ref:`adapt-enum` (:ticket:`#274`). - Add ``returning`` parameter to `~Cursor.executemany()` to retrieve query diff --git a/psycopg/psycopg/_pipeline.py b/psycopg/psycopg/_pipeline.py index f9d967c1d..80e47acb8 100644 --- a/psycopg/psycopg/_pipeline.py +++ b/psycopg/psycopg/_pipeline.py @@ -66,7 +66,7 @@ class BasePipeline: @staticmethod def is_supported() -> bool: - """Return `True` if the psycopg libpq wrapper suports pipeline mode.""" + """Return `!True` if the psycopg libpq wrapper suports pipeline mode.""" if BasePipeline._is_supported is None: # Support only depends on the libpq functions available in the pq # wrapper, not on the database version. diff --git a/psycopg/psycopg/connection.py b/psycopg/psycopg/connection.py index fc92c0512..8d01d8630 100644 --- a/psycopg/psycopg/connection.py +++ b/psycopg/psycopg/connection.py @@ -883,7 +883,7 @@ class Connection(BaseConnection[Row]): @contextmanager def pipeline(self) -> Iterator[Pipeline]: - """Context manager to switch the connection into pipeline mode.""" + """Switch the connection into pipeline mode.""" with self.lock: pipeline = self._pipeline if pipeline is None: