From: Daniele Varrazzo <daniele.varrazzo@gmail.com>
Date: Sun, 1 May 2022 22:56:42 +0000 (+0200)
Subject: perf: avoid unnecessary recvfrom() in cursor.stream()
X-Git-Tag: 3.1~125^2
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=7aba716e92bde573a63c2798172c4d3eed41a680;p=thirdparty%2Fpsycopg.git

perf: avoid unnecessary recvfrom() in cursor.stream()

Call PQisBusy() before PQconsumeInput() on fetching results. If not
busy, don't call PQconsumeInput() at all but just go to fetching results
and notifications.

This is especially useful in single-row mode because most of the times
the libpq can produce several results after a single network fetch.
Previously we were calling PQconsumeInput() also when results were
already on the client and there was nothing new to read, which forced
the libpq to run a select() to tell apart a lack of data from an EOF,
see `the grumble`_, and caused the overhead reported in #286.

Close #286.

.. _the grumble: https://github.com/postgres/postgres/blob/ed57cac84d1c5642737dab1e4c4b8cb4f0c4305f/src/interfaces/libpq/fe-misc.c#L681
---

diff --git a/docs/news.rst b/docs/news.rst
index 3a3d0b897..24c9cb1ae 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -25,6 +25,12 @@ Psycopg 3.1 (unreleased)
 - Drop support for Python 3.6.
 
 
+Psycopg 3.0.13 (unreleased)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Fix `Cursor.stream()` slowness (:ticket:`#286`).
+
+
 Current release
 ---------------
 
diff --git a/psycopg/psycopg/generators.py b/psycopg/psycopg/generators.py
index 4dd2d72ec..86b5a0bba 100644
--- a/psycopg/psycopg/generators.py
+++ b/psycopg/psycopg/generators.py
@@ -143,14 +143,16 @@ def fetch(pgconn: PGconn) -> PQGen[Optional[PGresult]]:
 
     Return a result from the database (whether success or error).
     """
-    while 1:
-        pgconn.consume_input()
-        if not pgconn.is_busy():
-            break
+    if pgconn.is_busy():
         yield Wait.R
+        while True:
+            pgconn.consume_input()
+            if not pgconn.is_busy():
+                break
+            yield Wait.R
 
     # Consume notifies
-    while 1:
+    while True:
         n = pgconn.notifies()
         if not n:
             break
diff --git a/psycopg_c/psycopg_c/_psycopg/generators.pyx b/psycopg_c/psycopg_c/_psycopg/generators.pyx
index 473e0ea8a..4afb53508 100644
--- a/psycopg_c/psycopg_c/_psycopg/generators.pyx
+++ b/psycopg_c/psycopg_c/_psycopg/generators.pyx
@@ -159,18 +159,20 @@ def fetch(pq.PGconn pgconn) -> PQGen[Optional[PGresult]]:
     cdef object notify_handler = pgconn.notify_handler
     cdef libpq.PGresult *pgres
 
-    while 1:
-        with nogil:
-            cires = libpq.PQconsumeInput(pgconn_ptr)
-            if cires == 1:
-                ibres = libpq.PQisBusy(pgconn_ptr)
-
-        if 1 != cires:
-            raise e.OperationalError(
-                f"consuming input failed: {error_message(pgconn)}")
-        if not ibres:
-            break
+    if libpq.PQisBusy(pgconn_ptr):
         yield WAIT_R
+        while 1:
+            with nogil:
+                cires = libpq.PQconsumeInput(pgconn_ptr)
+                if cires == 1:
+                    ibres = libpq.PQisBusy(pgconn_ptr)
+
+            if 1 != cires:
+                raise e.OperationalError(
+                    f"consuming input failed: {error_message(pgconn)}")
+            if not ibres:
+                break
+            yield WAIT_R
 
     # Consume notifies
     if notify_handler is not None: