From: Daniele Varrazzo Date: Sun, 18 Feb 2024 00:04:34 +0000 (+0100) Subject: docs(pool): reorder sections in a way that makes more sense X-Git-Tag: 3.2.0~72 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=refs%2Fpull%2F738%2Fhead;p=thirdparty%2Fpsycopg.git docs(pool): reorder sections in a way that makes more sense --- diff --git a/docs/advanced/pool.rst b/docs/advanced/pool.rst index 5badde817..e62c292b4 100644 --- a/docs/advanced/pool.rst +++ b/docs/advanced/pool.rst @@ -131,70 +131,105 @@ context stared by `~ConnectionPool.connection()`): the queue, to the first client waiting. -Debugging pool usage --------------------- - -The pool uses the `logging` module to log some key operations to the -``psycopg.pool`` logger. If you are trying to debug the pool behaviour you may -try to log at least the ``INFO`` operations on that logger. +Other ways to create a pool +--------------------------- -For example, the script: +Using the pool as a context manager is not mandatory: pools can be created and +used without using the context pattern. However, using the context is the +safest way to manage its resources. -.. code:: python +When the pool is created, if its `!open` parameter is `!True`, the connection +process starts immediately. In a simple program you might create a pool as a +global object and use it from the rest of your code:: - import time - import logging - from concurrent.futures import ThreadPoolExecutor, as_completed + # module db.py in your program from psycopg_pool import ConnectionPool - logging.basicConfig( - level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s" - ) - logging.getLogger("psycopg.pool").setLevel(logging.INFO) + pool = ConnectionPool(..., open=True, ...) + # the pool starts connecting immediately. - pool = ConnectionPool(min_size=2) - pool.wait() - logging.info("pool ready") + # in another module + from .db import pool - def square(n): + def my_function(): with pool.connection() as conn: - time.sleep(1) - rec = conn.execute("SELECT %s * %s", (n, n)).fetchone() - logging.info(f"The square of {n} is {rec[0]}.") + conn.execute(...) - with ThreadPoolExecutor(max_workers=4) as executor: - futures = [executor.submit(square, n) for n in range(4)] - for future in as_completed(futures): - future.result() +Using this pattern, the pool will start the connection process already at +import time. If that's too early, and you want to delay opening connections +until the application is ready, you can specify to create a closed pool and +call the `~ConnectionPool.open()` method (and optionally the +`~ClonnectionPool.close()` method) at application startup/shutdown. For +example, in FastAPI, you can use `startup/shutdown events`__:: -might print something like: + pool = AsyncConnectionPool(..., open=False, ...) -.. code:: text + @app.on_event("startup") + async def open_pool(): + await pool.open() - 2023-09-20 11:02:39,718 INFO psycopg.pool: waiting for pool 'pool-1' initialization - 2023-09-20 11:02:39,720 INFO psycopg.pool: adding new connection to the pool - 2023-09-20 11:02:39,720 INFO psycopg.pool: adding new connection to the pool - 2023-09-20 11:02:39,720 INFO psycopg.pool: pool 'pool-1' is ready to use - 2023-09-20 11:02:39,720 INFO root: pool ready - 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' - 2023-09-20 11:02:39,721 INFO psycopg.pool: connection given by 'pool-1' - 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' - 2023-09-20 11:02:39,721 INFO psycopg.pool: connection given by 'pool-1' - 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' - 2023-09-20 11:02:39,722 INFO psycopg.pool: connection requested from 'pool-1' - 2023-09-20 11:02:40,724 INFO root: The square of 0 is 0. - 2023-09-20 11:02:40,724 INFO root: The square of 1 is 1. - 2023-09-20 11:02:40,725 INFO psycopg.pool: returning connection to 'pool-1' - 2023-09-20 11:02:40,725 INFO psycopg.pool: connection given by 'pool-1' - 2023-09-20 11:02:40,725 INFO psycopg.pool: returning connection to 'pool-1' - 2023-09-20 11:02:40,726 INFO psycopg.pool: connection given by 'pool-1' - 2023-09-20 11:02:41,728 INFO root: The square of 3 is 9. - 2023-09-20 11:02:41,729 INFO root: The square of 2 is 4. - 2023-09-20 11:02:41,729 INFO psycopg.pool: returning connection to 'pool-1' - 2023-09-20 11:02:41,730 INFO psycopg.pool: returning connection to 'pool-1' + @app.on_event("shutdown") + async def close_pool(): + await pool.close() -Please do not rely on the messages generated to remain unchanged across -versions: they don't constitute a stable interface. +.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown + +.. warning:: + The current default for the `!open` parameter is `!True`. However this + proved to be not the best idea and, in future releases, the default might + be changed to `!False`. As a consequence, if you rely on the pool to be + opened on creation, you should specify `!open=True` explicitly. + +.. warning:: + Opening an async pool in the constructor is deprecated and will be removed + in the future. When using `AsyncConnectionPool` you should call `await + pool.open()` or `async with ... as pool` explicitly. + + +.. _null-pool: + +Null connection pools +--------------------- + +.. versionadded:: 3.1 + +Sometimes you may want leave the choice of using or not using a connection +pool as a configuration parameter of your application. For instance, you might +want to use a pool if you are deploying a "large instance" of your application +and can dedicate it a handful of connections; conversely you might not want to +use it if you deploy the application in several instances, behind a load +balancer, and/or using an external connection pool process such as PgBouncer. + +Switching between using or not using a pool requires some code change, because +the `ConnectionPool` API is different from the normal `~psycopg.connect()` +function and because the pool can perform additional connection configuration +(in the `!configure` parameter) that, if the pool is removed, should be +performed in some different code path of your application. + +The `!psycopg_pool` 3.1 package introduces the `NullConnectionPool` class. +This class has the same interface, and largely the same behaviour, of the +`!ConnectionPool`, but doesn't create any connection beforehand. When a +connection is returned, unless there are other clients already waiting, it +is closed immediately and not kept in the pool state. + +A null pool is not only a configuration convenience, but can also be used to +regulate the access to the server by a client program. If `!max_size` is set to +a value greater than 0, the pool will make sure that no more than `!max_size` +connections are created at any given time. If more clients ask for further +connections, they will be queued and served a connection as soon as a previous +client has finished using it, like for the basic pool. Other mechanisms to +throttle client requests (such as `!timeout` or `!max_waiting`) are respected +too. + +.. note:: + + Queued clients will be handed an already established connection, as soon + as a previous client has finished using it (and after the pool has + returned it to idle state and called `!reset()` on it, if necessary). + +Because normally (i.e. unless queued) every client will be served a new +connection, the time to obtain the connection is paid by the waiting client; +background workers are not normally involved in obtaining new connections. Pool connection and sizing @@ -273,105 +308,72 @@ A simple implementation is available as the static method ... -Other ways to create a pool ---------------------------- - -Using the pool as a context manager is not mandatory: pools can be created and -used without using the context pattern. However, using the context is the -safest way to manage its resources. - -When the pool is created, if its `!open` parameter is `!True`, the connection -process starts immediately. In a simple program you might create a pool as a -global object and use it from the rest of your code:: - - # module db.py in your program - from psycopg_pool import ConnectionPool - - pool = ConnectionPool(..., open=True, ...) - # the pool starts connecting immediately. - - # in another module - from .db import pool - - def my_function(): - with pool.connection() as conn: - conn.execute(...) - -Using this pattern, the pool will start the connection process already at -import time. If that's too early, and you want to delay opening connections -until the application is ready, you can specify to create a closed pool and -call the `~ConnectionPool.open()` method (and optionally the -`~ClonnectionPool.close()` method) at application startup/shutdown. For -example, in FastAPI, you can use `startup/shutdown events`__:: - - pool = AsyncConnectionPool(..., open=False, ...) - - @app.on_event("startup") - async def open_pool(): - await pool.open() +.. _pool-logging: - @app.on_event("shutdown") - async def close_pool(): - await pool.close() +Pool operations logging +----------------------- -.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown - -.. warning:: - The current default for the `!open` parameter is `!True`. However this - proved to be not the best idea and, in future releases, the default might - be changed to `!False`. As a consequence, if you rely on the pool to be - opened on creation, you should specify `!open=True` explicitly. - -.. warning:: - Opening an async pool in the constructor is deprecated and will be removed - in the future. When using `AsyncConnectionPool` you should call `await - pool.open()` or `async with ... as pool` explicitly. +The pool uses the `logging` module to log some key operations to the +``psycopg.pool`` logger. If you are trying to debug the pool behaviour you may +try to log at least the ``INFO`` operations on that logger. +For example, the script: -.. _null-pool: +.. code:: python -Null connection pools ---------------------- + import time + import logging + from concurrent.futures import ThreadPoolExecutor, as_completed + from psycopg_pool import ConnectionPool -.. versionadded:: 3.1 + logging.basicConfig( + level=logging.INFO, format="%(asctime)s %(levelname)s %(name)s: %(message)s" + ) + logging.getLogger("psycopg.pool").setLevel(logging.INFO) -Sometimes you may want leave the choice of using or not using a connection -pool as a configuration parameter of your application. For instance, you might -want to use a pool if you are deploying a "large instance" of your application -and can dedicate it a handful of connections; conversely you might not want to -use it if you deploy the application in several instances, behind a load -balancer, and/or using an external connection pool process such as PgBouncer. + pool = ConnectionPool(min_size=2) + pool.wait() + logging.info("pool ready") -Switching between using or not using a pool requires some code change, because -the `ConnectionPool` API is different from the normal `~psycopg.connect()` -function and because the pool can perform additional connection configuration -(in the `!configure` parameter) that, if the pool is removed, should be -performed in some different code path of your application. + def square(n): + with pool.connection() as conn: + time.sleep(1) + rec = conn.execute("SELECT %s * %s", (n, n)).fetchone() + logging.info(f"The square of {n} is {rec[0]}.") -The `!psycopg_pool` 3.1 package introduces the `NullConnectionPool` class. -This class has the same interface, and largely the same behaviour, of the -`!ConnectionPool`, but doesn't create any connection beforehand. When a -connection is returned, unless there are other clients already waiting, it -is closed immediately and not kept in the pool state. + with ThreadPoolExecutor(max_workers=4) as executor: + futures = [executor.submit(square, n) for n in range(4)] + for future in as_completed(futures): + future.result() -A null pool is not only a configuration convenience, but can also be used to -regulate the access to the server by a client program. If `!max_size` is set to -a value greater than 0, the pool will make sure that no more than `!max_size` -connections are created at any given time. If more clients ask for further -connections, they will be queued and served a connection as soon as a previous -client has finished using it, like for the basic pool. Other mechanisms to -throttle client requests (such as `!timeout` or `!max_waiting`) are respected -too. +might print something like: -.. note:: +.. code:: text - Queued clients will be handed an already established connection, as soon - as a previous client has finished using it (and after the pool has - returned it to idle state and called `!reset()` on it, if necessary). + 2023-09-20 11:02:39,718 INFO psycopg.pool: waiting for pool 'pool-1' initialization + 2023-09-20 11:02:39,720 INFO psycopg.pool: adding new connection to the pool + 2023-09-20 11:02:39,720 INFO psycopg.pool: adding new connection to the pool + 2023-09-20 11:02:39,720 INFO psycopg.pool: pool 'pool-1' is ready to use + 2023-09-20 11:02:39,720 INFO root: pool ready + 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' + 2023-09-20 11:02:39,721 INFO psycopg.pool: connection given by 'pool-1' + 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' + 2023-09-20 11:02:39,721 INFO psycopg.pool: connection given by 'pool-1' + 2023-09-20 11:02:39,721 INFO psycopg.pool: connection requested from 'pool-1' + 2023-09-20 11:02:39,722 INFO psycopg.pool: connection requested from 'pool-1' + 2023-09-20 11:02:40,724 INFO root: The square of 0 is 0. + 2023-09-20 11:02:40,724 INFO root: The square of 1 is 1. + 2023-09-20 11:02:40,725 INFO psycopg.pool: returning connection to 'pool-1' + 2023-09-20 11:02:40,725 INFO psycopg.pool: connection given by 'pool-1' + 2023-09-20 11:02:40,725 INFO psycopg.pool: returning connection to 'pool-1' + 2023-09-20 11:02:40,726 INFO psycopg.pool: connection given by 'pool-1' + 2023-09-20 11:02:41,728 INFO root: The square of 3 is 9. + 2023-09-20 11:02:41,729 INFO root: The square of 2 is 4. + 2023-09-20 11:02:41,729 INFO psycopg.pool: returning connection to 'pool-1' + 2023-09-20 11:02:41,730 INFO psycopg.pool: returning connection to 'pool-1' -Because normally (i.e. unless queued) every client will be served a new -connection, the time to obtain the connection is paid by the waiting client; -background workers are not normally involved in obtaining new connections. +Please do not rely on the messages generated to remain unchanged across +versions: they don't constitute a stable interface. .. _pool-stats: