From: Daniele Varrazzo Date: Sat, 7 Oct 2023 01:49:46 +0000 (+0200) Subject: docs(pool): simpler, less detailed, more opinionated pool docs page X-Git-Tag: pool-3.2.0~19 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=209caba930d1c81a1608fd2f0847093f58be8b3f;p=thirdparty%2Fpsycopg.git docs(pool): simpler, less detailed, more opinionated pool docs page Show a very basic usage example, then give a more detailed connection life cycle illustration. Push lower and keep shorter the alternative creation methods. --- diff --git a/docs/advanced/pool.rst b/docs/advanced/pool.rst index 9b7b29cee..440199c04 100644 --- a/docs/advanced/pool.rst +++ b/docs/advanced/pool.rst @@ -21,128 +21,114 @@ the pool operations. :ref:`pool-installation`. -Pool life cycle ---------------- +Basic connection pool usage +--------------------------- -A simple way to use the pool is to create a single instance of it, as a -global object, and to use this object in the rest of the program, allowing -other functions, modules, threads to use it:: +A `ConnectionPool` object can be used to request connections from multiple +concurrent threads. A simple and safe way to use it is as a *context manager*. +Within the `!with` block, you can request the pool a connection using the +`~ConnectionPool.connection()` method, and use it as a context manager too:: - # module db.py in your program - from psycopg_pool import ConnectionPool - - pool = ConnectionPool(conninfo, **kwargs) - # the pool starts connecting immediately. + with ConnectionPool(...) as pool: + with pool.connection() as conn: + conn.execute("SELECT something FROM somewhere ...") - # in another module - from .db import pool + with conn.cursor() as cur: + cur.execute("SELECT something else...") - def my_function(): - with pool.connection() as conn: - conn.execute(...) + # At the end of the `connection()` context, the transaction is committed + # or rolled back, and the connection returned to the pool -Ideally you may want to call `~ConnectionPool.close()` when the use of the -pool is finished. Failing to call `!close()` at the end of the program is not -terribly bad: probably it will just result in some warnings printed on stderr. -However, if you think that it's sloppy, you could use the `atexit` module to -have `!close()` called at the end of the program. + # At the end of the pool context, all the resources used by the pool are released -If you want to avoid starting to connect to the database at import time, and -want to wait for the application to be ready, you can create the pool using -`!open=False`, and call the `~ConnectionPool.open()` and -`~ConnectionPool.close()` methods when the conditions are right. Certain -frameworks provide callbacks triggered when the program is started and stopped -(for instance `FastAPI startup/shutdown events`__): they are perfect to -initiate and terminate the pool operations:: +The `!connection()` context behaves like the `~psycopg.Connection` object +context: at the end of the block, if there is a transaction open, it will be +committed if the context is exited normally, or rolled back if the context is +exited with an exception. See :ref:`transaction-context` for details. - pool = ConnectionPool(conninfo, open=False, **kwargs) +The pool manages a certain amount of connections (between `!min_size` and +`!max_size`). If the pool has a connection ready in its state, it is served +immediately to the `~connection()` caller, otherwise the caller is put in a +queue and is served a connection as soon as it's available. - @app.on_event("startup") - def open_pool(): - pool.open() +If instead of threads your application uses async code you can use the +`AsyncConnectionPool` instead and use the `!async` and `!await` keywords with +the methods requiring them:: - @app.on_event("shutdown") - def close_pool(): - pool.close() + async with AsyncConnectionPool(...) as pool: + async with pool.connection() as conn: + await conn.execute("SELECT something FROM somewhere ...") -.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown + with conn.cursor() as cur: + await cur.execute("SELECT something else...") -Creating a single pool as a global variable is not the mandatory use: your -program can create more than one pool, which might be useful to connect to -more than one database, or to provide different types of connections, for -instance to provide separate read/write and read-only connections. The pool -also acts as a context manager and is open and closed, if necessary, on -entering and exiting the context block:: - from psycopg_pool import ConnectionPool +Pool startup check +------------------ - with ConnectionPool(conninfo, **kwargs) as pool: - run_app(pool) +After a pool is open, it can accept new clients even if it doesn't have +`!min_size` connections ready yet. However, if the application is +misconfigured and cannot connect to the database server, the clients will +block until failing with a `PoolTimeout`. - # the pool is now closed +If you want to make sure early in the application lifetime that the +environment is well configured, you can use the `~ConnectionPool.wait()` method +after opening the pool, which will block until `!min_size` connections have +been acquired, or fail with a `!PoolTimeout` if it doesn't happen in time:: -When the pool is open, the pool's background workers start creating the -requested `!min_size` connections, while the constructor (or the `!open()` -method) returns immediately. This allows the program some leeway to start -before the target database is up and running. However, if your application is -misconfigured, or the network is down, it means that the program will be able -to start, but the threads requesting a connection will fail with a -`PoolTimeout` only after the timeout on `~ConnectionPool.connection()` is -expired. If this behaviour is not desirable (and you prefer your program to -crash hard and fast, if the surrounding conditions are not right, because -something else will respawn it) you should call the `~ConnectionPool.wait()` -method after creating the pool, or call `!open(wait=True)`: these methods will -block until the pool is full, or will raise a `PoolTimeout` exception if the -pool isn't ready within the allocated time. + with ConnectionPool(...) as pool: + pool.wait() + use_the(pool) Connections life cycle ---------------------- -The pool background workers create connections according to the parameters -`!conninfo`, `!kwargs`, and `!connection_class` passed to `ConnectionPool` -constructor, invoking something like :samp:`{connection_class}({conninfo}, -**{kwargs})`. Once a connection is created it is also passed to the -`!configure()` callback, if provided, after which it is put in the pool (or -passed to a client requesting it, if someone is already knocking at the door). +When the pool needs a new connection (because it was just opened, or because +an existing connection was closed, or because a spike of activity requires new +connections), it uses a background pool worker to prepare it in the background: -If a connection expires (it passes `!max_lifetime`), or is returned to the pool -in broken state, or is found closed by `~ConnectionPool.check()`), then the -pool will dispose of it and will start a new connection attempt in the -background. +- the worker creates a connection according to the parameters `!conninfo`, + `!kwargs`, and `!connection_class` passed to `ConnectionPool` constructor, + calling something similar to :samp:`{connection_class}({conninfo}, + **{kwargs})`; +- if a `!configure` callback was provided, it is called with the new connection + as parameter. This can be used, for instance, to configure the connection + adapters. -Using connections from the pool -------------------------------- +Once the connection is prepared, it is stored in the pool state, or it is +passed to a client if someone is already in the requests queue. -The pool can be used to request connections from multiple threads or -concurrent tasks - it is hardly useful otherwise! If more connections than the -ones available in the pool are requested, the requesting threads are queued -and are served a connection as soon as one is available, either because -another client has finished using it or because the pool is allowed to grow -(when `!max_size` > `!min_size`) and a new connection is ready. +When a client asks for a connection (typically entering a +`~ConnectionPool.connection()` context): -The main way to use the pool is to obtain a connection using the -`~ConnectionPool.connection()` context, which returns a `~psycopg.Connection` -or subclass:: +- if there is a connection available in the pool, it is served to the client + immediately; - with my_pool.connection() as conn: - conn.execute("what you want") +- if no connection is available, the client is put in a queue, and will be + served a connection once one becomes available (because returned by another + client or because a new one is created). -The `!connection()` context behaves like the `~psycopg.Connection` object -context: at the end of the block, if there is a transaction open, it will be -committed, or rolled back if the context is exited with as exception. +When a client has finished to use the connection (typically at the end of the +context stared by `~ConnectionPool.connection()`): + +- if there is a transaction open, the transaction is committed (if the block + is exited normally) or rolled back (if it is exited with an exception); + +- if a `!reset` callback was provided, the connection is passed to it, to + allow application-specific cleanup if needed; -At the end of the block the connection is returned to the pool and shouldn't -be used anymore by the code which obtained it. If a `!reset()` function is -specified in the pool constructor, it is called on the connection before -returning it to the pool. Note that the `!reset()` function is called in a -worker thread, so that the thread which used the connection can keep its -execution without being slowed down by it. +- if, along this process, the connection is found in broken state, or if it + passed the `!max_lifetime` configured at pool creation, it is discarded and + a new connection is requested to a worker; + +- the connection is finally returned to the pool, or, if there are clients in + the queue, to the first client waiting. Debugging pool usage -^^^^^^^^^^^^^^^^^^^^ +-------------------- The pool uses the `logging` module to log some key operations to the ``psycopg.pool`` logger. If you are trying to debug the pool behaviour you may @@ -254,6 +240,56 @@ to tune the configuration parameters. The size of the pool can also be changed at runtime using the `~ConnectionPool.resize()` method. +Other ways to create a pool +--------------------------- + +Using the pool as a context manager is not mandatory: pools can be created and +used without using the context pattern. However, using the context is the +safest way to manage its resources. + +When the pool is created, if its `!open` parameter is `!True`, the connection +process starts immediately. In a simple program you might create a pool as a +global object and use it from the rest of your code:: + + # module db.py in your program + from psycopg_pool import ConnectionPool + + pool = ConnectionPool(..., open=True, ...) + # the pool starts connecting immediately. + + # in another module + from .db import pool + + def my_function(): + with pool.connection() as conn: + conn.execute(...) + +Using this pattern, the pool will start the connection process already at +import time. If that's too early, and you want to delay opening connections +until the application is ready, you can specify to create a closed pool and +call the `~ConnectionPool.open()` method (and optionally the +`~ClonnectionPool.close()` method) at application startup/shutdown. For +example, in FastAPI, you can use `startup/shutdown events`__:: + + pool = ConnectionPool(..., open=False, ...) + + @app.on_event("startup") + def open_pool(): + pool.open() + + @app.on_event("shutdown") + def close_pool(): + pool.close() + +.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown + +.. warning:: + The current default for the `!open` parameter is `!True`. However this + proved to be not the best idea and, in future releases, the default might + be changed to `!False`. As a consequence, if you rely on the pool to be + opened on creation, you should specify `!open=True` explicitly. + + .. _null-pool: Null connection pools