From: Daniele Varrazzo Date: Sun, 9 Jan 2022 14:04:55 +0000 (+0100) Subject: Improve pool docs, especially pools and connections life cycle X-Git-Tag: pool-3.1~23 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=448245d80b6e0c6f87edd760317684ae4caab1e8;p=thirdparty%2Fpsycopg.git Improve pool docs, especially pools and connections life cycle --- diff --git a/docs/advanced/pool.rst b/docs/advanced/pool.rst index e370183d3..73bc267ca 100644 --- a/docs/advanced/pool.rst +++ b/docs/advanced/pool.rst @@ -24,62 +24,103 @@ the pool operations. Pool life cycle --------------- -A typical way to use the pool is to create a single instance of it, as a +A simple way to use the pool is to create a single instance of it, as a global object, and to use this object in the rest of the program, allowing -other functions, modules, threads to use it. This is only a common use -however, and not the necessary one; in particular the connection pool acts as -a context manager and can be closed automatically at the end of its ``with`` -block:: +other functions, modules, threads to use it:: + # module db.py in your program from psycopg_pool import ConnectionPool - with ConnectionPool(conninfo, **kwargs) as my_pool: - run_app(my_pool) + pool = ConnectionPool(conninfo, **kwargs) + # the pool starts connecting immediately. + + # in another module + from .db import pool + + def my_function(): + with poolconnection() as conn: + conn.execute(...) + +Ideally you may want to call `~ConnectionPool.close()` when the use of the +pool is finished. Failing to call `!close()` at the end of the program is not +terribly bad: probably it will just result in some warnings printed on stderr. +However, if you think that it's sloppy, you could use the `atexit` module to +have `!close()` called at the end of the program. + +If you want to avoid starting to connect to the database at import time, and +want to wait for the application to be ready, you can create the pool using +*open* = `!False`, and call the `~ConnectionPool.open()` and +`~ConnectionPool.close()` methods when the conditions are right. Certain +frameworks provide callbacks triggered when the program is started and stopped +(for instance `FastAPI startup/shutdown events`__): they are perfect to +initiate and terminate the pool operations:: + + pool = ConnectionPool(conninfo, open=False, **kwargs) + + @app.on_event("startup") + def open_pool(): + pool.open() + + @app.on_event("shutdown") + def close_pool(): + pool.close() + +.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown + +Creating a single pool as a global variable is not the mandatory use: your +program can create more than one pool, which might be useful to connect to +more than one database, or to provide different types of connections, for +instance to provide separate read/write and read-only connections. The pool +also acts as a context manager and is open and closed, if necessary, on +entering and exiting the context block:: + + from psycopg_pool import ConnectionPool + + with ConnectionPool(conninfo, **kwargs) as pool: + run_app(pool) # the pool is now closed -If necessary, or convenient, your application may create more than one pool, -for instance to connect to more than one database or to provide separate -read-only and read/write connections. +When the pool is open, the pool's background workers start creating the +requested *min_size* connections, while the constructor (or the `!open()` +method) returns immediately. This allows the program some leeway to start +before the target database is up and running. However, if your application is +misconfigured, or the network is down, it means that the program will be able +to start, but the threads requesting a connection will fail with a +`PoolTimeout` only after the timeout on `~ConnectionPool.connection()` is +expired. If this behaviour is not desirable (and you prefer your program to +crash hard and fast, if the surrounding conditions are not right, because +something else will respawn it) you should call the `~ConnectionPool.wait()` +method after creating the pool, or call `!open(wait=True)`: these methods will +block until the pool is full, or will raise a `PoolTimeout` exception if the +pool isn't ready within the allocated time. -Once a pool is instantiated, the constructor returns immediately, while the -background workers try to create the required number of connections to fill -the pool. If your application is misconfigured, or the network is down, it -means that the pool will be available but threads requesting a connection will -fail with a `PoolTimeout` after the `~ConnectionPool.connection()` timeout is -expired. If this behaviour is not desirable you should call the -`~ConnectionPool.wait()` method after creating the pool, which will block -until the pool is full or will throw a `PoolTimeout` if the pool isn't ready -within an allocated time. + +Connections life cycle +---------------------- The pool background workers create connections according to the parameters *conninfo*, *kwargs*, and *connection_class* passed to `ConnectionPool` -constructor. Once a connection is created it is also passed to the +constructor, invoking something like :samp:`{connection_class}({conninfo}, +**{kwargs})`. Once a connection is created it is also passed to the *configure()* callback, if provided, after which it is put in the pool (or passed to a client requesting it, if someone is already knocking at the door). + If a connection expires (it passes *max_lifetime*), or is returned to the pool -in broken state, or is found closed by `~ConnectionPool.check()`, then the +in broken state, or is found closed by `~ConnectionPool.check()`), then the pool will dispose of it and will start a new connection attempt in the background. -When the pool is no more to be used, you should call the -`~ConnectionPool.close()` method (unless the ``with`` syntax was used). If the -pool is a module-level object it may be unclear how to do so. Missing a call -to `!close()` shouldn't be a big problem, it should just result in a few -warnings printed. However, if you think that's sloppy, you can use the -`atexit` module to have the `!close()` method called at the end of the -program. - Using connections from the pool ------------------------------- -The pool can be used to request connections from multiple threads - it is -hardly useful otherwise! If more connections than the ones available in the -pool are requested, the requesting threads are queued and are served a -connection as soon as one is available again: either because another client -has finished using it or because the pool is allowed to grow and a new -connection is ready. +The pool can be used to request connections from multiple threads or +concurrent tasks - it is hardly useful otherwise! If more connections than the +ones available in the pool are requested, the requesting threads are queued +and are served a connection as soon as one is available, either because +another client has finished using it or because the pool is allowed to grow +(when *max_size* > *min_size*) and a new connection is ready. The main way to use the pool is to obtain a connection using the `~ConnectionPool.connection()` context, which returns a `~psycopg.Connection` @@ -88,12 +129,16 @@ or subclass:: with my_pool.connection() as conn: conn.execute("what you want") +The `!connection()` context behaves like the `~psycopg.Connection` object +context: at the end of the block, if there is a transaction open, it will be +committed, or rolled back if the context is exited with as exception. + At the end of the block the connection is returned to the pool and shouldn't be used anymore by the code which obtained it. If a *reset()* function is specified in the pool constructor, it is called on the connection before returning it to the pool. Note that the *reset()* function is called in a worker thread, so that the thread which used the connection can keep its -execution without being slowed down. +execution without being slowed down by it. Pool connection and sizing @@ -117,20 +162,20 @@ created, up to *max_size*. Note that the connections are always created by the background workers, not by the thread asking for the connection: if a client requests a new connection, and a previous client terminates its job before the new connection is ready, the waiting client will be served the existing -connection. This is especially useful in scenarios where the time to connect -is longer than the time the connection is used (see `this analysis`__, for -instance). +connection. This is especially useful in scenarios where the time to establish +a connection dominates the time for which the connection is used (see `this +analysis`__, for instance). .. __: https://github.com/brettwooldridge/HikariCP/blob/dev/documents/ Welcome-To-The-Jungle.md If a pool grows above *min_size*, but its usage decreases afterwards, a number -of connections are eventually closed: one each the *max_idle* time specified -in the pool constructor. +of connections are eventually closed: one every time a connection is unused +after the *max_idle* time specified in the pool constructor. -What's the right size for the pool -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +What's the right size for the pool? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Big question. Who knows. However, probably not as large as you imagine. Please take a look at `this analysis`__ for some ideas. @@ -138,9 +183,9 @@ take a look at `this analysis`__ for some ideas. .. __: https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing Something useful you can do is probably to use the -`~ConnectionPool.get_stats()` method and monitor the behaviour of your -program, eventually adjusting the size of the pool using the -`~ConnectionPool.resize()` method. +`~ConnectionPool.get_stats()` method and monitor the behaviour of your program +to tune the configuration parameters. The size of the pool can also be changed +at runtime using the `~ConnectionPool.resize()` method. Connection quality @@ -178,16 +223,16 @@ it should be polling each connection even faster than your program uses them. Your database server wouldn't be amused... Can you do something better than that? Of course you can, there is always a -better way than polling. You can use the same recipe of :ref:`disconnections`: -you can dedicate a thread (and a connection) to listen for activity on the -connection. If any activity is detected you can call the pool -`~ConnectionPool.check()` method, which will make every connection in the pool -briefly unavailable and run a quick check on them, returning them to the pool -if they are still working or creating a new connection if they aren't. +better way than polling. You can use the same recipe of :ref:`disconnections`, +reserving a connection and using a thread to monitor for any activity +happening on it. If any activity is detected, you can call the pool +`~ConnectionPool.check()` method, which will run a quick check on each +connection in the pool, removing the ones found in broken state, and using the +background workers to replace them with fresh ones. If you set up a similar check in your program, in case the database connection -is temporarily lost, we cannot do anything for the thread which already had -taken a connection from the pool, but no other thread should be served a +is temporarily lost, we cannot do anything for the threads which had taken +already a connection from the pool, but no other thread should be served a broken connection, because `!check()` would empty the pool and refill it with working connections, as soon as they are available. @@ -208,8 +253,8 @@ values can be sent to a monitoring system such as Graphite_ or Prometheus_. .. _Prometheus: https://prometheus.io/ The following values should be provided, but please don't consider them as a -rigid interface: it is possible that they might change. Keys whose value is 0 -may not be returned. +rigid interface: it is possible that they might change in the future. Keys +whose value is 0 may not be returned. ======================= ===================================================== diff --git a/docs/api/index.rst b/docs/api/index.rst index 0a787a722..91335b87a 100644 --- a/docs/api/index.rst +++ b/docs/api/index.rst @@ -17,10 +17,10 @@ This sections is a reference for all the public objects exposed by the sql rows errors + pool conninfo adapt types abc pq - pool dns diff --git a/docs/api/pool.rst b/docs/api/pool.rst index eb6aeb3ca..b08537a5c 100644 --- a/docs/api/pool.rst +++ b/docs/api/pool.rst @@ -46,7 +46,7 @@ The `!ConnectionPool` class :param min_size: The minimum number of connection the pool will hold. The pool will actively try to create new connections if some are lost (closed, broken) and will try to never go below - *min_size* + *min_size*. :type min_size: `!int`, default: 4 :param max_size: The maximum number of connections the pool will hold. If @@ -223,17 +223,15 @@ The `!AsyncConnectionPool` class -------------------------------- `!AsyncConnectionPool` has a very similar interface to the `ConnectionPool` -class but its blocking method are implemented as `async` coroutines. It -returns `~psycopg.AsyncConnection` instances, or its subclasses if specified -so in the *connection_class* parameter. +class but its blocking methods are implemented as ``async`` coroutines. It +returns instances of `~psycopg.AsyncConnection`, or of its subclass if +specified so in the *connection_class* parameter. -Only the function with different signature from `!ConnectionPool` are +Only the functions with different signature from `!ConnectionPool` are listed here. .. autoclass:: AsyncConnectionPool - All the other parameters are the same. - :param connection_class: The class of the connections to serve. It should be an `!AsyncConnection` subclass. :type connection_class: `!type`, default: `~psycopg.AsyncConnection` @@ -268,6 +266,8 @@ listed here. async with AsyncConnectionPool(...) as pool: # code using the pool + All the other constructor parameters are the same of `!ConnectionPool`. + .. automethod:: wait .. automethod:: resize .. automethod:: check diff --git a/docs/basic/transactions.rst b/docs/basic/transactions.rst index 51130986b..5473622ba 100644 --- a/docs/basic/transactions.rst +++ b/docs/basic/transactions.rst @@ -21,7 +21,7 @@ a `~rollback()` is called. If the cursor is closed with a transaction open, no COMMIT command is sent to the server, which will then discard the connection. Certain middleware (such -as pgbouncer) will also discard a connection left in transaction state, so, if +as PgBouncer) will also discard a connection left in transaction state, so, if possible you will want to commit or rollback a connection before finishing working with it. diff --git a/psycopg_pool/psycopg_pool/pool.py b/psycopg_pool/psycopg_pool/pool.py index 4e90a5f6d..aa1dd20b3 100644 --- a/psycopg_pool/psycopg_pool/pool.py +++ b/psycopg_pool/psycopg_pool/pool.py @@ -70,7 +70,8 @@ class ConnectionPool(BasePool[Connection[Any]]): """ Wait for the pool to be full (with `min_size` connections) after creation. - Raise `PoolTimeout` if not ready within *timeout* sec. + Close the pool, and raise `PoolTimeout`, if not ready within *timeout* + sec. Calling this method is not mandatory: you can try and use the pool immediately after its creation. The first client will be served as soon @@ -125,7 +126,7 @@ class ConnectionPool(BasePool[Connection[Any]]): self.putconn(conn) def getconn(self, timeout: Optional[float] = None) -> Connection[Any]: - """Obtain a contection from the pool. + """Obtain a connection from the pool. You should preferrably use `connection()`. Use this function only if it is not possible to use the connection as context manager.