Pool life cycle
---------------
-A typical way to use the pool is to create a single instance of it, as a
+A simple way to use the pool is to create a single instance of it, as a
global object, and to use this object in the rest of the program, allowing
-other functions, modules, threads to use it. This is only a common use
-however, and not the necessary one; in particular the connection pool acts as
-a context manager and can be closed automatically at the end of its ``with``
-block::
+other functions, modules, threads to use it::
+ # module db.py in your program
from psycopg_pool import ConnectionPool
- with ConnectionPool(conninfo, **kwargs) as my_pool:
- run_app(my_pool)
+ pool = ConnectionPool(conninfo, **kwargs)
+ # the pool starts connecting immediately.
+
+ # in another module
+ from .db import pool
+
+ def my_function():
+ with poolconnection() as conn:
+ conn.execute(...)
+
+Ideally you may want to call `~ConnectionPool.close()` when the use of the
+pool is finished. Failing to call `!close()` at the end of the program is not
+terribly bad: probably it will just result in some warnings printed on stderr.
+However, if you think that it's sloppy, you could use the `atexit` module to
+have `!close()` called at the end of the program.
+
+If you want to avoid starting to connect to the database at import time, and
+want to wait for the application to be ready, you can create the pool using
+*open* = `!False`, and call the `~ConnectionPool.open()` and
+`~ConnectionPool.close()` methods when the conditions are right. Certain
+frameworks provide callbacks triggered when the program is started and stopped
+(for instance `FastAPI startup/shutdown events`__): they are perfect to
+initiate and terminate the pool operations::
+
+ pool = ConnectionPool(conninfo, open=False, **kwargs)
+
+ @app.on_event("startup")
+ def open_pool():
+ pool.open()
+
+ @app.on_event("shutdown")
+ def close_pool():
+ pool.close()
+
+.. __: https://fastapi.tiangolo.com/advanced/events/#events-startup-shutdown
+
+Creating a single pool as a global variable is not the mandatory use: your
+program can create more than one pool, which might be useful to connect to
+more than one database, or to provide different types of connections, for
+instance to provide separate read/write and read-only connections. The pool
+also acts as a context manager and is open and closed, if necessary, on
+entering and exiting the context block::
+
+ from psycopg_pool import ConnectionPool
+
+ with ConnectionPool(conninfo, **kwargs) as pool:
+ run_app(pool)
# the pool is now closed
-If necessary, or convenient, your application may create more than one pool,
-for instance to connect to more than one database or to provide separate
-read-only and read/write connections.
+When the pool is open, the pool's background workers start creating the
+requested *min_size* connections, while the constructor (or the `!open()`
+method) returns immediately. This allows the program some leeway to start
+before the target database is up and running. However, if your application is
+misconfigured, or the network is down, it means that the program will be able
+to start, but the threads requesting a connection will fail with a
+`PoolTimeout` only after the timeout on `~ConnectionPool.connection()` is
+expired. If this behaviour is not desirable (and you prefer your program to
+crash hard and fast, if the surrounding conditions are not right, because
+something else will respawn it) you should call the `~ConnectionPool.wait()`
+method after creating the pool, or call `!open(wait=True)`: these methods will
+block until the pool is full, or will raise a `PoolTimeout` exception if the
+pool isn't ready within the allocated time.
-Once a pool is instantiated, the constructor returns immediately, while the
-background workers try to create the required number of connections to fill
-the pool. If your application is misconfigured, or the network is down, it
-means that the pool will be available but threads requesting a connection will
-fail with a `PoolTimeout` after the `~ConnectionPool.connection()` timeout is
-expired. If this behaviour is not desirable you should call the
-`~ConnectionPool.wait()` method after creating the pool, which will block
-until the pool is full or will throw a `PoolTimeout` if the pool isn't ready
-within an allocated time.
+
+Connections life cycle
+----------------------
The pool background workers create connections according to the parameters
*conninfo*, *kwargs*, and *connection_class* passed to `ConnectionPool`
-constructor. Once a connection is created it is also passed to the
+constructor, invoking something like :samp:`{connection_class}({conninfo},
+**{kwargs})`. Once a connection is created it is also passed to the
*configure()* callback, if provided, after which it is put in the pool (or
passed to a client requesting it, if someone is already knocking at the door).
+
If a connection expires (it passes *max_lifetime*), or is returned to the pool
-in broken state, or is found closed by `~ConnectionPool.check()`, then the
+in broken state, or is found closed by `~ConnectionPool.check()`), then the
pool will dispose of it and will start a new connection attempt in the
background.
-When the pool is no more to be used, you should call the
-`~ConnectionPool.close()` method (unless the ``with`` syntax was used). If the
-pool is a module-level object it may be unclear how to do so. Missing a call
-to `!close()` shouldn't be a big problem, it should just result in a few
-warnings printed. However, if you think that's sloppy, you can use the
-`atexit` module to have the `!close()` method called at the end of the
-program.
-
Using connections from the pool
-------------------------------
-The pool can be used to request connections from multiple threads - it is
-hardly useful otherwise! If more connections than the ones available in the
-pool are requested, the requesting threads are queued and are served a
-connection as soon as one is available again: either because another client
-has finished using it or because the pool is allowed to grow and a new
-connection is ready.
+The pool can be used to request connections from multiple threads or
+concurrent tasks - it is hardly useful otherwise! If more connections than the
+ones available in the pool are requested, the requesting threads are queued
+and are served a connection as soon as one is available, either because
+another client has finished using it or because the pool is allowed to grow
+(when *max_size* > *min_size*) and a new connection is ready.
The main way to use the pool is to obtain a connection using the
`~ConnectionPool.connection()` context, which returns a `~psycopg.Connection`
with my_pool.connection() as conn:
conn.execute("what you want")
+The `!connection()` context behaves like the `~psycopg.Connection` object
+context: at the end of the block, if there is a transaction open, it will be
+committed, or rolled back if the context is exited with as exception.
+
At the end of the block the connection is returned to the pool and shouldn't
be used anymore by the code which obtained it. If a *reset()* function is
specified in the pool constructor, it is called on the connection before
returning it to the pool. Note that the *reset()* function is called in a
worker thread, so that the thread which used the connection can keep its
-execution without being slowed down.
+execution without being slowed down by it.
Pool connection and sizing
background workers, not by the thread asking for the connection: if a client
requests a new connection, and a previous client terminates its job before the
new connection is ready, the waiting client will be served the existing
-connection. This is especially useful in scenarios where the time to connect
-is longer than the time the connection is used (see `this analysis`__, for
-instance).
+connection. This is especially useful in scenarios where the time to establish
+a connection dominates the time for which the connection is used (see `this
+analysis`__, for instance).
.. __: https://github.com/brettwooldridge/HikariCP/blob/dev/documents/
Welcome-To-The-Jungle.md
If a pool grows above *min_size*, but its usage decreases afterwards, a number
-of connections are eventually closed: one each the *max_idle* time specified
-in the pool constructor.
+of connections are eventually closed: one every time a connection is unused
+after the *max_idle* time specified in the pool constructor.
-What's the right size for the pool
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+What's the right size for the pool?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Big question. Who knows. However, probably not as large as you imagine. Please
take a look at `this analysis`__ for some ideas.
.. __: https://github.com/brettwooldridge/HikariCP/wiki/About-Pool-Sizing
Something useful you can do is probably to use the
-`~ConnectionPool.get_stats()` method and monitor the behaviour of your
-program, eventually adjusting the size of the pool using the
-`~ConnectionPool.resize()` method.
+`~ConnectionPool.get_stats()` method and monitor the behaviour of your program
+to tune the configuration parameters. The size of the pool can also be changed
+at runtime using the `~ConnectionPool.resize()` method.
Connection quality
Your database server wouldn't be amused...
Can you do something better than that? Of course you can, there is always a
-better way than polling. You can use the same recipe of :ref:`disconnections`:
-you can dedicate a thread (and a connection) to listen for activity on the
-connection. If any activity is detected you can call the pool
-`~ConnectionPool.check()` method, which will make every connection in the pool
-briefly unavailable and run a quick check on them, returning them to the pool
-if they are still working or creating a new connection if they aren't.
+better way than polling. You can use the same recipe of :ref:`disconnections`,
+reserving a connection and using a thread to monitor for any activity
+happening on it. If any activity is detected, you can call the pool
+`~ConnectionPool.check()` method, which will run a quick check on each
+connection in the pool, removing the ones found in broken state, and using the
+background workers to replace them with fresh ones.
If you set up a similar check in your program, in case the database connection
-is temporarily lost, we cannot do anything for the thread which already had
-taken a connection from the pool, but no other thread should be served a
+is temporarily lost, we cannot do anything for the threads which had taken
+already a connection from the pool, but no other thread should be served a
broken connection, because `!check()` would empty the pool and refill it with
working connections, as soon as they are available.
.. _Prometheus: https://prometheus.io/
The following values should be provided, but please don't consider them as a
-rigid interface: it is possible that they might change. Keys whose value is 0
-may not be returned.
+rigid interface: it is possible that they might change in the future. Keys
+whose value is 0 may not be returned.
======================= =====================================================
projects / thirdparty / psycopg.git / commitdiff
