From: Mike Bayer Date: Fri, 22 Dec 2017 17:35:30 +0000 (-0500) Subject: Add new errors section X-Git-Tag: rel_1_2_0~7 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=1abc8e5500f23e0cfdf4d125d354065dad28aa7a;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Add new errors section Adding this to master so that it can be published, in advance of the exception-level site integration feature. Change-Id: Ia2a61df8e5198b81ef8f5fbac91143767b70c26b --- diff --git a/doc/build/contents.rst b/doc/build/contents.rst index a7277cf90a..13d0439396 100644 --- a/doc/build/contents.rst +++ b/doc/build/contents.rst @@ -20,5 +20,6 @@ documentation, see :ref:`index_toplevel`. Indices and tables ------------------ +* :ref:`errors` * :ref:`glossary` * :ref:`genindex` diff --git a/doc/build/core/exceptions.rst b/doc/build/core/exceptions.rst index 63bbc1e154..68762b853c 100644 --- a/doc/build/core/exceptions.rst +++ b/doc/build/core/exceptions.rst @@ -1,3 +1,5 @@ +.. _core_exceptions_toplevel: + Core Exceptions =============== diff --git a/doc/build/errors.rst b/doc/build/errors.rst new file mode 100644 index 0000000000..a81f509a83 --- /dev/null +++ b/doc/build/errors.rst @@ -0,0 +1,506 @@ +:orphan: + +.. _errors: + +============== +Error Messages +============== + +This section lists descriptions and background for common error messages +and warnings raised or emitted by SQLAlchemy. + +SQLAlchemy normally raises errors within the context of a SQLAlchemy-specific +exception class. For details on these classes, see +:ref:`core_exceptions_toplevel` and :ref:`orm_exceptions_toplevel`. + +SQLAlchemy errors can roughly be separated into two categories, the +**programming-time error** and the **runtime error**. Programming-time +errors are raised as a result of functions or methods being called with +incorrect arguments, or from other configuration-oriented methods such as +mapper configurations that can't be resolved. The programming-time error is +typically immediate and deterministic. The runtime error on the other hand +represents a failure that occurs as a program runs in response to some +condition that occurs arbitrarily, such as database connections being +exhausted or some data-related issue occurring. Runtime errors are more +likely to be seen in the logs of a running application as the program +encounters these states in response to load and data being encountered. + +Since runtime errors are not as easy to reproduce and often occur in response +to some arbitrary condition as the program runs, they are more difficult to +debug and also affect programs that have already been put into production. + +Within this section, the goal is to try to provide background on some of the +most common runtime errors as well as programming time errors. + + +Connections and Transactions +============================ + +.. _error_3o7r: + +QueuePool limit of size overflow reached, connection timed out, timeout +----------------------------------------------------------------------------------- + +This is possibly the most common runtime error experienced, as it directly +involves the work load of the application surpassing a configured limit, one +which typically applies to nearly all SQLAlchemy applications. + +The following points summarize what this error means, beginning with the +most fundamental points that most SQLAlchemy users should already be +familiar with. + +* **The SQLAlchemy Engine object uses a pool of connections by default** - What + this means is that when one makes use of a SQL database connection resource + of an :class:`.Engine` object, and then :term:`releases` that resource, + the database connection itself remains connected to the database and + is returned to an internal queue where it can be used again. Even though + the code may appear to be ending its conversation with the database, in many + cases the application will still maintain a fixed number of database connections + that persist until the application ends or the pool is explicitly disposed. + +* Because of the pool, when an application makes use of a SQL database + connection, most typically from either making use of :meth:`.Engine.connect` + or when making queries using an ORM :class:`.Session`, this activity + does not necessarily establish a new connection to the database at the + moment the connection object is acquired; it instead consults the + connection pool for a connection, which will often retrieve an existing + connection from the pool to be re-used. If no connections are available, + the pool will create a new database connection, but only if the + pool has not surpassed a configured capacity. + +* The default pool used in most cases is called :class:`.QueuePool`. When + you ask this pool to give you a connection and none are available, it + will create a new connection **if the total number of connections in play + are less than a configured value**. This value is equal to the + **pool size plus the max overflow**. That means if you have configured + your engine as:: + + engine = create_engine("mysql://u:p@host/db", pool_size=10, max_overflow=20) + + The above :class:`.Engine` will allow **at most 30 connections** to be in + play at any time, not including connections that were detached from the + engine or invalidated. If a request for a new connection arrives and + 30 connections are already in use by other parts of the application, + the connection pool will block for a fixed period of time, + before timing out and raising this error message. + + In order to allow for a higher number of connections be in use at once, + the pool can be adjusted using the + :paramref:`.create_engine.pool_size` and :paramref:`.create_engine.max_overflow` + parameters as passed to the :func:`.create_engine` function. The timeout + to wait for a connection to be available is configured using the + :paramref:`.create_engine.pool_timeout` parameter. + +* The pool can be configured to have unlimited overflow by setting + :paramref:`.create_engine.max_overflow` to the value "-1". With this setting, + the pool will still maintain a fixed pool of connections, however it will + never block upon a new connection being requested; it will instead unconditionally + make a new connection if none are available. + + However, when running in this way, if the application has an issue where it + is using up all available connectivity resources, it will eventually hit the + configured limit of available connections on the database itself, which will + again return an error. More seriously, when the application exhausts the + database of connections, it usually will have caused a great + amount of resources to be used up before failing, and can also interfere + with other applications and database status mechanisms that rely upon being + able to connect to the database. + + Given the above, the connection pool can be looked at as a **safety valve + for connection use**, providing a critical layer of protection against + a rogue application causing the entire database to become unavailable + to all other applications. When receiving this error message, it is vastly + preferable to repair the issue using up too many connections and/or + configure the limits appropriately, rather than allowing for unlimited + overflow which does not actually solve the underlying issue. + +What causes an application to use up all the connections that it has available? + +* **The application is fielding too many concurrent requests to do work based + on the configured value for the pool** - This is the most straightforward + cause. If you have + an application that runs in a thread pool that allows for 30 concurrent + threads, with one connection in use per thread, if your pool is not configured + to allow at least 30 connections checked out at once, you will get this + error once your application receives enough concurrent requests. Solution + is to raise the limits on the pool or lower the number of concurrent threads. + +* **The application is not returning connections to the pool** - This is the + next most common reason, which is that the application is making use of the + connection pool, but the program is failing to :term:`release` these + connections and is instead leaving them open. The connection pool as well + as the ORM :class:`.Session` do have logic such that when the session and/or + connection object is garbage collected, it results in the underlying + connection resources being released, however this behavior cannot be relied + upon to release resources in a timely manner. + + A common reason this can occur is that the application uses ORM sessions and + does not call :meth:`.Session.close` upon them one the work involving that + session is complete. Solution is to make sure ORM sessions if using the ORM, + or engine-bound :class:`.Connection` objects if using Core, are explicitly + closed at the end of the work being done, either via the appropriate + ``.close()`` method, or by using one of the available context managers (e.g. + "with:" statement) to properly release the resource. + +* **The application is attempting to run long-running transactions** - A + database transaction is a very expensive resource, and should **never be + left idle waiting for some event to occur**. If an application is waiting + for a user to push a button, or a result to come off of a long running job + queue, or is holding a persistent connection open to a browser, **don't + keep a database transaction open for the whole time**. As the application + needs to work with the database and interact with an event, open a short-lived + transaction at that point and then close it. + +* **The application is deadlocking** - Also a common cause of this error and + more difficult to grasp, if an application is not able to complete its use + of a connection either due to an application-side or database-side deadlock, + the application can use up all the available connections which then leads to + additional requests receiving this error. Reasons for deadlocks include: + + * Using an implicit async system such as gevent or eventlet without + properly monkeypatching all socket libraries and drivers, or which + has bugs in not fully covering for all monkeypatched driver methods, + or less commonly when the async system is being used against CPU-bound + workloads and greenlets making use of database resources are simply waiting + too long to attend to them. Neither implicit nor explicit async + programming frameworks are typically + necessary or appropriate for the vast majority of relational database + operations; if an application must use an async system for some area + of functionality, it's best that database-oriented business methods + run within traditional threads that pass messages to the async part + of the application. + + * A database side deadlock, e.g. rows are mutually deadlocked + + * Threading errors, such as mutexes in a mutual deadlock, or calling + upon an already locked mutex in the same thread + +Keep in mind an alternative to using pooling is to turn off pooling entirely. +See the section :ref:`pool_switching` for background on this. However, note +that when this error message is occurring, it is **always** due to a bigger +problem in the application itself; the pool just helps to reveal the problem +sooner. + +.. seealso:: + + :ref:`pooling_toplevel` + + :ref:`connections_toplevel` + + +.. _error_dbapi: + +DBAPI Errors +============ + +The Python database API, or DBAPI, is a specification for database drivers +which can be located at `Pep-249 `_. +This API specifies a set of exception classes that accommodate the full range +of failure modes of the database. + +SQLAlchemy does not generate these exceptions directly. Instead, they are +intercepted from the database driver and wrapped by the SQLAlchemy-provided +exception :class:`.DBAPIError`, however the messaging within the exception is +**generated by the driver, not SQLAlchemy**. + +.. _error_rvf5: + +InterfaceError +-------------- + +Exception raised for errors that are related to the database interface rather +than the database itself. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +The ``InterfaceError`` is sometimes raised by drivers in the context +of the database connection being dropped, or not being able to connect +to the database. For tips on how to deal with this, see the section +:ref:`pool_disconnects`. + +.. _error_4xp6: + +DatabaseError +-------------- + +Exception raised for errors that are related to the database itself, and not +the interface or data being passed. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +.. _error_9h9h: + +DataError +--------- + +Exception raised for errors that are due to problems with the processed data +like division by zero, numeric value out of range, etc. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +.. _error_e3q8: + +OperationalError +----------------- + +Exception raised for errors that are related to the database's operation and +not necessarily under the control of the programmer, e.g. an unexpected +disconnect occurs, the data source name is not found, a transaction could not +be processed, a memory allocation error occurred during processing, etc. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +The ``OperationalError`` is the most common (but not the only) error class used +by drivers in the context of the database connection being dropped, or not +being able to connect to the database. For tips on how to deal with this, see +the section :ref:`pool_disconnects`. + +.. _error_gkpj: + +IntegrityError +-------------- + +Exception raised when the relational integrity of the database is affected, +e.g. a foreign key check fails. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +.. _error_2j85: + +InternalError +------------- + +Exception raised when the database encounters an internal error, e.g. the +cursor is not valid anymore, the transaction is out of sync, etc. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +The ``InternalError`` is sometimes raised by drivers in the context +of the database connection being dropped, or not being able to connect +to the database. For tips on how to deal with this, see the section +:ref:`pool_disconnects`. + +.. _error_f405: + +ProgrammingError +---------------- + +Exception raised for programming errors, e.g. table not found or already +exists, syntax error in the SQL statement, wrong number of parameters +specified, etc. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +The ``ProgrammingError`` is sometimes raised by drivers in the context +of the database connection being dropped, or not being able to connect +to the database. For tips on how to deal with this, see the section +:ref:`pool_disconnects`. + +.. _error_tw8g: + +NotSupportedError +------------------ + +Exception raised in case a method or database API was used which is not +supported by the database, e.g. requesting a .rollback() on a connection that +does not support transaction or has transactions turned off. + +This error is a :ref:`DBAPI Error ` and originates from +the database driver (DBAPI), not SQLAlchemy itself. + +SQL Expression Language +======================= + +.. _error_2afi: + +This Compiled object is not bound to any Engine or Connection +------------------------------------------------------------- + +This error refers to the concept of "bound metadata", described at +:ref:`dbengine_implicit`. The issue occurs when one invokes the +:meth:`.Executable.execute` method directly off of a Core expression object +that is not associated with any :class:`.Engine`:: + + metadata = MetaData() + table = Table('t', metadata, Column('q', Integer)) + + stmt = select([table]) + result = stmt.execute() # <--- raises + +What the logic is expecting is that the :class:`.MetaData` object has +been **bound** to a :class:`.Engine`:: + + engine = create_engine("mysql+pymysql://user:pass@host/db") + metadata = MetaData(bind=engine) + +Where above, any statement that derives from a :class:`.Table` which +in turn derives from that :class:`.MetaData` will implicitly make use of +the given :class:`.Engine` in order to invoke the statement. + +Note that the concept of bound metadata is a **legacy pattern** and in most +cases is **highly discouraged**. The best way to invoke the statement is +to pass it to the :meth:`.Connection.execute` method of a :class:`.Connection`:: + + with engine.connect() as conn: + result = conn.execute(stmt) + +When using the ORM, a similar facility is available via the :class:`.Session`:: + + result = session.exxecute(stmt) + +.. seealso:: + + :ref:`dbengine_implicit` + + +.. _error_cd3x: + +A value is required for bind parameter (in parameter group ) +------------------------------------------------------------------- + +This error occurs when a statement makes use of :func:`.bindparam` either +implicitly or explicitly and does not provide a value when the statement +is executed:: + + stmt = select([table.c.column]).where(table.c.id == bindparam('my_param')) + + result = conn.execute(stmt) + +Above, no value has been provided for the parameter "my_param". The correct +approach is to provide a value:: + + result = conn.execute(stmt, my_param=12) + +When the message takes the form "a value is required for bind parameter +in parameter group ", the message is referring to the "executemany" stye +of execution. In this case, the statement is typically an INSERT, UPDATE, +or DELETE and a list of parameters is being passed. In this format, the +statement may be generated dynamically to include parameter positions for +every parameter given in the argument list, where it will use the +**first set of parameters** to determine what these should be. + +For example, the statement below is calculated based on the first parameter +set to require the parameters, "a", "b", and "c" - these names determine +the final string format of the statement which will be used for each +set of parameters in the list. As the second entry does not contain "b", +this error is generated:: + + m = MetaData() + t = Table( + 't', m, + Column('a', Integer), + Column('b', Integer), + Column('c', Integer) + ) + + e.execute( + t.insert(), [ + {"a": 1, "b": 2, "c": 3}, + {"a": 2, "c": 4}, + {"a": 3, "b": 4, "c": 5}, + ] + ) + + sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) + A value is required for bind parameter 'b', in parameter group 1 + [SQL: u'INSERT INTO t (a, b, c) VALUES (?, ?, ?)'] + [parameters: [{'a': 1, 'c': 3, 'b': 2}, {'a': 2, 'c': 4}, {'a': 3, 'c': 5, 'b': 4}]] + +Since "b" is required, pass it as ``None`` so that the INSERT may proceed:: + + e.execute( + t.insert(), [ + {"a": 1, "b": 2, "c": 3}, + {"a": 2, "b": None, "c": 4}, + {"a": 3, "b": 4, "c": 5}, + ] + ) + +.. seealso:: + + :ref:`coretutorial_bind_param` + + :ref:`execute_multiple` + +Object Relational Mapping +========================= + +.. _error_bhk3: + +Parent instance is not bound to a Session; (lazy load/deferred load/refresh/etc.) operation cannot proceed +-------------------------------------------------------------------------------------------------------------- + +This is likely the most common error message when dealing with the ORM, and it +occurs as a result of the nature of a technique the ORM makes wide use of known +as :term:`lazy loading`. Lazy loading is a common object-relational pattern +whereby an object that's persisted by the ORM maintains a proxy to the database +itself, such that when various attributes upon the object are accessed, their +value may be retrieved from the database *lazily*. The advantage to this +approach is that objects can be retrieved from the database without having +to load all of their attributes or related data at once, and instead only that +data which is requested can be delivered at that time. The major disadvantage +is basically a mirror image of the advantage, which is that if lots of objects +are being loaded which are known to require a certain set of data in all cases, +it is wasteful to load that additional data piecemeal. + +Another caveat of lazy loading beyond the usual efficiency concerns is that +in order for lazy loading to proceed, the object has to **remain associated +with a Session** in order to be able to retrieve its state. This error message +means that an object has become de-associated with its :class:`.Session` and +is being asked to lazy load data from the database. + +The most common reason that objects become detached from their :class:`.Session` +is that the session itself was closed, typically via the :meth:`.Session.close` +method. The objects will then live on to be accessed further, very often +within web applications where they are delivered to a server-side templating +engine and are asked for further attributes which they cannot load. + +Mitigation of this error is via two general techniques: + +* **Don't close the session prematurely** - Often, applications will close + out a transaction before passing off related objects to some other system + which then fails due to this error. Sometimes the transaction doesn't need + to be closed so soon; an example is the web application closes out + the transaction before the view is rendered. This is often done in the name + of "correctness", but may be seen as a mis-application of "encapsulation", + as this term refers to code organization, not actual actions. The template that + uses an ORM object is making use of the `proxy pattern `_ + which keeps database logic encapsulated from the caller. If the + :class:`.Session` can be held open until the lifespan of the objects are done, + this is the best approach. + +* **Load everything that's needed up front** - It is very often impossible to + keep the transaction open, especially in more complex applications that need + to pass objects off to other systems that can't run in the same context + even though they're in the same process. In this case, the application + should try to make appropriate use of :term:`eager loading` to ensure + that objects have what they need up front. As an additional measure, + special directives like the :func:`.raiseload` option can ensure that + systems don't call upon lazy loading when its not expected. + + .. seealso:: + + :ref:`loading_toplevel` - detailed documentation on eager loading and other + relationship-oriented loading techniques + + +Core Exception Classes +====================== + +See :ref:`core_exceptions_toplevel` for Core exception classes. + + +ORM Exception Classes +====================== + +See :ref:`orm_exceptions_toplevel` for ORM exception classes. + + + diff --git a/doc/build/index.rst b/doc/build/index.rst index 1186e8eacf..125ac5f3d4 100644 --- a/doc/build/index.rst +++ b/doc/build/index.rst @@ -16,6 +16,7 @@ A high level view and getting set up. :doc:`Frequently Asked Questions ` | :doc:`Migration from 1.1 ` | :doc:`Glossary ` | +:doc:`Error Messages ` | :doc:`Changelog catalog ` SQLAlchemy ORM diff --git a/doc/build/orm/exceptions.rst b/doc/build/orm/exceptions.rst index 047c743e0d..f2e711424a 100644 --- a/doc/build/orm/exceptions.rst +++ b/doc/build/orm/exceptions.rst @@ -1,3 +1,5 @@ +.. _orm_exceptions_toplevel: + ORM Exceptions ==============