From c985e0d8096e750df35aa07dde71f432a57656b6 Mon Sep 17 00:00:00 2001 From: =?utf8?q?Vlastimil=20Z=C3=ADma?= Date: Thu, 26 May 2022 10:53:01 +0200 Subject: [PATCH] Add docs for data migrations --- docs/build/cookbook.rst | 39 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 38 insertions(+), 1 deletion(-) diff --git a/docs/build/cookbook.rst b/docs/build/cookbook.rst index 23202ea9..ee4bd5b7 100644 --- a/docs/build/cookbook.rst +++ b/docs/build/cookbook.rst @@ -1573,4 +1573,41 @@ the same ``env.py`` file can be invoked using asyncio as:: await conn.run_sync(run_upgrade, config.Config("alembic.ini")) - asyncio.run(run_async_upgrade()) \ No newline at end of file + asyncio.run(run_async_upgrade()) + + +Data migrations +=============== + +Alembic migrations are designed for schema migrations. +The nature of data migrations are inherently different and it's not in fact advisable in the general case to write data migrations that integrate with Alembic's schema versioning model. +For example downgrades are difficult to address since they might require deletion of data, which may even not be possible to detect. + +.. warning:: + + The solution needs to be designed specifically for each individual application and migration. + There are no general rules and the following text is only a recommendation based on experience. + +There are three basic approaches for the data migrations. + +Small data +---------- +Small data migrations are easy to perform, especially in cases of initial data to a new table. +These can be handled using :meth:`.Operations.bulk_insert`. + +Separate migration script +------------------------- +One possibility is a completely separate script aside of alembic migrations. +The complete migration is then processed in following steps: + +1. Run the initial alembic migrations (new columns etc.) +2. Run the separate data migration script +3. Run the final alembic migrations (database constraints, delete columns etc.) + +The data migration script may also need a separate ORM model to handle intermediate state of the database. + +Online migration +---------------- +The application maintains a version of schema with both versions. +Writes are performed on both places, while the background script move all the remaining data across. +This technique is very challenging and time demanding. -- 2.47.3