From: Mike Bayer Date: Mon, 5 Nov 2018 19:52:35 +0000 (-0500) Subject: Add new "all columns" naming convention tokens X-Git-Tag: rel_1_3_0b1~15^2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=be705595846cd2205c72f9d87c025f8dc530cb73;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Add new "all columns" naming convention tokens Added new naming convention tokens ``column_0N_name``, ``column_0_N_name``, etc., which will render the names / keys / labels for all columns referenced by a particular constraint in a sequence. In order to accommodate for the length of such a naming convention, the SQL compiler's auto-truncation feature now applies itself to constraint names as well, which creates a shortened, deterministically generated name for the constraint that will apply to a target backend without going over the character limit of that backend. Additional notes: 1. the SQLite dialect had a format_index method that was apparently not used, removed. 2. the naming convention logic has been applying the foreign key remote column spec to the naming convention, and not the actual column name. In the case where the referenced Table object uses .key inside the columns and these are what ForeignKey() references, the naming convention was doing the wrong thing. The patch here fixes this, however this isn't noted in the migration notes. Fixes: #3989 Change-Id: Ib24f4754b886676096c480fc54b2e5c2463ac99a --- diff --git a/doc/build/changelog/migration_13.rst b/doc/build/changelog/migration_13.rst index eab13bef95..f86f8adb3f 100644 --- a/doc/build/changelog/migration_13.rst +++ b/doc/build/changelog/migration_13.rst @@ -576,6 +576,89 @@ Note that this change may be revised if it leads to problems. New Features and Improvements - Core ==================================== +.. _change_3989: + +New multi-column naming convention tokens, long name truncation +---------------------------------------------------------------- + +To suit the case where a :class:`.MetaData` naming convention needs to +disambiguate between multiple-column constraints and wishes to use all the +columns within the generated constraint name, a new series of +naming convention tokens are added, including +``column_0N_name``, ``column_0_N_name``, ``column_0N_key``, ``column_0_N_key``, +``referred_column_0N_name``, ``referred_column_0_N_name``, etc., which render +the column name (or key or label) for all columns in the constraint, +joined together either with no separator or with an underscore +separator. Below we define a convention that will name :class:`.UniqueConstraint` +constraints with a name that joins together the names of all columns:: + + metadata = MetaData(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_N_name)s" + }) + + table = Table( + 'info', metadata, + Column('a', Integer), + Column('b', Integer), + Column('c', Integer), + UniqueConstraint('a', 'b', 'c') + ) + +The CREATE TABLE for the above table will render as:: + + CREATE TABLE info ( + a INTEGER, + b INTEGER, + c INTEGER, + CONSTRAINT uq_info_a_b_c UNIQUE (a, b, c) + ) + +In addition, long-name truncation logic is now applied to the names generated +by naming conventions, in particular to accommodate for multi-column labels +that can produce very long names. This logic, which is the same as that used +for truncating long label names in a SELECT statement, replaces excess +characters that go over the identifier-length limit for the target database +with a deterministically generated 4-character hash. For example, on +PostgreSQL where identifiers cannot be longer than 63 characters, a long +constraint name would normally be generated from the table definition below:: + + long_names = Table( + 'long_names', metadata, + Column('information_channel_code', Integer, key='a'), + Column('billing_convention_name', Integer, key='b'), + Column('product_identifier', Integer, key='c'), + UniqueConstraint('a', 'b', 'c') + ) + +The truncation logic will ensure a too-long name isn't generated for the +UNIQUE constraint:: + + CREATE TABLE long_names ( + information_channel_code INTEGER, + billing_convention_name INTEGER, + product_identifier INTEGER, + CONSTRAINT uq_long_names_information_channel_code_billing_conventi_a79e + UNIQUE (information_channel_code, billing_convention_name, product_identifier) + ) + +The above suffix ``a79e`` is based on the md5 hash of the long name and will +generate the same value every time to produce consistent names for a given +schema. + +The change also repairs two other issues. One is that the ``column_0_key`` +token wasn't available even though this token was documented, the other was +that the ``referred_column_0_name`` token would inadvertently render the +``.key`` and not the ``.name`` of the column if these two values were +different. + +.. seealso:: + + :ref:`constraint_naming_conventions` + + :paramref:`.MetaData.naming_convention` + +:ticket:`3989` + .. _change_3831: Binary comparison interpretation for SQL functions diff --git a/doc/build/changelog/unreleased_13/3989.rst b/doc/build/changelog/unreleased_13/3989.rst new file mode 100644 index 0000000000..02e02c926a --- /dev/null +++ b/doc/build/changelog/unreleased_13/3989.rst @@ -0,0 +1,23 @@ +.. change:: + :tags: feature, sql + :tickets: 3989 + + Added new naming convention tokens ``column_0N_name``, ``column_0_N_name``, + etc., which will render the names / keys / labels for all columns referenced + by a particular constraint in a sequence. In order to accommodate for the + length of such a naming convention, the SQL compiler's auto-truncation + feature now applies itself to constraint names as well, which creates a + shortened, deterministically generated name for the constraint that will + apply to a target backend without going over the character limit of that + backend. + + The change also repairs two other issues. One is that the ``column_0_key`` + token wasn't available even though this token was documented, the other was + that the ``referred_column_0_name`` token would inadvertently render the + ``.key`` and not the ``.name`` of the column if these two values were + different. + + .. seealso:: + + :ref:`change_3989` + diff --git a/doc/build/core/constraints.rst b/doc/build/core/constraints.rst index f26d2dee09..02b274a068 100644 --- a/doc/build/core/constraints.rst +++ b/doc/build/core/constraints.rst @@ -506,14 +506,53 @@ object that is created using the :paramref:`.Column.index` parameter:: >>> DEFAULT_NAMING_CONVENTION immutabledict({'ix': 'ix_%(column_0_label)s'}) -The tokens available include ``%(table_name)s``, -``%(referred_table_name)s``, ``%(column_0_name)s``, ``%(column_0_label)s``, -``%(column_0_key)s``, ``%(referred_column_0_name)s``, and ``%(constraint_name)s``; -the documentation for :paramref:`.MetaData.naming_convention` describes each -individually. New tokens can also be added, by specifying an additional -token and a callable within the naming_convention dictionary. For example, -if we wanted to name our foreign key constraints using a GUID scheme, -we could do that as follows:: +The tokens available include ``%(table_name)s``, ``%(referred_table_name)s``, +``%(column_0_name)s``, ``%(column_0_label)s``, ``%(column_0_key)s``, +``%(referred_column_0_name)s``, and ``%(constraint_name)s``, as well as +multiple-column versions of each including ``%(column_0N_name)s``, +``%(column_0_N_name)s``, ``%(referred_column_0_N_name)s`` which render all +column names separated with or without an underscore. The documentation for +:paramref:`.MetaData.naming_convention` has further detail on each of these +conventions. + +When a generated name, particularly those that use the multiple-column tokens, +is too long for the identifier length limit of the target database +(for example, PostgreSQL has a limit of 63 characters), the name will be +deterministically truncated using a 4-character suffix based on the md5 +hash of the long name. For example, the naming convention below will +generate very long names given the column names in use:: + + metadata = MetaData(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_N_name)s" + }) + + long_names = Table( + 'long_names', metadata, + Column('information_channel_code', Integer, key='a'), + Column('billing_convention_name', Integer, key='b'), + Column('product_identifier', Integer, key='c'), + UniqueConstraint('a', 'b', 'c') + ) + +On the PostgreSQL dialect, names longer than 63 characters will be truncated +as in the following example:: + + CREATE TABLE long_names ( + information_channel_code INTEGER, + billing_convention_name INTEGER, + product_identifier INTEGER, + CONSTRAINT uq_long_names_information_channel_code_billing_conventi_a79e + UNIQUE (information_channel_code, billing_convention_name, product_identifier) + ) + +The above suffix ``a79e`` is based on the md5 hash of the long name and will +generate the same value every time to produce consistent names for a given +schema. + +New tokens can also be added, by specifying an additional token +and a callable within the naming_convention dictionary. For example, if we +wanted to name our foreign key constraints using a GUID scheme, we could do +that as follows:: import uuid @@ -562,7 +601,10 @@ name as follows:: `The Importance of Naming Constraints `_ - in the Alembic documentation. -.. versionadded:: 0.9.2 Added the :paramref:`.MetaData.naming_convention` argument. + +.. versionadded:: 1.3.0 added multi-column naming tokens such as ``%(column_0_N_name)s``. + Generated names that go beyond the character limit for the target database will be + deterministically truncated. .. _naming_check_constraints: diff --git a/lib/sqlalchemy/dialects/sqlite/base.py b/lib/sqlalchemy/dialects/sqlite/base.py index 345ad901ed..cef6756b20 100644 --- a/lib/sqlalchemy/dialects/sqlite/base.py +++ b/lib/sqlalchemy/dialects/sqlite/base.py @@ -1020,19 +1020,6 @@ class SQLiteIdentifierPreparer(compiler.IdentifierPreparer): 'when', 'where', ]) - def format_index(self, index, use_schema=True, name=None): - """Prepare a quoted index and schema name.""" - - if name is None: - name = index.name - result = self.quote(name, index.quote) - if (not self.omit_schema and - use_schema and - getattr(index.table, "schema", None)): - result = self.quote_schema( - index.table.schema, index.table.quote_schema) + "." + result - return result - class SQLiteExecutionContext(default.DefaultExecutionContext): @util.memoized_property diff --git a/lib/sqlalchemy/sql/compiler.py b/lib/sqlalchemy/sql/compiler.py index 27ee4afc68..459e0ba2c0 100644 --- a/lib/sqlalchemy/sql/compiler.py +++ b/lib/sqlalchemy/sql/compiler.py @@ -2621,17 +2621,7 @@ class DDLCompiler(Compiled): else: schema_name = None - ident = index.name - if isinstance(ident, elements._truncated_label): - max_ = self.dialect.max_index_name_length or \ - self.dialect.max_identifier_length - if len(ident) > max_: - ident = ident[0:max_ - 8] + \ - "_" + util.md5_hex(ident)[-4:] - else: - self.dialect.validate_identifier(ident) - - index_name = self.preparer.quote(ident) + index_name = self.preparer.format_index(index) if schema_name: index_name = schema_name + "." + index_name @@ -3162,11 +3152,31 @@ class IdentifierPreparer(object): if isinstance(constraint.name, elements._defer_name): name = naming._constraint_name_for_table( constraint, constraint.table) - if name: - return self.quote(name) - elif isinstance(constraint.name, elements._defer_none_name): - return None - return self.quote(constraint.name) + + if name is None: + if isinstance(constraint.name, elements._defer_none_name): + return None + else: + name = constraint.name + else: + name = constraint.name + + if isinstance(name, elements._truncated_label): + if constraint.__visit_name__ == 'index': + max_ = self.dialect.max_index_name_length or \ + self.dialect.max_identifier_length + else: + max_ = self.dialect.max_identifier_length + if len(name) > max_: + name = name[0:max_ - 8] + \ + "_" + util.md5_hex(name)[-4:] + else: + self.dialect.validate_identifier(name) + + return self.quote(name) + + def format_index(self, index): + return self.format_constraint(index) def format_table(self, table, use_schema=True, name=None): """Prepare a quoted table and schema name.""" diff --git a/lib/sqlalchemy/sql/naming.py b/lib/sqlalchemy/sql/naming.py index 533429333b..0107ce7240 100644 --- a/lib/sqlalchemy/sql/naming.py +++ b/lib/sqlalchemy/sql/naming.py @@ -48,6 +48,12 @@ class ConventionDict(object): self.const.name = None return self._const_name + def _key_column_X_key(self, idx): + # note this method was missing before + # [ticket:3989], meaning tokens like ``%(column_0_key)s`` weren't + # working even though documented. + return self._column_X(idx).key + def _key_column_X_name(self, idx): return self._column_X(idx).name @@ -65,12 +71,10 @@ class ConventionDict(object): def _key_referred_column_X_name(self, idx): fk = self.const.elements[idx] - refs = fk.target_fullname.split(".") - if len(refs) == 3: - refschema, reftable, refcol = refs - else: - reftable, refcol = refs - return refcol + # note that before [ticket:3989], this method was returning + # the specification for the :class:`.ForeignKey` itself, which normally + # would be using the ``.key`` of the column, not the name. + return fk.column.name def __getitem__(self, key): if key in self.convention: @@ -78,13 +82,30 @@ class ConventionDict(object): elif hasattr(self, '_key_%s' % key): return getattr(self, '_key_%s' % key)() else: - col_template = re.match(r".*_?column_(\d+)_.+", key) + col_template = re.match(r".*_?column_(\d+)(_?N)?_.+", key) if col_template: idx = col_template.group(1) - attr = "_key_" + key.replace(idx, "X") - idx = int(idx) - if hasattr(self, attr): - return getattr(self, attr)(idx) + multiples = col_template.group(2) + + if multiples: + if self._is_fk: + elems = self.const.elements + else: + elems = list(self.const.columns) + tokens = [] + for idx, elem in enumerate(elems): + attr = "_key_" + key.replace("0" + multiples, "X") + try: + tokens.append(getattr(self, attr)(idx)) + except AttributeError: + raise KeyError(key) + sep = "_" if multiples.startswith("_") else "" + return sep.join(tokens) + else: + attr = "_key_" + key.replace(idx, "X") + idx = int(idx) + if hasattr(self, attr): + return getattr(self, attr)(idx) raise KeyError(key) _prefix_dict = { diff --git a/lib/sqlalchemy/sql/schema.py b/lib/sqlalchemy/sql/schema.py index 88050b87e3..e37c703eb6 100644 --- a/lib/sqlalchemy/sql/schema.py +++ b/lib/sqlalchemy/sql/schema.py @@ -3654,15 +3654,29 @@ class MetaData(SchemaItem): * ``%(column_0_name)s`` - the name of the :class:`.Column` at index position "0" within the constraint. - * ``%(column_0_label)s`` - the label of the :class:`.Column` at - index position "0", e.g. :attr:`.Column.label` - - * ``%(column_0_key)s`` - the key of the :class:`.Column` at - index position "0", e.g. :attr:`.Column.key` - - * ``%(referred_column_0_name)s`` - the name of a :class:`.Column` - at index position "0" referenced by a - :class:`.ForeignKeyConstraint`. + * ``%(column_0N_name)s`` - the name of all :class:`.Column` + objects in order within the constraint, joined without a + separator. + + * ``%(column_0_N_name)s`` - the name of all :class:`.Column` + objects in order within the constraint, joined with an + underscore as a separator. + + * ``%(column_0_label)s``, ``%(column_0N_label)s``, + ``%(column_0_N_label)s`` - the label of either the zeroth + :class:`.Column` or all :class:`.Columns`, separated with + or without an underscore + + * ``%(column_0_key)s``, ``%(column_0N_key)s``, + ``%(column_0_N_key)s`` - the key of either the zeroth + :class:`.Column` or all :class:`.Columns`, separated with + or without an underscore + + * ``%(referred_column_0_name)s``, ``%(referred_column_0N_name)s`` + ``%(referred_column_0_N_name)s``, ``%(referred_column_0_key)s``, + ``%(referred_column_0N_key)s``, ... column tokens which + render the names/keys/labels of columns that are referenced + by a :class:`.ForeignKeyConstraint`. * ``%(constraint_name)s`` - a special key that refers to the existing name given to the constraint. When this key is @@ -3675,7 +3689,10 @@ class MetaData(SchemaItem): it along with a ``fn(constraint, table)`` callable to the naming_convention dictionary. - .. versionadded:: 0.9.2 + .. versionadded:: 1.3.0 - added new ``%(column_0N_name)s``, + ``%(column_0_N_name)s``, and related tokens that produce + concatenations of names, keys, or labels for all columns referred + to by a given constraint. .. seealso:: diff --git a/test/sql/test_metadata.py b/test/sql/test_metadata.py index 4976e2fb10..d0fa04d78a 100644 --- a/test/sql/test_metadata.py +++ b/test/sql/test_metadata.py @@ -4163,6 +4163,8 @@ class NamingConventionTest(fixtures.TestBase, AssertsCompiledSQL): Column('id', Integer, primary_key=True), Column('version', Integer, primary_key=True), Column('data', String(30)), + Column('Data2', String(30), key="data2"), + Column('Data3', String(30), key="data3"), schema=table_schema ) @@ -4175,6 +4177,191 @@ class NamingConventionTest(fixtures.TestBase, AssertsCompiledSQL): uq = UniqueConstraint(u1.c.data) eq_(uq.name, "uq_user_data") + def test_uq_conv_name(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_name)s" + }) + uq = UniqueConstraint(u1.c.data, name=naming.conv("myname")) + self.assert_compile( + schema.AddConstraint(uq), + 'ALTER TABLE "user" ADD CONSTRAINT myname UNIQUE (data)', + dialect="default" + ) + + def test_uq_defer_name_no_convention(self): + u1 = self._fixture(naming_convention={}) + uq = UniqueConstraint(u1.c.data, name=naming._defer_name("myname")) + self.assert_compile( + schema.AddConstraint(uq), + 'ALTER TABLE "user" ADD CONSTRAINT myname UNIQUE (data)', + dialect="default" + ) + + def test_uq_defer_name_convention(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_name)s" + }) + uq = UniqueConstraint(u1.c.data, name=naming._defer_name("myname")) + self.assert_compile( + schema.AddConstraint(uq), + 'ALTER TABLE "user" ADD CONSTRAINT uq_user_data UNIQUE (data)', + dialect="default" + ) + + def test_uq_key(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_key)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2) + eq_(uq.name, "uq_user_data") + + def test_uq_label(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_label)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2) + eq_(uq.name, "uq_user_user_data") + + def test_uq_allcols_underscore_name(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0_N_name)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2, u1.c.data3) + eq_(uq.name, "uq_user_data_Data2_Data3") + + def test_uq_allcols_merged_name(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0N_name)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2, u1.c.data3) + eq_(uq.name, "uq_user_dataData2Data3") + + def test_uq_allcols_merged_key(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0N_key)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2, u1.c.data3) + eq_(uq.name, "uq_user_datadata2data3") + + def test_uq_allcols_truncated_name(self): + u1 = self._fixture(naming_convention={ + "uq": "uq_%(table_name)s_%(column_0N_name)s" + }) + uq = UniqueConstraint(u1.c.data, u1.c.data2, u1.c.data3) + + dialect = default.DefaultDialect() + self.assert_compile( + schema.AddConstraint(uq), + 'ALTER TABLE "user" ADD ' + 'CONSTRAINT "uq_user_dataData2Data3" ' + 'UNIQUE (data, "Data2", "Data3")', + dialect=dialect + ) + + dialect.max_identifier_length = 15 + self.assert_compile( + schema.AddConstraint(uq), + 'ALTER TABLE "user" ADD ' + 'CONSTRAINT uq_user_2769 UNIQUE (data, "Data2", "Data3")', + dialect=dialect + ) + + def test_fk_allcols_underscore_name(self): + u1 = self._fixture(naming_convention={ + "fk": "fk_%(table_name)s_%(column_0_N_name)s_" + "%(referred_table_name)s_%(referred_column_0_N_name)s"}) + + m1 = u1.metadata + a1 = Table('address', m1, + Column('id', Integer, primary_key=True), + Column('UserData', String(30), key="user_data"), + Column('UserData2', String(30), key="user_data2"), + Column('UserData3', String(30), key="user_data3") + ) + fk = ForeignKeyConstraint(['user_data', 'user_data2', 'user_data3'], + ['user.data', 'user.data2', 'user.data3']) + a1.append_constraint(fk) + self.assert_compile( + schema.AddConstraint(fk), + 'ALTER TABLE address ADD CONSTRAINT ' + '"fk_address_UserData_UserData2_UserData3_user_data_Data2_Data3" ' + 'FOREIGN KEY("UserData", "UserData2", "UserData3") ' + 'REFERENCES "user" (data, "Data2", "Data3")', + dialect=default.DefaultDialect() + ) + + def test_fk_allcols_merged_name(self): + u1 = self._fixture(naming_convention={ + "fk": "fk_%(table_name)s_%(column_0N_name)s_" + "%(referred_table_name)s_%(referred_column_0N_name)s"}) + + m1 = u1.metadata + a1 = Table('address', m1, + Column('id', Integer, primary_key=True), + Column('UserData', String(30), key="user_data"), + Column('UserData2', String(30), key="user_data2"), + Column('UserData3', String(30), key="user_data3") + ) + fk = ForeignKeyConstraint(['user_data', 'user_data2', 'user_data3'], + ['user.data', 'user.data2', 'user.data3']) + a1.append_constraint(fk) + self.assert_compile( + schema.AddConstraint(fk), + 'ALTER TABLE address ADD CONSTRAINT ' + '"fk_address_UserDataUserData2UserData3_user_dataData2Data3" ' + 'FOREIGN KEY("UserData", "UserData2", "UserData3") ' + 'REFERENCES "user" (data, "Data2", "Data3")', + dialect=default.DefaultDialect() + ) + + def test_fk_allcols_truncated_name(self): + u1 = self._fixture(naming_convention={ + "fk": "fk_%(table_name)s_%(column_0N_name)s_" + "%(referred_table_name)s_%(referred_column_0N_name)s"}) + + m1 = u1.metadata + a1 = Table('address', m1, + Column('id', Integer, primary_key=True), + Column('UserData', String(30), key="user_data"), + Column('UserData2', String(30), key="user_data2"), + Column('UserData3', String(30), key="user_data3") + ) + fk = ForeignKeyConstraint(['user_data', 'user_data2', 'user_data3'], + ['user.data', 'user.data2', 'user.data3']) + a1.append_constraint(fk) + + dialect = default.DefaultDialect() + dialect.max_identifier_length = 15 + self.assert_compile( + schema.AddConstraint(fk), + 'ALTER TABLE address ADD CONSTRAINT ' + 'fk_addr_f9ff ' + 'FOREIGN KEY("UserData", "UserData2", "UserData3") ' + 'REFERENCES "user" (data, "Data2", "Data3")', + dialect=dialect + ) + + def test_ix_allcols_truncation(self): + u1 = self._fixture(naming_convention={ + "ix": "ix_%(table_name)s_%(column_0N_name)s" + }) + ix = Index(None, u1.c.data, u1.c.data2, u1.c.data3) + dialect = default.DefaultDialect() + dialect.max_identifier_length = 15 + self.assert_compile( + schema.CreateIndex(ix), + 'CREATE INDEX ix_user_2de9 ON ' + '"user" (data, "Data2", "Data3")', + dialect=dialect + ) + + def test_ix_name(self): + u1 = self._fixture(naming_convention={ + "ix": "ix_%(table_name)s_%(column_0_name)s" + }) + ix = Index(None, u1.c.data) + eq_(ix.name, "ix_user_data") + def test_ck_name_required(self): u1 = self._fixture(naming_convention={ "ck": "ck_%(table_name)s_%(constraint_name)s"