From: Mike Bayer Date: Sat, 30 Jun 2007 21:45:13 +0000 (+0000) Subject: - implemented ORM-level composite column types [ticket:211]. X-Git-Tag: rel_0_4_6~147 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=d657993d4972afdfb9ebe4128f00baea428866bd;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git - implemented ORM-level composite column types [ticket:211]. constructed via composite(cls, *columns), allows multiple columns to be expressed as a single object attribute. can be used for primary key columns also. not yet supported for deferred column loading (but this is straightforward). - formatting to CHANGES - some test suite fixes --- diff --git a/CHANGES b/CHANGES index 80311f7466..a4a02eb323 100644 --- a/CHANGES +++ b/CHANGES @@ -1,63 +1,77 @@ 0.4.0 - orm + - new collection_class api and implementation [ticket:213] collections are now instrumented via decorations rather than proxying. you can now have collections that manage their own membership, and your class instance will be directly exposed on the relation property. the changes are transparent for most users. - - InstrumentedList (as it was) is removed, and relation properties no - longer have 'clear()', '.data', or any other added methods beyond those - provided by the collection type. you are free, of course, to add them - to a custom class. - - __setitem__-like assignments now fire remove events for the existing - value, if any. - - dict-likes used as collection classes no longer need to change __iter__ - semantics- itervalues() is used by default instead. this is a backwards - incompatible change. - - subclassing dict for a mapped collection is no longer needed in most cases. - orm.collections provides canned implementations that key objects by a - specified column or a custom function of your choice. - - collection assignment now requires a compatible type- assigning None - to clear a collection or assigning a list to a dict collection will now - raise an argument error. - - AttributeExtension moved to interfaces, and .delete is now .remove - The event method signature has also been swapped around. + - InstrumentedList (as it was) is removed, and relation properties + no longer have 'clear()', '.data', or any other added methods + beyond those provided by the collection type. you are free, of + course, to add them to a custom class. + - __setitem__-like assignments now fire remove events for the + existing value, if any. + - dict-likes used as collection classes no longer need to change + __iter__ semantics- itervalues() is used by default instead. this + is a backwards incompatible change. + - subclassing dict for a mapped collection is no longer needed in + most cases. orm.collections provides canned implementations that + key objects by a specified column or a custom function of your + choice. + - collection assignment now requires a compatible type- assigning + None to clear a collection or assigning a list to a dict + collection will now raise an argument error. + - AttributeExtension moved to interfaces, and .delete is now + .remove The event method signature has also been swapped around. + - major interface pare-down for Query: all selectXXX methods are deprecated. generative methods are now the standard way to do things, i.e. filter(), filter_by(), all(), one(), etc. Deprecated methods are docstring'ed with their new replacements. - - removed ancient query.select_by_attributename() capability. - - added "aliased joins" positional argument to the front of - filter_by(). this allows auto-creation of joins that are aliased - locally to the individual filter_by() call. This allows the - auto-construction of joins which cross the same paths but - are querying divergent criteria. ClauseElements at the front - of filter_by() are removed (use filter()). + - removed ancient query.select_by_attributename() capability. + - added "aliased joins" positional argument to the front of + filter_by(). this allows auto-creation of joins that are aliased + locally to the individual filter_by() call. This allows the + auto-construction of joins which cross the same paths but are + querying divergent criteria. ClauseElements at the front of + filter_by() are removed (use filter()). + + - added composite column properties. using the composite(cls, *columns) + function inside of the "properties" dict, instances of cls will be + created/mapped to a single attribute, comprised of the values + correponding to *columns [ticket:211] + - improved support for custom column_property() attributes which feature correlated subqueries...work better with eager loading now. + - along with recent speedups to ResultProxy, total number of function calls significantly reduced for large loads. test/perf/masseagerload.py reports 0.4 as having the fewest number of function calls across all SA versions (0.1, 0.2, and 0.3) + - primary key "collapse" behavior; the mapper will analyze all columns - in its given selectable for primary key "equivalence", that is, columns - which are equivalent via foreign key relationship or via an explicit - inherit_condition. primarily for joined-table inheritance scenarios - where different named PK columns in inheriting tables should "collapse" - into a single-valued (or fewer-valued) primary key. fixes things - like [ticket:611]. + in its given selectable for primary key "equivalence", that is, + columns which are equivalent via foreign key relationship or via an + explicit inherit_condition. primarily for joined-table inheritance + scenarios where different named PK columns in inheriting tables + should "collapse" into a single-valued (or fewer-valued) primary key. + fixes things like [ticket:611]. + - secondary inheritance loading: polymorphic mappers can be constructed *without* a select_table argument. inheriting mappers whose tables were not represented in the initial load will issue a second SQL query immediately, once per instance (i.e. not very efficient for large lists), in order to load the remaining columns. - - secondary inheritance loading can also move its second query into - a column- level "deferred" load, via the "polymorphic_fetch" - argument, which can be set to 'select' or 'deferred' + - secondary inheritance loading can also move its second query into + a column- level "deferred" load, via the "polymorphic_fetch" + argument, which can be set to 'select' or 'deferred' + - added undefer_group() MapperOption, sets a set of "deferred" columns joined by a "group" to load as "undeferred". + - sql - significant architectural overhaul to SQL elements (ClauseElement). all elements share a common "mutability" framework which allows a diff --git a/lib/sqlalchemy/orm/__init__.py b/lib/sqlalchemy/orm/__init__.py index e61f5d8cfe..eab93fed55 100644 --- a/lib/sqlalchemy/orm/__init__.py +++ b/lib/sqlalchemy/orm/__init__.py @@ -13,14 +13,14 @@ from sqlalchemy import exceptions from sqlalchemy import util as sautil from sqlalchemy.orm.mapper import Mapper, object_mapper, class_mapper, mapper_registry from sqlalchemy.orm.interfaces import SynonymProperty, MapperExtension, EXT_PASS, ExtensionOption -from sqlalchemy.orm.properties import PropertyLoader, ColumnProperty, BackRef +from sqlalchemy.orm.properties import PropertyLoader, ColumnProperty, CompositeProperty, BackRef from sqlalchemy.orm import mapper as mapperlib from sqlalchemy.orm.query import Query from sqlalchemy.orm.util import polymorphic_union from sqlalchemy.orm.session import Session as create_session from sqlalchemy.orm.session import object_session, attribute_manager -__all__ = ['relation', 'column_property', 'backref', 'eagerload', 'lazyload', 'noload', 'deferred', 'defer', 'undefer', 'undefer_group', 'extension', +__all__ = ['relation', 'column_property', 'composite', 'backref', 'eagerload', 'lazyload', 'noload', 'deferred', 'defer', 'undefer', 'undefer_group', 'extension', 'mapper', 'clear_mappers', 'compile_mappers', 'class_mapper', 'object_mapper', 'MapperExtension', 'Query', 'polymorphic_union', 'create_session', 'synonym', 'contains_alias', 'contains_eager', 'EXT_PASS', 'object_session' ] @@ -37,30 +37,30 @@ def relation(*args, **kwargs): def column_property(*args, **kwargs): """Provide a column-level property for use with a Mapper. + + Column-based properties can normally be applied to the mapper's + ``properties`` dictionary using the ``schema.Column`` element directly. + Use this function when the given column is not directly present within + the mapper's selectable; examples include SQL expressions, functions, + and scalar SELECT queries. + + Columns that arent present in the mapper's selectable won't be persisted + by the mapper and are effectively "read-only" attributes. + """ - Normally, custom column-level properties that represent columns - directly or indirectly present within the mapped selectable - can just be added to the ``properties`` dictionary directly, - in which case this function's usage is not necessary. - - In the case of a ``ColumnElement`` directly present within the - ``properties`` dictionary, the given column is converted to be the exact column - located within the mapped selectable, in the case that the mapped selectable - is not the exact parent selectable of the given column, but shares a common - base table relationship with that column. - - Use this function when the column expression being added does not - correspond to any single column within the mapped selectable, - such as a labeled function or scalar-returning subquery, to force the element - to become a mapped property regardless of it not being present within the - mapped selectable. + return ColumnProperty(*args, **kwargs) + +def composite(class_, *cols, **kwargs): + """Return a composite column-based property for use with a Mapper. - Note that persistence of instances is driven from the collection of columns - within the mapped selectable, so column properties attached to a Mapper which have - no direct correspondence to the mapped selectable will effectively be non-persisted - attributes. + This is very much like a column-based property except the given class + is used to construct values composed of one or more columns. The class must + implement a constructor with positional arguments matching the order of + columns given, as well as a __colset__() method which returns its attributes + in column order. """ - return ColumnProperty(*args, **kwargs) + + return CompositeProperty(class_, *cols, **kwargs) def _relation_loader(mapper, secondary=None, primaryjoin=None, secondaryjoin=None, lazy=True, **kwargs): return PropertyLoader(mapper, secondary, primaryjoin, secondaryjoin, lazy=lazy, **kwargs) diff --git a/lib/sqlalchemy/orm/mapper.py b/lib/sqlalchemy/orm/mapper.py index 8c4a9c8a31..838d6fd58a 100644 --- a/lib/sqlalchemy/orm/mapper.py +++ b/lib/sqlalchemy/orm/mapper.py @@ -678,7 +678,7 @@ class Mapper(object): self.__log("adding ColumnProperty %s" % (column_key)) elif isinstance(prop, ColumnProperty): if prop.parent is not self: - prop = ColumnProperty(deferred=prop.deferred, group=prop.group, *prop.columns) + prop = prop.copy() prop.set_parent(self) self.__props[column_key] = prop prop.columns.append(column) @@ -1050,12 +1050,12 @@ class Mapper(object): if prop is None: return NO_ATTRIBUTE #print "get column attribute '%s' from instance %s" % (column.key, mapperutil.instance_str(obj)) - return prop.getattr(obj) + return prop.getattr(obj, column) def set_attr_by_column(self, obj, column, value): """Set the value of an instance attribute using a Column as the key.""" - self.columntoproperty[column][0].setattr(obj, value) + self.columntoproperty[column][0].setattr(obj, value, column) def save_obj(self, objects, uowtransaction, postupdate=False, post_update_cols=None, single=False): """Issue ``INSERT`` and/or ``UPDATE`` statements for a list of objects. @@ -1181,7 +1181,7 @@ class Mapper(object): if history: a = history.added_items() if len(a): - params[col.key] = a[0] + params[col.key] = prop.get_col_value(col, a[0]) hasdata = True else: # doing an INSERT, non primary key col ? diff --git a/lib/sqlalchemy/orm/properties.py b/lib/sqlalchemy/orm/properties.py index c183556766..54898c4552 100644 --- a/lib/sqlalchemy/orm/properties.py +++ b/lib/sqlalchemy/orm/properties.py @@ -18,7 +18,7 @@ from sqlalchemy.orm import util as mapperutil import sets, random from sqlalchemy.orm.interfaces import * -__all__ = ['ColumnProperty', 'PropertyLoader', 'BackRef'] +__all__ = ['ColumnProperty', 'CompositeProperty', 'PropertyLoader', 'BackRef'] class ColumnProperty(StrategizedProperty): """Describes an object attribute that corresponds to a table column.""" @@ -33,17 +33,20 @@ class ColumnProperty(StrategizedProperty): self.columns = list(columns) self.group = kwargs.pop('group', None) self.deferred = kwargs.pop('deferred', False) - + def create_strategy(self): if self.deferred: return strategies.DeferredColumnLoader(self) else: return strategies.ColumnLoader(self) - - def getattr(self, object): + + def copy(self): + return ColumnProperty(deferred=self.deferred, group=self.group, *self.columns) + + def getattr(self, object, column): return getattr(object, self.key) - def setattr(self, object, value): + def setattr(self, object, value, column): setattr(object, self.key, value) def get_history(self, obj, passive=False): @@ -55,9 +58,43 @@ class ColumnProperty(StrategizedProperty): def compare(self, value): return self.columns[0] == value + def get_col_value(self, column, value): + return value + ColumnProperty.logger = logging.class_logger(ColumnProperty) mapper.ColumnProperty = ColumnProperty + +class CompositeProperty(ColumnProperty): + """subclasses ColumnProperty to provide composite type support.""" + + def __init__(self, class_, *columns, **kwargs): + super(CompositeProperty, self).__init__(*columns, **kwargs) + self.composite_class = class_ + + def copy(self): + return CompositeProperty(deferred=self.deferred, group=self.group, composite_class=self.composite_class, *self.columns) + + def getattr(self, object, column): + obj = getattr(object, self.key) + return self.get_col_value(column, obj) + + def setattr(self, object, value, column): + obj = getattr(object, self.key, None) + if obj is None: + obj = self.composite_class(*[None for c in self.columns]) + for a, b in zip(self.columns, value.__colset__()): + if a is column: + setattr(obj, b, value) + + def compare(self, value): + return sql.and_([a==b for a, b in zip(self.columns, value.__colset__())]) + + def get_col_value(self, column, value): + for a, b in zip(self.columns, value.__colset__()): + if a is column: + return b + class PropertyLoader(StrategizedProperty): """Describes an object property that holds a single item or list diff --git a/lib/sqlalchemy/orm/query.py b/lib/sqlalchemy/orm/query.py index 5a2e04c7b5..417bbd04e1 100644 --- a/lib/sqlalchemy/orm/query.py +++ b/lib/sqlalchemy/orm/query.py @@ -67,6 +67,13 @@ class Query(object): ret = self._extension.get(self, ident, **kwargs) if ret is not mapper.EXT_PASS: return ret + + # convert composite types to individual args + # TODO: account for the order of columns in the + # ColumnProperty it corresponds to + if hasattr(ident, '__colset__'): + ident = ident.__colset__() + key = self.mapper.identity_key(ident) return self._get(key, ident, **kwargs) @@ -684,6 +691,7 @@ class Query(object): else: ident = util.to_list(ident) params = {} + for i, primary_key in enumerate(self.primary_key_columns): params[primary_key._label] = ident[i] try: diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py index a9cfed6c55..918551024a 100644 --- a/lib/sqlalchemy/orm/strategies.py +++ b/lib/sqlalchemy/orm/strategies.py @@ -19,6 +19,7 @@ class ColumnLoader(LoaderStrategy): super(ColumnLoader, self).init() self.columns = self.parent_property.columns self._should_log_debug = logging.is_debug_enabled(self.logger) + self.is_composite = hasattr(self.parent_property, 'composite_class') def setup_query(self, context, eagertable=None, parentclauses=None, **kwargs): for c in self.columns: @@ -28,12 +29,43 @@ class ColumnLoader(LoaderStrategy): context.statement.append_column(c) def init_class_attribute(self): + if self.is_composite: + self._init_composite_attribute() + else: + self._init_scalar_attribute() + + def _init_composite_attribute(self): + self.logger.info("register managed composite attribute %s on class %s" % (self.key, self.parent.class_.__name__)) + def copy(obj): + return self.parent_property.composite_class(*obj.__colset__()) + def compare(a, b): + for col, aprop, bprop in zip(self.columns, a.__colset__(), b.__colset__()): + if not col.type.compare_values(aprop, bprop): + return False + else: + return True + sessionlib.attribute_manager.register_attribute(self.parent.class_, self.key, uselist=False, copy_function=copy, compare_function=compare, mutable_scalars=True) + + def _init_scalar_attribute(self): self.logger.info("register managed attribute %s on class %s" % (self.key, self.parent.class_.__name__)) coltype = self.columns[0].type sessionlib.attribute_manager.register_attribute(self.parent.class_, self.key, uselist=False, copy_function=coltype.copy_value, compare_function=coltype.compare_values, mutable_scalars=self.columns[0].type.is_mutable()) - + def create_row_processor(self, selectcontext, mapper, row): - if self.columns[0] in row: + if self.is_composite: + for c in self.columns: + if c not in row: + break + else: + def execute(instance, row, isnew, ispostselect=None, **flags): + if isnew or ispostselect: + if self._should_log_debug: + self.logger.debug("populating %s with %s/%s..." % (mapperutil.attribute_str(instance, self.key), row.__class__.__name__, self.columns[0].key)) + instance.__dict__[self.key] = self.parent_property.composite_class(*[row[c] for c in self.columns]) + self.logger.debug("Returning active composite column fetcher for %s %s" % (mapper, self.key)) + return (execute, None) + + elif self.columns[0] in row: def execute(instance, row, isnew, ispostselect=None, **flags): if isnew or ispostselect: if self._should_log_debug: @@ -41,20 +73,20 @@ class ColumnLoader(LoaderStrategy): instance.__dict__[self.key] = row[self.columns[0]] self.logger.debug("Returning active column fetcher for %s %s" % (mapper, self.key)) return (execute, None) - else: - (hosted_mapper, needs_tables) = selectcontext.attributes.get(('polymorphic_fetch', mapper), (None, None)) - if hosted_mapper is None: - return (None, None) - - if hosted_mapper.polymorphic_fetch == 'deferred': - def execute(instance, row, isnew, **flags): - if isnew: - sessionlib.attribute_manager.init_instance_attribute(instance, self.key, callable_=self._get_deferred_loader(instance, mapper, needs_tables)) - self.logger.debug("Returning deferred column fetcher for %s %s" % (mapper, self.key)) - return (execute, None) - else: - self.logger.debug("Returning no column fetcher for %s %s" % (mapper, self.key)) - return (None, None) + + (hosted_mapper, needs_tables) = selectcontext.attributes.get(('polymorphic_fetch', mapper), (None, None)) + if hosted_mapper is None: + return (None, None) + + if hosted_mapper.polymorphic_fetch == 'deferred': + def execute(instance, row, isnew, **flags): + if isnew: + sessionlib.attribute_manager.init_instance_attribute(instance, self.key, callable_=self._get_deferred_loader(instance, mapper, needs_tables)) + self.logger.debug("Returning deferred column fetcher for %s %s" % (mapper, self.key)) + return (execute, None) + else: + self.logger.debug("Returning no column fetcher for %s %s" % (mapper, self.key)) + return (None, None) def _get_deferred_loader(self, instance, mapper, needs_tables): def load(): @@ -113,13 +145,15 @@ class DeferredColumnLoader(LoaderStrategy): def init(self): super(DeferredColumnLoader, self).init() + if hasattr(self.parent_property, 'composite_class'): + raise NotImplementedError("Deferred loading for composite types not implemented yet") self.columns = self.parent_property.columns self.group = self.parent_property.group self._should_log_debug = logging.is_debug_enabled(self.logger) def init_class_attribute(self): self.logger.info("register managed attribute %s on class %s" % (self.key, self.parent.class_.__name__)) - sessionlib.attribute_manager.register_attribute(self.parent.class_, self.key, uselist=False, callable_=lambda i:self.setup_loader(i), copy_function=lambda x: self.columns[0].type.copy_value(x), compare_function=lambda x,y:self.columns[0].type.compare_values(x,y), mutable_scalars=self.columns[0].type.is_mutable()) + sessionlib.attribute_manager.register_attribute(self.parent.class_, self.key, uselist=False, callable_=self.setup_loader, copy_function=self.columns[0].type.copy_value, compare_function=self.columns[0].type.compare_values, mutable_scalars=self.columns[0].type.is_mutable()) def setup_query(self, context, **kwargs): if self.group is not None and context.attributes.get(('undefer', self.group), False): diff --git a/test/orm/mapper.py b/test/orm/mapper.py index f5aa1f7e7a..4ea769a202 100644 --- a/test/orm/mapper.py +++ b/test/orm/mapper.py @@ -1,6 +1,6 @@ """tests general mapper operations with an emphasis on selecting/loading""" -from testbase import PersistTest, AssertMixin +from testbase import PersistTest, AssertMixin, ORMTest import testbase import unittest, sys, os from sqlalchemy import * @@ -817,8 +817,124 @@ class DeferredTest(MapperSuperTest): print item.item_name self.assert_sql_count(db, go, 0) self.assert_(item.item_name == 'item 4') - +class CompositeTypesTest(ORMTest): + def define_tables(self, metadata): + global graphs, edges + graphs = Table('graphs', metadata, + Column('id', Integer, primary_key=True), + Column('version_id', Integer, primary_key=True), + Column('name', String(30))) + + edges = Table('edges', metadata, + Column('id', Integer, primary_key=True), + Column('graph_id', Integer, ForeignKey('graphs.id'), nullable=False), + Column('x1', Integer), + Column('y1', Integer), + Column('x2', Integer), + Column('y2', Integer)) + + def test_basic(self): + class Point(object): + def __init__(self, x, y): + self.x = x + self.y = y + def __colset__(self): + return [self.x, self.y] + def __eq__(self, other): + return other.x == self.x and other.y == self.y + def __ne__(self, other): + return not self.__eq__(other) + + class Graph(object): + pass + class Edge(object): + def __init__(self, start, end): + self.start = start + self.end = end + + mapper(Graph, graphs, properties={ + 'edges':relation(Edge) + }) + mapper(Edge, edges, properties={ + 'start':composite(Point, edges.c.x1, edges.c.y1), + 'end':composite(Point, edges.c.x2, edges.c.y2) + }) + + sess = create_session() + g = Graph() + g.id = 1 + g.version_id=1 + g.edges.append(Edge(Point(3, 4), Point(5, 6))) + g.edges.append(Edge(Point(14, 5), Point(2, 7))) + sess.save(g) + sess.flush() + + sess.clear() + g2 = sess.query(Graph).get([g.id, g.version_id]) + for e1, e2 in zip(g.edges, g2.edges): + assert e1.start == e2.start + assert e1.end == e2.end + + g2.edges[1].end = Point(18, 4) + sess.flush() + sess.clear() + e = sess.query(Edge).get(g2.edges[1].id) + assert e.end == Point(18, 4) + + e.end.x = 19 + e.end.y = 5 + sess.flush() + sess.clear() + assert sess.query(Edge).get(g2.edges[1].id).end == Point(19, 5) + + g.edges[1].end = Point(19, 5) + + sess.clear() + def go(): + g2 = sess.query(Graph).options(eagerload('edges')).get([g.id, g.version_id]) + for e1, e2 in zip(g.edges, g2.edges): + assert e1.start == e2.start + assert e1.end == e2.end + self.assert_sql_count(testbase.db, go, 1) + + def test_pk(self): + """test using a composite type as a primary key""" + + class Version(object): + def __init__(self, id, version): + self.id = id + self.version = version + def __colset__(self): + return [self.id, self.version] + def __eq__(self, other): + return other.id == self.id and other.version == self.version + def __ne__(self, other): + return not self.__eq__(other) + + class Graph(object): + def __init__(self, version): + self.version = version + + mapper(Graph, graphs, properties={ + 'version':composite(Version, graphs.c.id, graphs.c.version_id) + }) + + sess = create_session() + g = Graph(Version(1, 1)) + sess.save(g) + sess.flush() + + sess.clear() + g2 = sess.query(Graph).get([1, 1]) + assert g.version == g2.version + sess.clear() + + g2 = sess.query(Graph).get(Version(1, 1)) + assert g.version == g2.version + + + class NoLoadTest(MapperSuperTest): def testbasic(self): """tests a basic one-to-many lazy load""" diff --git a/test/testbase.py b/test/testbase.py index 4e3fc2d659..854f348569 100644 --- a/test/testbase.py +++ b/test/testbase.py @@ -308,21 +308,21 @@ class ORMTest(AssertMixin): keep_mappers = False keep_data = False def setUpAll(self): - global metadata - metadata = BoundMetaData(db) - self.define_tables(metadata) - metadata.create_all() - def define_tables(self, metadata): + global _otest_metadata + _otest_metadata = BoundMetaData(db) + self.define_tables(_otest_metadata) + _otest_metadata.create_all() + def define_tables(self, _otest_metadata): raise NotImplementedError() def get_metadata(self): - return metadata + return _otest_metadata def tearDownAll(self): - metadata.drop_all() + _otest_metadata.drop_all() def tearDown(self): if not self.keep_mappers: clear_mappers() if not self.keep_data: - for t in metadata.table_iterator(reverse=True): + for t in _otest_metadata.table_iterator(reverse=True): t.delete().execute().close() class TestData(object):