+++ /dev/null
-<%flags>inherit='document_base.myt'</%flags>
-<%attr>title='Data Mapping'</%attr>
-
-<&|doclib.myt:item, name="datamapping", description="Basic Data Mapping" &>
-<p>Data mapping describes the process of defining <b>Mapper</b> objects, which associate table metadata with user-defined classes.
-
-The Mapper's role is to perform SQL operations upon the database, associating individual table rows with instances of those classes, and individual database columns with properties upon those instances, to transparently associate in-memory objects with a persistent database representation. </p>
-
-<p>When a Mapper is created to associate a Table object with a class, all of the columns defined in the Table object are associated with the class via property accessors, which add overriding functionality to the normal process of setting and getting object attributes. These property accessors keep track of changes to object attributes; these changes will be stored to the database when the application "commits" the current transactional context (known as a <b>Unit of Work</b>). The <code>__init__()</code> method of the object is also decorated to communicate changes when new instances of the object are created.</p>
-
-<p>The Mapper also provides the interface by which instances of the object are loaded from the database. The primary method for this is its <code>select()</code> method, which has similar arguments to a <code>sqlalchemy.sql.Select</code> object. But this select method executes automatically and returns results, instead of awaiting an execute() call. Instead of returning a cursor-like object, it returns an array of objects.</p>
-
-<p>The three elements to be defined, i.e. the Table metadata, the user-defined class, and the Mapper, are typically defined as module-level variables, and may be defined in any fashion suitable to the application, with the only requirement being that the class and table metadata are described before the mapper. For the sake of example, we will be defining these elements close together, but this should not be construed as a requirement; since SQLAlchemy is not a framework, those decisions are left to the developer or an external framework.
-</p>
-<&|doclib.myt:item, name="synopsis", description="Synopsis" &>
- <p>This is the simplest form of a full "round trip" of creating table meta data, creating a class, mapping the class to the table, getting some results, and saving changes. For each concept, the following sections will dig in deeper to the available capabilities.</p>
- <&|formatting.myt:code&>
- from sqlalchemy import *
-
- # engine
- engine = create_engine("sqlite://mydb.db")
-
- # table metadata
- users = Table('users', engine,
- Column('user_id', Integer, primary_key=True),
- Column('user_name', String(16)),
- Column('password', String(20))
- )
-
- # class definition
- class User(object):
- pass
-
- # create a mapper
- usermapper = mapper(User, users)
-
- # select
-<&formatting.myt:poplink&>user = usermapper.select_by(user_name='fred')[0]
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
-users.password AS users_password
-FROM users
-WHERE users.user_name = :users_user_name ORDER BY users.oid
-
-{'users_user_name': 'fred'}
- </&>
- # modify
- user.user_name = 'fred jones'
-
- # commit - saves everything that changed
-<&formatting.myt:poplink&>objectstore.commit()
-<&|formatting.myt:codepopper, link="sql" &>
-UPDATE users SET user_name=:user_name
- WHERE users.user_id = :user_id
-
-[{'user_name': 'fred jones', 'user_id': 1}]
- </&>
-
-
- </&>
- <&|doclib.myt:item, name="attaching", description="Attaching Mappers to their Class"&>
- <p>For convenience's sake, the Mapper can be attached as an attribute on the class itself as well:</p>
- <&|formatting.myt:code&>
- User.mapper = mapper(User, users)
-
- userlist = User.mapper.select_by(user_id=12)
- </&>
- <p>There is also a full-blown "monkeypatch" function that creates a primary mapper, attaches the above mapper class property, and also the methods <code>get, get_by, select, select_by, selectone, selectfirst, commit, expire, refresh, expunge</code> and <code>delete</code>:</p>
- <&|formatting.myt:code&>
- # "assign" a mapper to the User class/users table
- assign_mapper(User, users)
-
- # methods are attached to the class for selecting
- userlist = User.select_by(user_id=12)
-
- myuser = User.get(1)
-
- # mark an object as deleted for the next commit
- myuser.delete()
-
- # commit the changes on a specific object
- myotheruser.commit()
- </&>
- <p>Other methods of associating mappers and finder methods with their corresponding classes, such as via common base classes or mixins, can be devised as well. SQLAlchemy does not aim to dictate application architecture and will always allow the broadest variety of architectural patterns, but may include more helper objects and suggested architectures in the future.</p>
- </&>
- <&|doclib.myt:item, name="overriding", description="Overriding Properties"&>
- <p>A common request is the ability to create custom class properties that override the behavior of setting/getting an attribute. Currently, the easiest way to do this in SQLAlchemy is how it would be done in any Python program; define your attribute with a different name, such as "_attribute", and use a property to get/set its value. The mapper just needs to be told of the special name:</p>
- <&|formatting.myt:code&>
- class MyClass(object):
- def _set_email(self, email):
- self._email = email
- def _get_email(self, email):
- return self._email
- email = property(_get_email, _set_email)
-
- m = mapper(MyClass, mytable, properties = {
- # map the '_email' attribute to the "email" column
- # on the table
- '_email': mytable.c.email
- })
- </&>
- <p>In a later release, SQLAlchemy will also allow _get_email and _set_email to be attached directly to the "email" property created by the mapper, and will also allow this association to occur via decorators.</p>
- </&>
-</&>
-<&|doclib.myt:item, name="selecting", description="Selecting from a Mapper" &>
- <p>There are a variety of ways to select from a mapper. These range from minimalist to explicit. Below is a synopsis of the these methods:</p>
- <&|formatting.myt:code&>
- # select_by, using property names or column names as keys
- # the keys are grouped together by an AND operator
- result = mapper.select_by(name='john', street='123 green street')
-
- # select_by can also combine SQL criterion with key/value properties
- result = mapper.select_by(users.c.user_name=='john',
- addresses.c.zip_code=='12345, street='123 green street')
-
- # get_by, which takes the same arguments as select_by
- # returns a single scalar result or None if no results
- user = mapper.get_by(id=12)
-
- # "dynamic" versions of select_by and get_by - everything past the
- # "select_by_" or "get_by_" is used as the key, and the function argument
- # as the value
- result = mapper.select_by_name('fred')
- u = mapper.get_by_name('fred')
-
- # get an object directly from its primary key. this will bypass the SQL
- # call if the object has already been loaded
- u = mapper.get(15)
-
- # get an object that has a composite primary key of three columns.
- # the order of the arguments matches that of the table meta data.
- myobj = mapper.get(27, 3, 'receipts')
-
- # using a WHERE criterion
- result = mapper.select(or_(users.c.user_name == 'john', users.c.user_name=='fred'))
-
- # using a WHERE criterion to get a scalar
- u = mapper.selectfirst(users.c.user_name=='john')
-
- # selectone() is a stricter version of selectfirst() which
- # will raise an exception if there is not exactly one row
- u = mapper.selectone(users.c.user_name=='john')
-
- # using a full select object
- result = mapper.select(users.select(users.c.user_name=='john'))
-
- # using straight text
- result = mapper.select_text("select * from users where user_name='fred'")
-
- # or using a "text" object
- result = mapper.select(text("select * from users where user_name='fred'", engine=engine))
- </&>
- <p>Some of the above examples above illustrate the usage of the mapper's Table object to provide the columns for a WHERE Clause. These columns are also accessible off of the mapped class directly. When a mapper is assigned to a class, it also attaches a special property accessor <code>c</code> to the class itself, which can be used just like the table metadata to access the columns of the table:</p>
- <&|formatting.myt:code&>
- User.mapper = mapper(User, users)
-
- userlist = User.mapper.select(User.c.user_id==12)
- </&>
-</&>
-<&|doclib.myt:item, name="saving", description="Saving Objects" &>
- <p>When objects corresponding to mapped classes are created or manipulated, all changes are logged by a package called <code>sqlalchemy.mapping.objectstore</code>. The changes are then written to the database when an application calls <code>objectstore.commit()</code>. This pattern is known as a <b>Unit of Work</b>, and has many advantages over saving individual objects or attributes on those objects with individual method invocations. Domain models can be built with far greater complexity with no concern over the order of saves and deletes, excessive database round-trips and write operations, or deadlocking issues. The commit() operation uses a transaction as well, and will also perform "concurrency checking" to insure the proper number of rows were in fact affected (not supported with the current MySQL drivers). Transactional resources are used effectively in all cases; the unit of work handles all the details.</p>
- <p>The Unit of Work is a powerful tool, and has some important concepts that must be understood in order to use it effectively. While this section illustrates rudimentary Unit of Work usage, it is strongly encouraged to consult the <&formatting.myt:link, path="unitofwork"&> section for a full description on all its operations, including session control, deletion, and developmental guidelines.</p>
- <p>When a mapper is created, the target class has its mapped properties decorated by specialized property accessors that track changes, and its <code>__init__()</code> method is also decorated to mark new objects as "new".</p>
- <&|formatting.myt:code&>
- User.mapper = mapper(User, users)
-
- # create a new User
- myuser = User()
- myuser.user_name = 'jane'
- myuser.password = 'hello123'
-
- # create another new User
- myuser2 = User()
- myuser2.user_name = 'ed'
- myuser2.password = 'lalalala'
-
- # load a third User from the database
-<&formatting.myt:poplink&>myuser3 = User.mapper.select(User.c.user_name=='fred')[0]
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT users.user_id AS users_user_id,
-users.user_name AS users_user_name, users.password AS users_password
-FROM users WHERE users.user_name = :users_user_name
-{'users_user_name': 'fred'}
-</&>
- myuser3.user_name = 'fredjones'
-
- # save all changes
-<&formatting.myt:poplink&>objectstore.commit()
-<&|formatting.myt:codepopper, link="sql" &>
-UPDATE users SET user_name=:user_name
-WHERE users.user_id =:users_user_id
-[{'users_user_id': 1, 'user_name': 'fredjones'}]
-
-INSERT INTO users (user_name, password) VALUES (:user_name, :password)
-{'password': 'hello123', 'user_name': 'jane'}
-
-INSERT INTO users (user_name, password) VALUES (:user_name, :password)
-{'password': 'lalalala', 'user_name': 'ed'}
-</&>
- </&>
- <p>In the examples above, we defined a User class with basically no properties or methods. Theres no particular reason it has to be this way, the class can explicitly set up whatever properties it wants, whether or not they will be managed by the mapper. It can also specify a constructor, with the restriction that the constructor is able to function with no arguments being passed to it (this restriction can be lifted with some extra parameters to the mapper; more on that later):</p>
- <&|formatting.myt:code&>
- class User(object):
- def __init__(self, user_name = None, password = None):
- self.user_id = None
- self.user_name = user_name
- self.password = password
- def get_name(self):
- return self.user_name
- def __repr__(self):
- return "User id %s name %s password %s" % (repr(self.user_id),
- repr(self.user_name), repr(self.password))
- User.mapper = mapper(User, users)
-
- u = User('john', 'foo')
-<&formatting.myt:poplink&>objectstore.commit()
-<&|formatting.myt:codepopper, link="sql" &>
-INSERT INTO users (user_name, password) VALUES (:user_name, :password)
-{'password': 'foo', 'user_name': 'john'}
-</&>
- >>> u
- User id 1 name 'john' password 'foo'
-
- </&>
-
-<p>Recent versions of SQLAlchemy will only put modified object attributes columns into the UPDATE statements generated upon commit. This is to conserve database traffic and also to successfully interact with a "deferred" attribute, which is a mapped object attribute against the mapper's primary table that isnt loaded until referenced by the application.</p>
-
-</&>
-
-<&|doclib.myt:item, name="relations", description="Defining and Using Relationships" &>
-<p>So that covers how to map the columns in a table to an object, how to load objects, create new ones, and save changes. The next step is how to define an object's relationships to other database-persisted objects. This is done via the <code>relation</code> function provided by the mapper module. So with our User class, lets also define the User has having one or more mailing addresses. First, the table metadata:</p>
- <&|formatting.myt:code&>
- from sqlalchemy import *
- engine = create_engine('sqlite://filename=mydb')
-
- # define user table
- users = Table('users', engine,
- Column('user_id', Integer, primary_key=True),
- Column('user_name', String(16)),
- Column('password', String(20))
- )
-
- # define user address table
- addresses = Table('addresses', engine,
- Column('address_id', Integer, primary_key=True),
- Column('user_id', Integer, ForeignKey("users.user_id")),
- Column('street', String(100)),
- Column('city', String(80)),
- Column('state', String(2)),
- Column('zip', String(10))
- )
- </&>
-<p>Of importance here is the addresses table's definition of a <b>foreign key</b> relationship to the users table, relating the user_id column into a parent-child relationship. When a Mapper wants to indicate a relation of one object to another, this ForeignKey object is the default method by which the relationship is determined (although if you didn't define ForeignKeys, or you want to specify explicit relationship columns, that is available as well). </p>
-<p>So then lets define two classes, the familiar User class, as well as an Address class:
-
- <&|formatting.myt:code&>
- class User(object):
- def __init__(self, user_name = None, password = None):
- self.user_name = user_name
- self.password = password
-
- class Address(object):
- def __init__(self, street=None, city=None, state=None, zip=None):
- self.street = street
- self.city = city
- self.state = state
- self.zip = zip
- </&>
-<p>And then a Mapper that will define a relationship of the User and the Address classes to each other as well as their table metadata. We will add an additional mapper keyword argument <code>properties</code> which is a dictionary relating the name of an object property to a database relationship, in this case a <code>relation</code> object against a newly defined mapper for the Address class:</p>
- <&|formatting.myt:code&>
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(mapper(Address, addresses))
- }
- )
- </&>
-<p>Lets do some operations with these classes and see what happens:</p>
-
- <&|formatting.myt:code&>
- u = User('jane', 'hihilala')
- u.addresses.append(Address('123 anywhere street', 'big city', 'UT', '76543'))
- u.addresses.append(Address('1 Park Place', 'some other city', 'OK', '83923'))
-
- objectstore.commit()
-<&|formatting.myt:poppedcode, link="sql" &>INSERT INTO users (user_name, password) VALUES (:user_name, :password)
-{'password': 'hihilala', 'user_name': 'jane'}
-
-INSERT INTO addresses (user_id, street, city, state, zip) VALUES (:user_id, :street, :city, :state, :zip)
-{'city': 'big city', 'state': 'UT', 'street': '123 anywhere street', 'user_id':1, 'zip': '76543'}
-
-INSERT INTO addresses (user_id, street, city, state, zip) VALUES (:user_id, :street, :city, :state, :zip)
-{'city': 'some other city', 'state': 'OK', 'street': '1 Park Place', 'user_id':1, 'zip': '83923'}
-</&>
- </&>
-<p>A lot just happened there! The Mapper object figured out how to relate rows in the addresses table to the users table, and also upon commit had to determine the proper order in which to insert rows. After the insert, all the User and Address objects have all their new primary and foreign keys populated.</p>
-
-<p>Also notice that when we created a Mapper on the User class which defined an 'addresses' relation, the newly created User instance magically had an "addresses" attribute which behaved like a list. This list is in reality a property accessor function, which returns an instance of <code>sqlalchemy.util.HistoryArraySet</code>, which fulfills the full set of Python list accessors, but maintains a <b>unique</b> set of objects (based on their in-memory identity), and also tracks additions and deletions to the list:</p>
- <&|formatting.myt:code&>
- del u.addresses[1]
- u.addresses.append(Address('27 New Place', 'Houston', 'TX', '34839'))
-
- objectstore.commit()
-
-<&|formatting.myt:poppedcode, link="sql" &>UPDATE addresses SET user_id=:user_id
- WHERE addresses.address_id = :addresses_address_id
-[{'user_id': None, 'addresses_address_id': 2}]
-
-INSERT INTO addresses (user_id, street, city, state, zip)
-VALUES (:user_id, :street, :city, :state, :zip)
-{'city': 'Houston', 'state': 'TX', 'street': '27 New Place', 'user_id': 1, 'zip': '34839'}
-</&>
-
- </&>
-<&|doclib.myt:item, name="private", description="Useful Feature: Private Relations" &>
-<p>So our one address that was removed from the list, was updated to have a user_id of <code>None</code>, and a new address object was inserted to correspond to the new Address added to the User. But now, theres a mailing address with no user_id floating around in the database of no use to anyone. How can we avoid this ? This is acheived by using the <code>private=True</code> parameter of <code>relation</code>:
-
- <&|formatting.myt:code&>
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(mapper(Address, addresses), private=True)
- }
- )
- del u.addresses[1]
- u.addresses.append(Address('27 New Place', 'Houston', 'TX', '34839'))
-
- objectstore.commit() <&|formatting.myt:poppedcode, link="sql" &>
-INSERT INTO addresses (user_id, street, city, state, zip)
-VALUES (:user_id, :street, :city, :state, :zip)
-{'city': 'Houston', 'state': 'TX', 'street': '27 New Place', 'user_id': 1, 'zip': '34839'}
-
-DELETE FROM addresses WHERE addresses.address_id = :address_id
-[{'address_id': 2}]
-</&>
-
- </&>
-<p>In this case, with the private flag set, the element that was removed from the addresses list was also removed from the database. By specifying the <code>private</code> flag on a relation, it is indicated to the Mapper that these related objects exist only as children of the parent object, otherwise should be deleted.</p>
-</&>
-<&|doclib.myt:item, name="backreferences", description="Useful Feature: Backreferences" &>
-<p>By creating relations with the <code>backref</code> keyword, a bi-directional relationship can be created which will keep both ends of the relationship updated automatically, even without any database queries being executed. Below, the User mapper is created with an "addresses" property, and the corresponding Address mapper receives a "backreference" to the User object via the property name "user":
- <&|formatting.myt:code&>
- Address.mapper = mapper(Address, addresses)
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(Address.mapper, backref='user')
- }
- )
-
- u = User('fred', 'hi')
- a1 = Address('123 anywhere street', 'big city', 'UT', '76543')
- a2 = Address('1 Park Place', 'some other city', 'OK', '83923')
-
- # append a1 to u
- u.addresses.append(a1)
-
- # attach u to a2
- a2.user = u
-
- # the bi-directional relation is maintained
- >>> u.addresses == [a1, a2]
- True
- >>> a1.user is user and a2.user is user
- True
- </&>
-
-<p>The backreference feature also works with many-to-many relationships, which are described later. When creating a backreference, a corresponding property is placed on the child mapper. The default arguments to this property can be overridden using the <code>backref()</code> function:
- <&|formatting.myt:code&>
- Address.mapper = mapper(Address, addresses)
-
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(Address.mapper,
- backref=backref('user', lazy=False, private=True))
- }
- )
- </&>
-</&>
-<&|doclib.myt:item, name="cascade", description="Creating Relationships Automatically with cascade_mappers" &>
-<p>The mapper package has a helper function <code>cascade_mappers()</code> which can simplify the task of linking several mappers together. Given a list of classes and/or mappers, it identifies the foreign key relationships between the given mappers or corresponding class mappers, and creates relation() objects representing those relationships, including a backreference. Attempts to find
-the "secondary" table in a many-to-many relationship as well. The names of the relations
-are a lowercase version of the related class. In the case of one-to-many or many-to-many,
-the name is "pluralized", which currently is based on the English language (i.e. an 's' or
-'es' added to it):</p>
- <&|formatting.myt:code&>
- # create two mappers. the 'users' and 'addresses' tables have a foreign key
- # relationship
- mapper1 = mapper(User, users)
- mapper2 = mapper(Address, addresses)
-
- # cascade the two mappers together (can also specify User, Address as the arguments)
- cascade_mappers(mapper1, mapper2)
-
- # two new object instances
- u = User('user1')
- a = Address('test')
-
- # "addresses" and "user" property are automatically added
- u.addresses.append(a)
- print a.user
- </&>
-
-</&>
- <&|doclib.myt:item, name="lazyload", description="Selecting from Relationships: Lazy Load" &>
- <P>We've seen how the <code>relation</code> specifier affects the saving of an object and its child items, how does it affect selecting them? By default, the relation keyword indicates that the related property should be attached a <b>Lazy Loader</b> when instances of the parent object are loaded from the database; this is just a callable function that when accessed will invoke a second SQL query to load the child objects of the parent.</p>
-
- <&|formatting.myt:code&>
- # define a mapper
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(mapper(Address, addresses), private=True)
- })
-
- # select users where username is 'jane', get the first element of the list
- # this will incur a load operation for the parent table
- user = User.mapper.select(user_name='jane')[0]
-
-<&|formatting.myt:poppedcode, link="sql" &>SELECT users.user_id AS users_user_id,
-users.user_name AS users_user_name, users.password AS users_password
-FROM users WHERE users.user_name = :users_user_name ORDER BY users.oid
-{'users_user_name': 'jane'}
-</&>
-
- # iterate through the User object's addresses. this will incur an
- # immediate load of those child items
- for a in user.addresses:
-<&|formatting.myt:poppedcode, link="sql" &>SELECT addresses.address_id AS addresses_address_id,
-addresses.user_id AS addresses_user_id, addresses.street AS addresses_street,
-addresses.city AS addresses_city, addresses.state AS addresses_state,
-addresses.zip AS addresses_zip FROM addresses
-WHERE addresses.user_id = :users_user_id ORDER BY addresses.oid
-{'users_user_id': 1}</&>
- print repr(a)
-
- </&>
- <&|doclib.myt:item, name="relselectby", description="Useful Feature: Creating Joins via select_by" &>
- <p>In mappers that have relationships, the <code>select_by</code> method and its cousins include special functionality that can be used to create joins. Just specify a key in the argument list which is not present in the primary mapper's list of properties or columns, but *is* present in the property list of one of its relationships:
- <&|formatting.myt:code&>
- <&formatting.myt:poplink&>l = User.mapper.select_by(street='123 Green Street')
-<&|formatting.myt:codepopper, link="sql" &>SELECT users.user_id AS users_user_id,
-users.user_name AS users_user_name, users.password AS users_password
-FROM users, addresses
-WHERE users.user_id=addresses.user_id
-AND addresses.street=:addresses_street
-ORDER BY users.oid
-{'addresses_street', '123 Green Street'}
-</&>
- </&>
- <p>The above example is shorthand for:</p>
- <&|formatting.myt:code&>
- l = User.mapper.select(and_(
- Address.c.user_id==User.c.user_id,
- Address.c.street=='123 Green Street')
- )
- </&>
-
- </&>
- <&|doclib.myt:item, name="refreshing", description="How to Refresh the List?" &>
- <p>Once the child list of Address objects is loaded, it is done loading for the lifetime of the object instance. Changes to the list will not be interfered with by subsequent loads, and upon commit those changes will be saved. Similarly, if a new User object is created and child Address objects added, a subsequent select operation which happens to touch upon that User instance, will also not affect the child list, since it is already loaded.</p>
-
- <p>The issue of when the mapper actually gets brand new objects from the database versus when it assumes the in-memory version is fine the way it is, is a subject of <b>transactional scope</b>. Described in more detail in the Unit of Work section, for now it should be noted that the total storage of all newly created and selected objects, <b>within the scope of the current thread</b>, can be reset via releasing or otherwise disregarding all current object instances, and calling:</p>
- <&|formatting.myt:code&>
- objectstore.clear()
- </&>
- <p>This operation will clear out all currently mapped object instances, and subsequent select statements will load fresh copies from the databse.</p>
-
- <p>To operate upon a single object, just use the <code>remove</code> function:</p>
- <&|formatting.myt:code&>
- # (this function coming soon)
- objectstore.remove(myobject)
- </&>
-
- </&>
- </&>
- <&|doclib.myt:item, name="eagerload", description="Selecting from Relationships: Eager Load" &>
- <p>With just a single parameter "lazy=False" specified to the relation object, the parent and child SQL queries can be joined together.
-
- <&|formatting.myt:code&>
- Address.mapper = mapper(Address, addresses)
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(Address.mapper, lazy=False)
- }
- )
-
- user = User.mapper.get_by(user_name='jane')
-
-<&|formatting.myt:poppedcode, link="sql" &>SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
-users.password AS users_password,
-addresses.address_id AS addresses_address_id, addresses.user_id AS addresses_user_id,
-addresses.street AS addresses_street, addresses.city AS addresses_city,
-addresses.state AS addresses_state, addresses.zip AS addresses_zip
-FROM users LEFT OUTER JOIN addresses ON users.user_id = addresses.user_id
-WHERE users.user_name = :users_user_name ORDER BY users.oid, addresses.oid
-{'users_user_name': 'jane'}
-</&>
- for a in user.addresses:
- print repr(a)
-
- </&>
- <P>Above, a pretty ambitious query is generated just by specifying that the User should be loaded with its child Addresses in one query. When the mapper processes the results, it uses an <b>Identity Map</b> to keep track of objects that were already loaded, based on their primary key identity. Through this method, the redundant rows produced by the join are organized into the distinct object instances they represent.</p>
-
- <p>The generation of this query is also immune to the effects of additional joins being specified in the original query. To use our select_by example above, joining against the "addresses" table to locate users with a certain street results in this behavior:
- <&|formatting.myt:code&>
- users = User.mapper.select_by(street='123 Green Street')
-
-<&|formatting.myt:poppedcode, link="sql" &>SELECT users.user_id AS users_user_id,
-users.user_name AS users_user_name, users.password AS users_password,
-addresses.address_id AS addresses_address_id,
-addresses.user_id AS addresses_user_id, addresses.street AS addresses_street,
-addresses.city AS addresses_city, addresses.state AS addresses_state,
-addresses.zip AS addresses_zip
-FROM addresses AS addresses_417c,
-users LEFT OUTER JOIN addresses ON users.user_id = addresses.user_id
-WHERE addresses_417c.street = :addresses_street
-AND users.user_id = addresses_417c.user_id
-ORDER BY users.oid, addresses.oid
-{'addresses_street': '123 Green Street'}
-</&>
- </&>
- <p>The join implied by passing the "street" parameter is converted into an "aliasized" clause by the eager loader, so that it does not conflict with the join used to eager load the child address objects.</p>
- </&>
- <&|doclib.myt:item, name="options", description="Switching Lazy/Eager, No Load" &>
- <p>The <code>options</code> method of mapper provides an easy way to get alternate forms of a mapper from an original one. The most common use of this feature is to change the "eager/lazy" loading behavior of a particular mapper, via the functions <code>eagerload()</code>, <code>lazyload()</code> and <code>noload()</code>:
- </p>
- <&|formatting.myt:code&>
- # user mapper with lazy addresses
- User.mapper = mapper(User, users, properties = {
- 'addresses' : relation(mapper(Address, addresses))
- }
- )
-
- # make an eager loader
- eagermapper = User.mapper.options(eagerload('addresses'))
- u = eagermapper.select()
-
- # make another mapper that wont load the addresses at all
- plainmapper = User.mapper.options(noload('addresses'))
-
- # multiple options can be specified
- mymapper = oldmapper.options(lazyload('tracker'), noload('streets'), eagerload('members'))
-
- # to specify a relation on a relation, separate the property names by a "."
- mymapper = oldmapper.options(eagerload('orders.items'))
-
- </&>
-
- </&>
-
-</&>
-
-
-<&|doclib.myt:item, name="onetoone", description="One to One/Many to One" &>
-<p>The above examples focused on the "one-to-many" relationship. To do other forms of relationship is easy, as the <code>relation</code> function can usually figure out what you want:</p>
-
- <&|formatting.myt:code&>
- # a table to store a user's preferences for a site
- prefs = Table('user_prefs', engine,
- Column('pref_id', Integer, primary_key = True),
- Column('stylename', String(20)),
- Column('save_password', Boolean, nullable = False),
- Column('timezone', CHAR(3), nullable = False)
- )
-
- # user table gets 'preference_id' column added
- users = Table('users', engine,
- Column('user_id', Integer, primary_key = True),
- Column('user_name', String(16), nullable = False),
- Column('password', String(20), nullable = False),
- Column('preference_id', Integer, ForeignKey("prefs.pref_id"))
- )
-
- # class definition for preferences
- class UserPrefs(object):
- pass
- UserPrefs.mapper = mapper(UserPrefs, prefs)
-
- # address mapper
- Address.mapper = mapper(Address, addresses)
-
- # make a new mapper referencing everything.
- m = mapper(User, users, properties = dict(
- addresses = relation(Address.mapper, lazy=True, private=True),
- preferences = relation(UserPrefs.mapper, lazy=False, private=True),
- ))
-
- # select
-<&formatting.myt:poplink&>user = m.get_by(user_name='fred')
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
-users.password AS users_password, users.preference_id AS users_preference_id,
-user_prefs.pref_id AS user_prefs_pref_id, user_prefs.stylename AS user_prefs_stylename,
-user_prefs.save_password AS user_prefs_save_password, user_prefs.timezone AS user_prefs_timezone
-FROM users LEFT OUTER JOIN user_prefs ON user_prefs.pref_id = users.preference_id
-WHERE users.user_name = :users_user_name ORDER BY users.oid, user_prefs.oid
-
-{'users_user_name': 'fred'}
- </&>
- save_password = user.preferences.save_password
-
- # modify
- user.preferences.stylename = 'bluesteel'
-<&formatting.myt:poplink&>user.addresses.append(Address('freddy@hi.org'))
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT email_addresses.address_id AS email_addresses_address_id,
-email_addresses.user_id AS email_addresses_user_id,
-email_addresses.email_address AS email_addresses_email_address
-FROM email_addresses
-WHERE email_addresses.user_id = :users_user_id
-ORDER BY email_addresses.oid, email_addresses.oid
-
-{'users_user_id': 1}
- </&>
- # commit
- <&formatting.myt:poplink&>objectstore.commit()
-<&|formatting.myt:codepopper, link="sql" &>
-UPDATE user_prefs SET stylename=:stylename
-WHERE user_prefs.pref_id = :pref_id
-
-[{'stylename': 'bluesteel', 'pref_id': 1}]
-
-INSERT INTO email_addresses (address_id, user_id, email_address)
-VALUES (:address_id, :user_id, :email_address)
-
-{'email_address': 'freddy@hi.org', 'address_id': None, 'user_id': 1}
-</&>
- </&>
-</&>
-
-<&|doclib.myt:item, name="manytomany", description="Many to Many" &>
-<p>The <code>relation</code> function handles a basic many-to-many relationship when you specify the association table:</p>
- <&|formatting.myt:code&>
- articles = Table('articles', engine,
- Column('article_id', Integer, primary_key = True),
- Column('headline', String(150), key='headline'),
- Column('body', TEXT, key='body'),
- )
-
- keywords = Table('keywords', engine,
- Column('keyword_id', Integer, primary_key = True),
- Column('keyword_name', String(50))
- )
-
- itemkeywords = Table('article_keywords', engine,
- Column('article_id', Integer, ForeignKey("articles.article_id")),
- Column('keyword_id', Integer, ForeignKey("keywords.keyword_id"))
- )
-
- # class definitions
- class Keyword(object):
- def __init__(self, name = None):
- self.keyword_name = name
-
- class Article(object):
- pass
-
- # define a mapper that does many-to-many on the 'itemkeywords' association
- # table
- Article.mapper = mapper(Article, articles, properties = dict(
- keywords = relation(mapper(Keyword, keywords), itemkeywords, lazy=False)
- )
- )
-
- article = Article()
- article.headline = 'a headline'
- article.body = 'this is the body'
- article.keywords.append(Keyword('politics'))
- article.keywords.append(Keyword('entertainment'))
- <&formatting.myt:poplink&>
- objectstore.commit() <&|formatting.myt:codepopper, link="sql" &>
-INSERT INTO keywords (name) VALUES (:name)
-
-{'name': 'politics'}
-
-INSERT INTO keywords (name) VALUES (:name)
-
-{'name': 'entertainment'}
-
-INSERT INTO articles (article_headline, article_body) VALUES (:article_headline, :article_body)
-
-{'article_body': 'this is the body', 'article_headline': 'a headline'}
-
-INSERT INTO article_keywords (article_id, keyword_id) VALUES (:article_id, :keyword_id)
-
-[{'keyword_id': 1, 'article_id': 1}, {'keyword_id': 2, 'article_id': 1}]
-</&>
- # select articles based on a keyword. select_by will handle the extra joins.
- <&formatting.myt:poplink&>articles = Article.mapper.select_by(keyword_name='politics')
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT articles.article_id AS articles_article_id,
-articles.article_headline AS articles_article_headline,
-articles.article_body AS articles_article_body,
-keywords.keyword_id AS keywords_keyword_id,
-keywords.keyword_name AS keywords_keyword_name
-FROM keywords AS keywords_f008,
-article_keywords AS article_keywords_dbf0,
-articles LEFT OUTER JOIN article_keywords ON
-articles.article_id = article_keywords.article_id
-LEFT OUTER JOIN keywords ON
-keywords.keyword_id = article_keywords.keyword_id
-WHERE (keywords_f008.keyword_name = :keywords_keyword_name
-AND articles.article_id = article_keywords_dbf0.article_id)
-AND keywords_f008.keyword_id = article_keywords_dbf0.keyword_id
-ORDER BY articles.oid, article_keywords.oid
-{'keywords_keyword_name': 'politics'}
-</&>
- # modify
- a = articles[0]
- del a.keywords[:]
- a.keywords.append(Keyword('topstories'))
- a.keywords.append(Keyword('government'))
-
- # commit. individual INSERT/DELETE operations will take place only for the list
- # elements that changed.
-<&formatting.myt:poplink&>
- objectstore.commit()
-<&|formatting.myt:codepopper &>
-INSERT INTO keywords (name) VALUES (:name)
-
-{'name': 'topstories'}
-
-INSERT INTO keywords (name) VALUES (:name)
-
-{'name': 'government'}
-
-DELETE FROM article_keywords
-WHERE article_keywords.article_id = :article_id
-AND article_keywords.keyword_id = :keyword_id
-
-[{'keyword_id': 1, 'article_id': 1}, {'keyword_id': 2, 'article_id': 1}]
-
-INSERT INTO article_keywords (article_id, keyword_id) VALUES (:article_id, :keyword_id)
-
-[{'keyword_id': 3, 'article_id': 1}, {'keyword_id': 4, 'article_id': 1}]
-</&>
-
-
- </&>
-</&>
-<&|doclib.myt:item, name="association", description="Association Object" &>
-
- <p>Many to Many can also be done with an association object, that adds additional information about how two items are related. This association object is set up in basically the same way as any other mapped object. However, since an association table typically has no primary key columns, you have to tell the mapper what columns will compose its "primary key", which are the two (or more) columns involved in the association. Also, the relation function needs an additional hint as to the fact that this mapped object is an association object, via the "association" argument which points to the class or mapper representing the other side of the association.</p>
- <&|formatting.myt:code&>
- # add "attached_by" column which will reference the user who attached this keyword
- itemkeywords = Table('article_keywords', engine,
- Column('article_id', Integer, ForeignKey("articles.article_id")),
- Column('keyword_id', Integer, ForeignKey("keywords.keyword_id")),
- Column('attached_by', Integer, ForeignKey("users.user_id"))
- )
-
- # define an association class
- class KeywordAssociation(object):
- pass
-
- # mapper for KeywordAssociation
- # specify "primary key" columns manually
- KeywordAssociation.mapper = mapper(KeywordAssociation, itemkeywords,
- primary_key = [itemkeywords.c.article_id, itemkeywords.c.keyword_id],
- properties={
- 'keyword' : relation(Keyword, lazy = False), # uses primary Keyword mapper
- 'user' : relation(User, lazy = True) # uses primary User mapper
- }
- )
-
- # mappers for Users, Keywords
- User.mapper = mapper(User, users)
- Keyword.mapper = mapper(Keyword, keywords)
-
- # define the mapper.
- m = mapper(Article, articles, properties={
- 'keywords':relation(KeywordAssociation.mapper, lazy=False, association=Keyword)
- }
- )
-
- # bonus step - well, we do want to load the users in one shot,
- # so modify the mapper via an option.
- # this returns a new mapper with the option switched on.
- m2 = mapper.options(eagerload('keywords.user'))
-
- # select by keyword again
- <&formatting.myt:poplink&>alist = m2.select_by(keyword_name='jacks_stories')
-<&|formatting.myt:codepopper, link="sql" &>
-SELECT articles.article_id AS articles_article_id,
-articles.article_headline AS articles_article_headline,
-articles.article_body AS articles_article_body,
-article_keywords.article_id AS article_keywords_article_id,
-article_keywords.keyword_id AS article_keywords_keyword_id,
-article_keywords.attached_by AS article_keywords_attached_by,
-users.user_id AS users_user_id, users.user_name AS users_user_name,
-users.password AS users_password, users.preference_id AS users_preference_id,
-keywords.keyword_id AS keywords_keyword_id, keywords.name AS keywords_name
-FROM article_keywords article_keywords_3a64, keywords keywords_11b7,
-articles LEFT OUTER JOIN article_keywords ON articles.article_id = article_keywords.article_id
-LEFT OUTER JOIN users ON users.user_id = article_keywords.attached_by
-LEFT OUTER JOIN keywords ON keywords.keyword_id = article_keywords.keyword_id
-WHERE keywords_11b7.keyword_id = article_keywords_3a64.keyword_id
-AND article_keywords_3a64.article_id = articles.article_id
-AND keywords_11b7.name = :keywords_name
-ORDER BY articles.oid, article_keywords.oid, users.oid, keywords.oid
-
-{'keywords_name': 'jacks_stories'}
-</&>
- # user is available
- for a in alist:
- for k in a.keywords:
- if k.keyword.name == 'jacks_stories':
- print k.user.user_name
-
-</&>
-
-</&>
-
-</&>
--- /dev/null
+Data Mapping
+============
+
+### Basic Data Mapping {@name=datamapping}
+
+Data mapping describes the process of defining *Mapper* objects, which associate table metadata with user-defined classes.
+
+The Mapper's role is to perform SQL operations upon the database, associating individual table rows with instances of those classes, and individual database columns with properties upon those instances, to transparently associate in-memory objects with a persistent database representation.
+
+When a Mapper is created to associate a Table object with a class, all of the columns defined in the Table object are associated with the class via property accessors, which add overriding functionality to the normal process of setting and getting object attributes. These property accessors keep track of changes to object attributes; these changes will be stored to the database when the application "commits" the current transactional context (known as a *Unit of Work*). The `__init__()` method of the object is also decorated to communicate changes when new instances of the object are created.
+
+The Mapper also provides the interface by which instances of the object are loaded from the database. The primary method for this is its `select()` method, which has similar arguments to a `sqlalchemy.sql.Select` object. But this select method executes automatically and returns results, instead of awaiting an execute() call. Instead of returning a cursor-like object, it returns an array of objects.
+
+The three elements to be defined, i.e. the Table metadata, the user-defined class, and the Mapper, are typically defined as module-level variables, and may be defined in any fashion suitable to the application, with the only requirement being that the class and table metadata are described before the mapper. For the sake of example, we will be defining these elements close together, but this should not be construed as a requirement; since SQLAlchemy is not a framework, those decisions are left to the developer or an external framework.
+
+### Synopsis {@name=synopsis}
+
+This is the simplest form of a full "round trip" of creating table meta data, creating a class, mapping the class to the table, getting some results, and saving changes. For each concept, the following sections will dig in deeper to the available capabilities.
+
+ {python}
+ from sqlalchemy import *
+
+ # engine
+ engine = create_engine("sqlite://mydb.db")
+
+ # table metadata
+ users = Table('users', engine,
+ Column('user_id', Integer, primary_key=True),
+ Column('user_name', String(16)),
+ Column('password', String(20))
+ )
+
+ # class definition
+ class User(object):
+ pass
+
+ # create a mapper
+ usermapper = mapper(User, users)
+
+ # select
+ {sql}user = usermapper.select_by(user_name='fred')[0]
+ SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
+ users.password AS users_password
+ FROM users
+ WHERE users.user_name = :users_user_name ORDER BY users.oid
+ {'users_user_name': 'fred'}
+
+ # modify
+ user.user_name = 'fred jones'
+
+ # commit - saves everything that changed
+ {sql}objectstore.commit()
+ UPDATE users SET user_name=:user_name
+ WHERE users.user_id = :user_id
+ [{'user_name': 'fred jones', 'user_id': 1}]
+
+
+#### Attaching Mappers to their Class {@name=attaching}
+
+For convenience's sake, the Mapper can be attached as an attribute on the class itself as well:
+
+ {python}
+ User.mapper = mapper(User, users)
+
+ userlist = User.mapper.select_by(user_id=12)
+
+There is also a full-blown "monkeypatch" function that creates a primary mapper, attaches the above mapper class property, and also the methods `get, get_by, select, select_by, selectone, selectfirst, commit, expire, refresh, expunge` and `delete`:
+
+ {python}
+ # "assign" a mapper to the User class/users table
+ assign_mapper(User, users)
+
+ # methods are attached to the class for selecting
+ userlist = User.select_by(user_id=12)
+
+ myuser = User.get(1)
+
+ # mark an object as deleted for the next commit
+ myuser.delete()
+
+ # commit the changes on a specific object
+ myotheruser.commit()
+
+Other methods of associating mappers and finder methods with their corresponding classes, such as via common base classes or mixins, can be devised as well. SQLAlchemy does not aim to dictate application architecture and will always allow the broadest variety of architectural patterns, but may include more helper objects and suggested architectures in the future.
+
+#### Overriding Properties {@name=overriding}
+
+A common request is the ability to create custom class properties that override the behavior of setting/getting an attribute. Currently, the easiest way to do this in SQLAlchemy is how it would be done in any Python program; define your attribute with a different name, such as "_attribute", and use a property to get/set its value. The mapper just needs to be told of the special name:
+
+ {python}
+ class MyClass(object):
+ def _set_email(self, email):
+ self._email = email
+ def _get_email(self, email):
+ return self._email
+ email = property(_get_email, _set_email)
+
+ m = mapper(MyClass, mytable, properties = {
+ # map the '_email' attribute to the "email" column
+ # on the table
+ '_email': mytable.c.email
+ })
+
+In a later release, SQLAlchemy will also allow _get_email and _set_email to be attached directly to the "email" property created by the mapper, and will also allow this association to occur via decorators.
+
+### Selecting from a Mapper {@name=selecting}
+
+There are a variety of ways to select from a mapper. These range from minimalist to explicit. Below is a synopsis of the these methods:
+
+ {python}
+ # select_by, using property names or column names as keys
+ # the keys are grouped together by an AND operator
+ result = mapper.select_by(name='john', street='123 green street')
+
+ # select_by can also combine SQL criterion with key/value properties
+ result = mapper.select_by(users.c.user_name=='john',
+ addresses.c.zip_code=='12345, street='123 green street')
+
+ # get_by, which takes the same arguments as select_by
+ # returns a single scalar result or None if no results
+ user = mapper.get_by(id=12)
+
+ # "dynamic" versions of select_by and get_by - everything past the
+ # "select_by_" or "get_by_" is used as the key, and the function argument
+ # as the value
+ result = mapper.select_by_name('fred')
+ u = mapper.get_by_name('fred')
+
+ # get an object directly from its primary key. this will bypass the SQL
+ # call if the object has already been loaded
+ u = mapper.get(15)
+
+ # get an object that has a composite primary key of three columns.
+ # the order of the arguments matches that of the table meta data.
+ myobj = mapper.get(27, 3, 'receipts')
+
+ # using a WHERE criterion
+ result = mapper.select(or_(users.c.user_name == 'john', users.c.user_name=='fred'))
+
+ # using a WHERE criterion to get a scalar
+ u = mapper.selectfirst(users.c.user_name=='john')
+
+ # selectone() is a stricter version of selectfirst() which
+ # will raise an exception if there is not exactly one row
+ u = mapper.selectone(users.c.user_name=='john')
+
+ # using a full select object
+ result = mapper.select(users.select(users.c.user_name=='john'))
+
+ # using straight text
+ result = mapper.select_text("select * from users where user_name='fred'")
+
+ # or using a "text" object
+ result = mapper.select(text("select * from users where user_name='fred'", engine=engine))
+
+Some of the above examples above illustrate the usage of the mapper's Table object to provide the columns for a WHERE Clause. These columns are also accessible off of the mapped class directly. When a mapper is assigned to a class, it also attaches a special property accessor `c` to the class itself, which can be used just like the table metadata to access the columns of the table:
+
+ {python}
+ User.mapper = mapper(User, users)
+
+ userlist = User.mapper.select(User.c.user_id==12)
+
+
+### Saving Objects {@name=saving}
+
+When objects corresponding to mapped classes are created or manipulated, all changes are logged by a package called `sqlalchemy.mapping.objectstore`. The changes are then written to the database when an application calls `objectstore.commit()`. This pattern is known as a *Unit of Work*, and has many advantages over saving individual objects or attributes on those objects with individual method invocations. Domain models can be built with far greater complexity with no concern over the order of saves and deletes, excessive database round-trips and write operations, or deadlocking issues. The commit() operation uses a transaction as well, and will also perform "concurrency checking" to insure the proper number of rows were in fact affected (not supported with the current MySQL drivers). Transactional resources are used effectively in all cases; the unit of work handles all the details.
+
+The Unit of Work is a powerful tool, and has some important concepts that must be understood in order to use it effectively. While this section illustrates rudimentary Unit of Work usage, it is strongly encouraged to consult the [unitofwork](rel:unitofwork) section for a full description on all its operations, including session control, deletion, and developmental guidelines.
+
+When a mapper is created, the target class has its mapped properties decorated by specialized property accessors that track changes, and its `__init__()` method is also decorated to mark new objects as "new".
+
+ {python}
+ User.mapper = mapper(User, users)
+
+ # create a new User
+ myuser = User()
+ myuser.user_name = 'jane'
+ myuser.password = 'hello123'
+
+ # create another new User
+ myuser2 = User()
+ myuser2.user_name = 'ed'
+ myuser2.password = 'lalalala'
+
+ # load a third User from the database
+ {sql}myuser3 = User.mapper.select(User.c.user_name=='fred')[0]
+ SELECT users.user_id AS users_user_id,
+ users.user_name AS users_user_name, users.password AS users_password
+ FROM users WHERE users.user_name = :users_user_name
+ {'users_user_name': 'fred'}
+
+ myuser3.user_name = 'fredjones'
+
+ # save all changes
+ {sql}objectstore.commit()
+ UPDATE users SET user_name=:user_name
+ WHERE users.user_id =:users_user_id
+ [{'users_user_id': 1, 'user_name': 'fredjones'}]
+ INSERT INTO users (user_name, password) VALUES (:user_name, :password)
+ {'password': 'hello123', 'user_name': 'jane'}
+ INSERT INTO users (user_name, password) VALUES (:user_name, :password)
+ {'password': 'lalalala', 'user_name': 'ed'}
+
+In the examples above, we defined a User class with basically no properties or methods. Theres no particular reason it has to be this way, the class can explicitly set up whatever properties it wants, whether or not they will be managed by the mapper. It can also specify a constructor, with the restriction that the constructor is able to function with no arguments being passed to it (this restriction can be lifted with some extra parameters to the mapper; more on that later):
+
+ {python}
+ class User(object):
+ def __init__(self, user_name = None, password = None):
+ self.user_id = None
+ self.user_name = user_name
+ self.password = password
+ def get_name(self):
+ return self.user_name
+ def __repr__(self):
+ return "User id %s name %s password %s" % (repr(self.user_id),
+ repr(self.user_name), repr(self.password))
+ User.mapper = mapper(User, users)
+
+ u = User('john', 'foo')
+ {sql}objectstore.commit()
+ INSERT INTO users (user_name, password) VALUES (:user_name, :password)
+ {'password': 'foo', 'user_name': 'john'}
+
+ >>> u
+ User id 1 name 'john' password 'foo'
+
+Recent versions of SQLAlchemy will only put modified object attributes columns into the UPDATE statements generated upon commit. This is to conserve database traffic and also to successfully interact with a "deferred" attribute, which is a mapped object attribute against the mapper's primary table that isnt loaded until referenced by the application.
+
+### Defining and Using Relationships {@name=relations}
+
+So that covers how to map the columns in a table to an object, how to load objects, create new ones, and save changes. The next step is how to define an object's relationships to other database-persisted objects. This is done via the `relation` function provided by the mapper module. So with our User class, lets also define the User has having one or more mailing addresses. First, the table metadata:
+
+ {python}
+ from sqlalchemy import *
+ engine = create_engine('sqlite://filename=mydb')
+
+ # define user table
+ users = Table('users', engine,
+ Column('user_id', Integer, primary_key=True),
+ Column('user_name', String(16)),
+ Column('password', String(20))
+ )
+
+ # define user address table
+ addresses = Table('addresses', engine,
+ Column('address_id', Integer, primary_key=True),
+ Column('user_id', Integer, ForeignKey("users.user_id")),
+ Column('street', String(100)),
+ Column('city', String(80)),
+ Column('state', String(2)),
+ Column('zip', String(10))
+ )
+
+Of importance here is the addresses table's definition of a *foreign key* relationship to the users table, relating the user_id column into a parent-child relationship. When a Mapper wants to indicate a relation of one object to another, this ForeignKey object is the default method by which the relationship is determined (although if you didn't define ForeignKeys, or you want to specify explicit relationship columns, that is available as well).
+
+So then lets define two classes, the familiar User class, as well as an Address class:
+
+ {python}
+ class User(object):
+ def __init__(self, user_name = None, password = None):
+ self.user_name = user_name
+ self.password = password
+
+ class Address(object):
+ def __init__(self, street=None, city=None, state=None, zip=None):
+ self.street = street
+ self.city = city
+ self.state = state
+ self.zip = zip
+
+And then a Mapper that will define a relationship of the User and the Address classes to each other as well as their table metadata. We will add an additional mapper keyword argument `properties` which is a dictionary relating the name of an object property to a database relationship, in this case a `relation` object against a newly defined mapper for the Address class:
+
+ {python}
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(mapper(Address, addresses))
+ }
+ )
+
+Lets do some operations with these classes and see what happens:
+
+ {python}
+ u = User('jane', 'hihilala')
+ u.addresses.append(Address('123 anywhere street', 'big city', 'UT', '76543'))
+ u.addresses.append(Address('1 Park Place', 'some other city', 'OK', '83923'))
+
+ objectstore.commit()
+ {opensql}INSERT INTO users (user_name, password) VALUES (:user_name, :password)
+ {'password': 'hihilala', 'user_name': 'jane'}
+ INSERT INTO addresses (user_id, street, city, state, zip) VALUES (:user_id, :street, :city, :state, :zip)
+ {'city': 'big city', 'state': 'UT', 'street': '123 anywhere street', 'user_id':1, 'zip': '76543'}
+ INSERT INTO addresses (user_id, street, city, state, zip) VALUES (:user_id, :street, :city, :state, :zip)
+ {'city': 'some other city', 'state': 'OK', 'street': '1 Park Place', 'user_id':1, 'zip': '83923'}
+
+A lot just happened there! The Mapper object figured out how to relate rows in the addresses table to the users table, and also upon commit had to determine the proper order in which to insert rows. After the insert, all the User and Address objects have all their new primary and foreign keys populated.
+
+Also notice that when we created a Mapper on the User class which defined an 'addresses' relation, the newly created User instance magically had an "addresses" attribute which behaved like a list. This list is in reality a property accessor function, which returns an instance of `sqlalchemy.util.HistoryArraySet`, which fulfills the full set of Python list accessors, but maintains a *unique* set of objects (based on their in-memory identity), and also tracks additions and deletions to the list:
+
+ {python}
+ del u.addresses[1]
+ u.addresses.append(Address('27 New Place', 'Houston', 'TX', '34839'))
+
+ objectstore.commit()
+
+ {opensql}UPDATE addresses SET user_id=:user_id
+ WHERE addresses.address_id = :addresses_address_id
+ [{'user_id': None, 'addresses_address_id': 2}]
+ INSERT INTO addresses (user_id, street, city, state, zip)
+ VALUES (:user_id, :street, :city, :state, :zip)
+ {'city': 'Houston', 'state': 'TX', 'street': '27 New Place', 'user_id': 1, 'zip': '34839'}
+
+#### Useful Feature: Private Relations {@name=private}
+
+So our one address that was removed from the list, was updated to have a user_id of `None`, and a new address object was inserted to correspond to the new Address added to the User. But now, theres a mailing address with no user_id floating around in the database of no use to anyone. How can we avoid this ? This is acheived by using the `private=True` parameter of `relation`:
+
+ {python}
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(mapper(Address, addresses), private=True)
+ }
+ )
+ del u.addresses[1]
+ u.addresses.append(Address('27 New Place', 'Houston', 'TX', '34839'))
+
+ objectstore.commit()
+ {opensql}INSERT INTO addresses (user_id, street, city, state, zip)
+ VALUES (:user_id, :street, :city, :state, :zip)
+ {'city': 'Houston', 'state': 'TX', 'street': '27 New Place', 'user_id': 1, 'zip': '34839'}
+ DELETE FROM addresses WHERE addresses.address_id = :address_id
+ [{'address_id': 2}]
+
+In this case, with the private flag set, the element that was removed from the addresses list was also removed from the database. By specifying the `private` flag on a relation, it is indicated to the Mapper that these related objects exist only as children of the parent object, otherwise should be deleted.
+
+#### Useful Feature: Backreferences {@name=backreferences}
+
+By creating relations with the `backref` keyword, a bi-directional relationship can be created which will keep both ends of the relationship updated automatically, even without any database queries being executed. Below, the User mapper is created with an "addresses" property, and the corresponding Address mapper receives a "backreference" to the User object via the property name "user":
+
+ {python}
+ Address.mapper = mapper(Address, addresses)
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(Address.mapper, backref='user')
+ }
+ )
+
+ u = User('fred', 'hi')
+ a1 = Address('123 anywhere street', 'big city', 'UT', '76543')
+ a2 = Address('1 Park Place', 'some other city', 'OK', '83923')
+
+ # append a1 to u
+ u.addresses.append(a1)
+
+ # attach u to a2
+ a2.user = u
+
+ # the bi-directional relation is maintained
+ >>> u.addresses == [a1, a2]
+ True
+ >>> a1.user is user and a2.user is user
+ True
+
+The backreference feature also works with many-to-many relationships, which are described later. When creating a backreference, a corresponding property is placed on the child mapper. The default arguments to this property can be overridden using the `backref()` function:
+
+ {python}
+ Address.mapper = mapper(Address, addresses)
+
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(Address.mapper,
+ backref=backref('user', lazy=False, private=True))
+ }
+ )
+
+#### Creating Relationships Automatically with cascade_mappers {@name=cascade}
+
+The mapper package has a helper function `cascade_mappers()` which can simplify the task of linking several mappers together. Given a list of classes and/or mappers, it identifies the foreign key relationships between the given mappers or corresponding class mappers, and creates relation() objects representing those relationships, including a backreference. Attempts to find the "secondary" table in a many-to-many relationship as well. The names of the relations are a lowercase version of the related class. In the case of one-to-many or many-to-many, the name is "pluralized", which currently is based on the English language (i.e. an 's' or 'es' added to it):
+
+ {python}
+ # create two mappers. the 'users' and 'addresses' tables have a foreign key
+ # relationship
+ mapper1 = mapper(User, users)
+ mapper2 = mapper(Address, addresses)
+
+ # cascade the two mappers together (can also specify User, Address as the arguments)
+ cascade_mappers(mapper1, mapper2)
+
+ # two new object instances
+ u = User('user1')
+ a = Address('test')
+
+ # "addresses" and "user" property are automatically added
+ u.addresses.append(a)
+ print a.user
+
+#### Selecting from Relationships: Lazy Load {@name=lazyload}
+
+We've seen how the `relation` specifier affects the saving of an object and its child items, how does it affect selecting them? By default, the relation keyword indicates that the related property should be attached a *Lazy Loader* when instances of the parent object are loaded from the database; this is just a callable function that when accessed will invoke a second SQL query to load the child objects of the parent.
+
+ {python}
+ # define a mapper
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(mapper(Address, addresses), private=True)
+ })
+
+ # select users where username is 'jane', get the first element of the list
+ # this will incur a load operation for the parent table
+ {sql}user = User.mapper.select(user_name='jane')[0]
+ SELECT users.user_id AS users_user_id,
+ users.user_name AS users_user_name, users.password AS users_password
+ FROM users WHERE users.user_name = :users_user_name ORDER BY users.oid
+ {'users_user_name': 'jane'}
+
+ # iterate through the User object's addresses. this will incur an
+ # immediate load of those child items
+ {sql}for a in user.addresses:
+ SELECT addresses.address_id AS addresses_address_id,
+ addresses.user_id AS addresses_user_id, addresses.street AS addresses_street,
+ addresses.city AS addresses_city, addresses.state AS addresses_state,
+ addresses.zip AS addresses_zip FROM addresses
+ WHERE addresses.user_id = :users_user_id ORDER BY addresses.oid
+ {'users_user_id': 1}
+ print repr(a)
+
+##### Useful Feature: Creating Joins via select_by {@name=relselectby}
+
+In mappers that have relationships, the `select_by` method and its cousins include special functionality that can be used to create joins. Just specify a key in the argument list which is not present in the primary mapper's list of properties or columns, but *is* present in the property list of one of its relationships:
+
+ {python}
+ {sql}l = User.mapper.select_by(street='123 Green Street')
+ SELECT users.user_id AS users_user_id,
+ users.user_name AS users_user_name, users.password AS users_password
+ FROM users, addresses
+ WHERE users.user_id=addresses.user_id
+ AND addresses.street=:addresses_street
+ ORDER BY users.oid
+ {'addresses_street', '123 Green Street'}
+
+The above example is shorthand for:
+
+ {python}
+ l = User.mapper.select(and_(
+ Address.c.user_id==User.c.user_id,
+ Address.c.street=='123 Green Street')
+ )
+
+##### How to Refresh the List? {@name=refreshing}
+
+Once the child list of Address objects is loaded, it is done loading for the lifetime of the object instance. Changes to the list will not be interfered with by subsequent loads, and upon commit those changes will be saved. Similarly, if a new User object is created and child Address objects added, a subsequent select operation which happens to touch upon that User instance, will also not affect the child list, since it is already loaded.
+
+The issue of when the mapper actually gets brand new objects from the database versus when it assumes the in-memory version is fine the way it is, is a subject of *transactional scope*. Described in more detail in the Unit of Work section, for now it should be noted that the total storage of all newly created and selected objects, *within the scope of the current thread*, can be reset via releasing or otherwise disregarding all current object instances, and calling:
+
+ {python}
+ objectstore.clear()
+
+This operation will clear out all currently mapped object instances, and subsequent select statements will load fresh copies from the databse.
+
+To operate upon a single object, just use the `remove` function:
+
+ {python}
+ # (this function coming soon)
+ objectstore.remove(myobject)
+
+
+#### Selecting from Relationships: Eager Load {@name=eagerload}
+
+With just a single parameter "lazy=False" specified to the relation object, the parent and child SQL queries can be joined together.
+
+ {python}
+ Address.mapper = mapper(Address, addresses)
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(Address.mapper, lazy=False)
+ }
+ )
+
+ {sql}user = User.mapper.get_by(user_name='jane')
+ SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
+ users.password AS users_password,
+ addresses.address_id AS addresses_address_id, addresses.user_id AS addresses_user_id,
+ addresses.street AS addresses_street, addresses.city AS addresses_city,
+ addresses.state AS addresses_state, addresses.zip AS addresses_zip
+ FROM users LEFT OUTER JOIN addresses ON users.user_id = addresses.user_id
+ WHERE users.user_name = :users_user_name ORDER BY users.oid, addresses.oid
+ {'users_user_name': 'jane'}
+
+ for a in user.addresses:
+ print repr(a)
+
+
+Above, a pretty ambitious query is generated just by specifying that the User should be loaded with its child Addresses in one query. When the mapper processes the results, it uses an *Identity Map* to keep track of objects that were already loaded, based on their primary key identity. Through this method, the redundant rows produced by the join are organized into the distinct object instances they represent.
+
+The generation of this query is also immune to the effects of additional joins being specified in the original query. To use our select_by example above, joining against the "addresses" table to locate users with a certain street results in this behavior:
+
+ {python}
+ {sql}users = User.mapper.select_by(street='123 Green Street')
+ SELECT users.user_id AS users_user_id,
+ users.user_name AS users_user_name, users.password AS users_password,
+ addresses.address_id AS addresses_address_id,
+ addresses.user_id AS addresses_user_id, addresses.street AS addresses_street,
+ addresses.city AS addresses_city, addresses.state AS addresses_state,
+ addresses.zip AS addresses_zip
+ FROM addresses AS addresses_417c,
+ users LEFT OUTER JOIN addresses ON users.user_id = addresses.user_id
+ WHERE addresses_417c.street = :addresses_street
+ AND users.user_id = addresses_417c.user_id
+ ORDER BY users.oid, addresses.oid
+ {'addresses_street': '123 Green Street'}
+
+The join implied by passing the "street" parameter is converted into an "aliasized" clause by the eager loader, so that it does not conflict with the join used to eager load the child address objects.
+
+#### Switching Lazy/Eager, No Load {@name=options}
+
+The `options` method of mapper provides an easy way to get alternate forms of a mapper from an original one. The most common use of this feature is to change the "eager/lazy" loading behavior of a particular mapper, via the functions `eagerload()`, `lazyload()` and `noload()`:
+
+ {python}
+ # user mapper with lazy addresses
+ User.mapper = mapper(User, users, properties = {
+ 'addresses' : relation(mapper(Address, addresses))
+ }
+ )
+
+ # make an eager loader
+ eagermapper = User.mapper.options(eagerload('addresses'))
+ u = eagermapper.select()
+
+ # make another mapper that wont load the addresses at all
+ plainmapper = User.mapper.options(noload('addresses'))
+
+ # multiple options can be specified
+ mymapper = oldmapper.options(lazyload('tracker'), noload('streets'), eagerload('members'))
+
+ # to specify a relation on a relation, separate the property names by a "."
+ mymapper = oldmapper.options(eagerload('orders.items'))
+
+### One to One/Many to One {@name=onetoone}
+
+The above examples focused on the "one-to-many" relationship. To do other forms of relationship is easy, as the `relation` function can usually figure out what you want:
+
+ {python}
+ # a table to store a user's preferences for a site
+ prefs = Table('user_prefs', engine,
+ Column('pref_id', Integer, primary_key = True),
+ Column('stylename', String(20)),
+ Column('save_password', Boolean, nullable = False),
+ Column('timezone', CHAR(3), nullable = False)
+ )
+
+ # user table gets 'preference_id' column added
+ users = Table('users', engine,
+ Column('user_id', Integer, primary_key = True),
+ Column('user_name', String(16), nullable = False),
+ Column('password', String(20), nullable = False),
+ Column('preference_id', Integer, ForeignKey("prefs.pref_id"))
+ )
+
+ # class definition for preferences
+ class UserPrefs(object):
+ pass
+ UserPrefs.mapper = mapper(UserPrefs, prefs)
+
+ # address mapper
+ Address.mapper = mapper(Address, addresses)
+
+ # make a new mapper referencing everything.
+ m = mapper(User, users, properties = dict(
+ addresses = relation(Address.mapper, lazy=True, private=True),
+ preferences = relation(UserPrefs.mapper, lazy=False, private=True),
+ ))
+
+ # select
+ {sql}user = m.get_by(user_name='fred')
+ SELECT users.user_id AS users_user_id, users.user_name AS users_user_name,
+ users.password AS users_password, users.preference_id AS users_preference_id,
+ user_prefs.pref_id AS user_prefs_pref_id, user_prefs.stylename AS user_prefs_stylename,
+ user_prefs.save_password AS user_prefs_save_password, user_prefs.timezone AS user_prefs_timezone
+ FROM users LEFT OUTER JOIN user_prefs ON user_prefs.pref_id = users.preference_id
+ WHERE users.user_name = :users_user_name ORDER BY users.oid, user_prefs.oid
+ {'users_user_name': 'fred'}
+
+ save_password = user.preferences.save_password
+
+ # modify
+ user.preferences.stylename = 'bluesteel'
+ {sql}user.addresses.append(Address('freddy@hi.org'))
+ SELECT email_addresses.address_id AS email_addresses_address_id,
+ email_addresses.user_id AS email_addresses_user_id,
+ email_addresses.email_address AS email_addresses_email_address
+ FROM email_addresses
+ WHERE email_addresses.user_id = :users_user_id
+ ORDER BY email_addresses.oid, email_addresses.oid
+ {'users_user_id': 1}
+
+ # commit
+ {sql}objectstore.commit()
+ UPDATE user_prefs SET stylename=:stylename
+ WHERE user_prefs.pref_id = :pref_id
+ [{'stylename': 'bluesteel', 'pref_id': 1}]
+ INSERT INTO email_addresses (address_id, user_id, email_address)
+ VALUES (:address_id, :user_id, :email_address)
+ {'email_address': 'freddy@hi.org', 'address_id': None, 'user_id': 1}
+
+### Many to Many {@name=manytomany}
+
+The `relation` function handles a basic many-to-many relationship when you specify the association table:
+
+ {python}
+ articles = Table('articles', engine,
+ Column('article_id', Integer, primary_key = True),
+ Column('headline', String(150), key='headline'),
+ Column('body', TEXT, key='body'),
+ )
+
+ keywords = Table('keywords', engine,
+ Column('keyword_id', Integer, primary_key = True),
+ Column('keyword_name', String(50))
+ )
+
+ itemkeywords = Table('article_keywords', engine,
+ Column('article_id', Integer, ForeignKey("articles.article_id")),
+ Column('keyword_id', Integer, ForeignKey("keywords.keyword_id"))
+ )
+
+ # class definitions
+ class Keyword(object):
+ def __init__(self, name = None):
+ self.keyword_name = name
+
+ class Article(object):
+ pass
+
+ # define a mapper that does many-to-many on the 'itemkeywords' association
+ # table
+ Article.mapper = mapper(Article, articles, properties = dict(
+ keywords = relation(mapper(Keyword, keywords), itemkeywords, lazy=False)
+ )
+ )
+
+ article = Article()
+ article.headline = 'a headline'
+ article.body = 'this is the body'
+ article.keywords.append(Keyword('politics'))
+ article.keywords.append(Keyword('entertainment'))
+ {sql}objectstore.commit()
+ INSERT INTO keywords (name) VALUES (:name)
+ {'name': 'politics'}
+ INSERT INTO keywords (name) VALUES (:name)
+ {'name': 'entertainment'}
+ INSERT INTO articles (article_headline, article_body) VALUES (:article_headline, :article_body)
+ {'article_body': 'this is the body', 'article_headline': 'a headline'}
+ INSERT INTO article_keywords (article_id, keyword_id) VALUES (:article_id, :keyword_id)
+ [{'keyword_id': 1, 'article_id': 1}, {'keyword_id': 2, 'article_id': 1}]
+
+ # select articles based on a keyword. select_by will handle the extra joins.
+ {sql}articles = Article.mapper.select_by(keyword_name='politics')
+ SELECT articles.article_id AS articles_article_id,
+ articles.article_headline AS articles_article_headline,
+ articles.article_body AS articles_article_body,
+ keywords.keyword_id AS keywords_keyword_id,
+ keywords.keyword_name AS keywords_keyword_name
+ FROM keywords AS keywords_f008,
+ article_keywords AS article_keywords_dbf0,
+ articles LEFT OUTER JOIN article_keywords ON
+ articles.article_id = article_keywords.article_id
+ LEFT OUTER JOIN keywords ON
+ keywords.keyword_id = article_keywords.keyword_id
+ WHERE (keywords_f008.keyword_name = :keywords_keyword_name
+ AND articles.article_id = article_keywords_dbf0.article_id)
+ AND keywords_f008.keyword_id = article_keywords_dbf0.keyword_id
+ ORDER BY articles.oid, article_keywords.oid
+ {'keywords_keyword_name': 'politics'}
+
+ # modify
+ a = articles[0]
+ del a.keywords[:]
+ a.keywords.append(Keyword('topstories'))
+ a.keywords.append(Keyword('government'))
+
+ # commit. individual INSERT/DELETE operations will take place only for the list
+ # elements that changed.
+ {sql}objectstore.commit()
+ INSERT INTO keywords (name) VALUES (:name)
+ {'name': 'topstories'}
+ INSERT INTO keywords (name) VALUES (:name)
+ {'name': 'government'}
+ DELETE FROM article_keywords
+ WHERE article_keywords.article_id = :article_id
+ AND article_keywords.keyword_id = :keyword_id
+ [{'keyword_id': 1, 'article_id': 1}, {'keyword_id': 2, 'article_id': 1}]
+ INSERT INTO article_keywords (article_id, keyword_id) VALUES (:article_id, :keyword_id)
+ [{'keyword_id': 3, 'article_id': 1}, {'keyword_id': 4, 'article_id': 1}]
+
+### Association Object {@name=association}
+
+Many to Many can also be done with an association object, that adds additional information about how two items are related. This association object is set up in basically the same way as any other mapped object. However, since an association table typically has no primary key columns, you have to tell the mapper what columns will compose its "primary key", which are the two (or more) columns involved in the association. Also, the relation function needs an additional hint as to the fact that this mapped object is an association object, via the "association" argument which points to the class or mapper representing the other side of the association.
+
+ {python}
+ # add "attached_by" column which will reference the user who attached this keyword
+ itemkeywords = Table('article_keywords', engine,
+ Column('article_id', Integer, ForeignKey("articles.article_id")),
+ Column('keyword_id', Integer, ForeignKey("keywords.keyword_id")),
+ Column('attached_by', Integer, ForeignKey("users.user_id"))
+ )
+
+ # define an association class
+ class KeywordAssociation(object):
+ pass
+
+ # mapper for KeywordAssociation
+ # specify "primary key" columns manually
+ KeywordAssociation.mapper = mapper(KeywordAssociation, itemkeywords,
+ primary_key = [itemkeywords.c.article_id, itemkeywords.c.keyword_id],
+ properties={
+ 'keyword' : relation(Keyword, lazy = False), # uses primary Keyword mapper
+ 'user' : relation(User, lazy = True) # uses primary User mapper
+ }
+ )
+
+ # mappers for Users, Keywords
+ User.mapper = mapper(User, users)
+ Keyword.mapper = mapper(Keyword, keywords)
+
+ # define the mapper.
+ m = mapper(Article, articles, properties={
+ 'keywords':relation(KeywordAssociation.mapper, lazy=False, association=Keyword)
+ }
+ )
+
+ # bonus step - well, we do want to load the users in one shot,
+ # so modify the mapper via an option.
+ # this returns a new mapper with the option switched on.
+ m2 = mapper.options(eagerload('keywords.user'))
+
+ # select by keyword again
+ {sql}alist = m2.select_by(keyword_name='jacks_stories')
+ SELECT articles.article_id AS articles_article_id,
+ articles.article_headline AS articles_article_headline,
+ articles.article_body AS articles_article_body,
+ article_keywords.article_id AS article_keywords_article_id,
+ article_keywords.keyword_id AS article_keywords_keyword_id,
+ article_keywords.attached_by AS article_keywords_attached_by,
+ users.user_id AS users_user_id, users.user_name AS users_user_name,
+ users.password AS users_password, users.preference_id AS users_preference_id,
+ keywords.keyword_id AS keywords_keyword_id, keywords.name AS keywords_name
+ FROM article_keywords article_keywords_3a64, keywords keywords_11b7,
+ articles LEFT OUTER JOIN article_keywords ON articles.article_id = article_keywords.article_id
+ LEFT OUTER JOIN users ON users.user_id = article_keywords.attached_by
+ LEFT OUTER JOIN keywords ON keywords.keyword_id = article_keywords.keyword_id
+ WHERE keywords_11b7.keyword_id = article_keywords_3a64.keyword_id
+ AND article_keywords_3a64.article_id = articles.article_id
+ AND keywords_11b7.name = :keywords_name
+ ORDER BY articles.oid, article_keywords.oid, users.oid, keywords.oid
+ {'keywords_name': 'jacks_stories'}
+
+ # user is available
+ for a in alist:
+ for k in a.keywords:
+ if k.keyword.name == 'jacks_stories':
+ print k.user.user_name
-Database Engines
+Database Engines {@name=dbengine}
================
A database engine is a subclass of `sqlalchemy.engine.SQLEngine`, and is the starting point for where SQLAlchemy provides a layer of abstraction on top of the various DBAPI2 database modules. It serves as an abstract factory for database-specific implementation objects as well as a layer of abstraction over the most essential tasks of a database connection, including connecting, executing queries, returning result sets, and managing transactions.
### Database Engine Options {@name=options}
-The remaining arguments to `create_engine` are keyword arguments that are passed to the specific subclass of `sqlalchemy.engine.SQLEngine` being used, as well as the underlying `sqlalchemy.pool.Pool` instance. All of the options described in the previous section [pooling_configuration][pooling_configuration] can be specified, as well as engine-specific options:
+The remaining arguments to `create_engine` are keyword arguments that are passed to the specific subclass of `sqlalchemy.engine.SQLEngine` being used, as well as the underlying `sqlalchemy.pool.Pool` instance. All of the options described in the previous section [pooling_configuration](rel:pooling_configuration) can be specified, as well as engine-specific options:
* pool=None : an instance of `sqlalchemy.pool.Pool` to be used as the underlying source for connections, overriding the engine's connect arguments (pooling is described in the previous section). If None, a default Pool (QueuePool or SingletonThreadPool as appropriate) will be created using the engine's connect arguments.
+++ /dev/null
-<%flags>inherit='document_base.myt'</%flags>
-<%attr>title='Database Meta Data'</%attr>
-<&|doclib.myt:item, name="metadata", description="Database Meta Data" &>
- <&|doclib.myt:item, name="tables", description="Describing Tables with MetaData" &>
- <p>The core of SQLAlchemy's query and object mapping operations is table metadata, which are Python objects that describe tables. Metadata objects can be created by explicitly naming the table and all its properties, using the Table, Column, ForeignKey, and Sequence objects imported from <span class="codeline">sqlalchemy.schema</span>, and a database engine constructed as described in the previous section, or they can be automatically pulled from an existing database schema. First, the explicit version: </p>
- <&|formatting.myt:code&>
- from sqlalchemy import *
- engine = create_engine('sqlite', {'filename':':memory:'}, **opts)
-
- users = Table('users', engine,
- Column('user_id', Integer, primary_key = True),
- Column('user_name', String(16), nullable = False),
- Column('email_address', String(60), key='email'),
- Column('password', String(20), nullable = False)
- )
-
- user_prefs = Table('user_prefs', engine,
- Column('pref_id', Integer, primary_key=True),
- Column('user_id', Integer, ForeignKey("users.user_id"), nullable=False),
- Column('pref_name', String(40), nullable=False),
- Column('pref_value', String(100))
- )
- </&>
- <p>The specific datatypes, such as Integer, String, etc. are defined in <&formatting.myt:link, path="types", text="sqlalchemy.types"&> and are automatically pulled in when you import * from <span class="codeline">sqlalchemy</span>. Note that for Column objects, an altername name can be specified via the "key" parameter; if this parameter is given, then all programmatic references to this Column object will be based on its key, instead of its actual column name.</p>
-
- <p>Once constructed, the Table object provides a clean interface to the table's properties as well as that of its columns:
-
- <&|formatting.myt:code&>
- employees = Table('employees', engine,
- Column('employee_id', Integer, primary_key=True),
- Column('employee_name', String(60), nullable=False, key='name'),
- Column('employee_dept', Integer, ForeignKey("departments.department_id"))
- )
-
- # access the column "EMPLOYEE_ID":
- employees.columns.employee_id
-
- # or just
- employees.c.employee_id
-
- # via string
- employees.c['employee_id']
-
- # iterate through all columns
- for c in employees.c:
- # ...
-
- # get the table's primary key columns
- for primary_key in employees.primary_key:
- # ...
-
- # get the table's foreign key objects:
- for fkey in employees.foreign_keys:
- # ...
-
- # access the table's SQLEngine object:
- employees.engine
-
- # access a column's name, type, nullable, primary key, foreign key
- employees.c.employee_id.name
- employees.c.employee_id.type
- employees.c.employee_id.nullable
- employees.c.employee_id.primary_key
- employees.c.employee_dept.foreign_key
-
- # get the "key" of a column, which defaults to its name, but can
- # be any user-defined string:
- employees.c.name.key
-
- # access a column's table:
- employees.c.employee_id.table is employees
- >>> True
-
- # get the table related by a foreign key
- fcolumn = employees.c.employee_dept.foreign_key.column.table
- </&>
- </p>
-
- <p>Metadata objects can also be <b>reflected</b> from tables that already exist in the database. Reflection means based on a table name, the names, datatypes, and attributes of all columns, including foreign keys, will be loaded automatically. This feature is supported by all database engines:</p>
- <&|formatting.myt:code&>
- >>> messages = Table('messages', engine, autoload = True)
- >>> [c.name for c in messages.columns]
- ['message_id', 'message_name', 'date']
- </&>
-
- <p>
- Note that if a reflected table has a foreign key referencing another table, then the metadata for the related table will be loaded as well, even if it has not been defined by the application:
- </p>
- <&|formatting.myt:code&>
- >>> shopping_cart_items = Table('shopping_cart_items', engine, autoload = True)
- >>> print shopping_cart_items.c.cart_id.table.name
- shopping_carts
- </&>
- <p>To get direct access to 'shopping_carts', simply instantiate it via the Table constructor. You'll get the same instance of the shopping cart Table as the one that is attached to shopping_cart_items:
- <&|formatting.myt:code&>
- >>> shopping_carts = Table('shopping_carts', engine)
- >>> shopping_carts is shopping_cart_items.c.cart_id.table.name
- True
- </&>
- <p>This works because when the Table constructor is called for a particular name and database engine, if the table has already been created then the instance returned will be the same as the original. This is a <b>singleton</b> constructor:</p>
- <&|formatting.myt:code&>
- >>> news_articles = Table('news', engine,
- ... Column('article_id', Integer, primary_key = True),
- ... Column('url', String(250), nullable = False)
- ... )
- >>> othertable = Table('news', engine)
- >>> othertable is news_articles
- True
- </&>
- </&>
- <&|doclib.myt:item, name="creating", description="Creating and Dropping Database Tables" &>
- <p>Creating and dropping is easy, just use the <span class="codeline">create()</span> and <span class="codeline">drop()</span> methods:
- <&|formatting.myt:code&>
- <&formatting.myt:poplink&>employees = Table('employees', engine,
- Column('employee_id', Integer, primary_key=True),
- Column('employee_name', String(60), nullable=False, key='name'),
- Column('employee_dept', Integer, ForeignKey("departments.department_id"))
- )
- employees.create() <&|formatting.myt:codepopper, link="sql" &>
-CREATE TABLE employees(
- employee_id SERIAL NOT NULL PRIMARY KEY,
- employee_name VARCHAR(60) NOT NULL,
- employee_dept INTEGER REFERENCES departments(department_id)
-)
-{} </&>
-
- <&formatting.myt:poplink&>employees.drop() <&|formatting.myt:codepopper, link="sql" &>
-DROP TABLE employees
-{} </&>
- </&>
- </&>
-
-
- <&|doclib.myt:item, name="defaults", description="Column Defaults and OnUpdates" &>
- <p>SQLAlchemy includes flexible constructs in which to create default values for columns upon the insertion of rows, as well as upon update. These defaults can take several forms: a constant, a Python callable to be pre-executed before the SQL is executed, a SQL expression or function to be pre-executed before the SQL is executed, a pre-executed Sequence (for databases that support sequences), or a "passive" default, which is a default function triggered by the database itself upon insert, the value of which can then be post-fetched by the engine, provided the row provides a primary key in which to call upon.</p>
- <&|doclib.myt:item, name="oninsert", description="Pre-Executed Insert Defaults" &>
- <p>A basic default is most easily specified by the "default" keyword argument to Column. This defines a value, function, or SQL expression that will be pre-executed to produce the new value, before the row is inserted:</p>
- <&|formatting.myt:code&>
- # a function to create primary key ids
- i = 0
- def mydefault():
- global i
- i += 1
- return i
-
- t = Table("mytable", db,
- # function-based default
- Column('id', Integer, primary_key=True, default=mydefault),
-
- # a scalar default
- Column('key', String(10), default="default")
- )
- </&>
- <p>The "default" keyword can also take SQL expressions, including select statements or direct function calls:</p>
- <&|formatting.myt:code&>
- t = Table("mytable", db,
- Column('id', Integer, primary_key=True),
-
- # define 'create_date' to default to now()
- Column('create_date', DateTime, default=func.now()),
-
- # define 'key' to pull its default from the 'keyvalues' table
- Column('key', String(20), default=keyvalues.select(keyvalues.c.type='type1', limit=1))
- )
- </&>
- <p>The "default" keyword argument is shorthand for using a ColumnDefault object in a column definition. This syntax is optional, but is required for other types of defaults, futher described below:</p>
- <&|formatting.myt:code&>
- Column('mycolumn', String(30), ColumnDefault(func.get_data()))
- </&>
- </&>
-
- <&|doclib.myt:item, name="onupdate", description="Pre-Executed OnUpdate Defaults" &>
- <p>Similar to an on-insert default is an on-update default, which is most easily specified by the "onupdate" keyword to Column, which also can be a constant, plain Python function or SQL expression:</p>
- <&|formatting.myt:code&>
- t = Table("mytable", db,
- Column('id', Integer, primary_key=True),
-
- # define 'last_updated' to be populated with current_timestamp (the ANSI-SQL version of now())
- Column('last_updated', DateTime, onupdate=func.current_timestamp()),
- )
- </&>
- <p>To use an explicit ColumnDefault object to specify an on-update, use the "for_update" keyword argument:</p>
- <&|formatting.myt:code&>
- Column('mycolumn', String(30), ColumnDefault(func.get_data(), for_update=True))
- </&>
- </&>
-
- <&|doclib.myt:item, name="passive", description="Inline Default Execution: PassiveDefault" &>
- <p>A PassiveDefault indicates a column default or on-update value that is executed automatically by the database. This construct is used to specify a SQL function that will be specified as "DEFAULT" when creating tables, and also to indicate the presence of new data that is available to be "post-fetched" after an insert or update execution.</p>
- <&|formatting.myt:code&>
- t = Table('test', e,
- Column('mycolumn', DateTime, PassiveDefault("sysdate"))
- )
- </&>
- <p>A create call for the above table will produce:</p>
- <&|formatting.myt:code&>
- CREATE TABLE test (
- mycolumn datetime default sysdate
- )
- </&>
- <p>PassiveDefaults also send a message to the SQLEngine that data is available after update or insert. The object-relational mapper system uses this information to post-fetch rows after insert or update, so that instances can be refreshed with the new data. Below is a simplified version:</p>
- <&|formatting.myt:code&>
- # table with passive defaults
- mytable = Table('mytable', engine,
- Column('my_id', Integer, primary_key=True),
-
- # an on-insert database-side default
- Column('data1', Integer, PassiveDefault("d1_func")),
-
- # an on-update database-side default
- Column('data2', Integer, PassiveDefault("d2_func", for_update=True))
- )
- # insert a row
- mytable.insert().execute(name='fred')
-
- # ask the engine: were there defaults fired off on that row ?
- if table.engine.lastrow_has_defaults():
- # postfetch the row based on primary key.
- # this only works for a table with primary key columns defined
- primary_key = table.engine.last_inserted_ids()
- row = table.select(table.c.id == primary_key[0])
- </&>
- <p>When Tables are reflected from the database using <code>autoload=True</code>, any DEFAULT values set on the columns will be reflected in the Table object as PassiveDefault instances.</p>
-
- <&|doclib.myt:item, name="postgres", description="The Catch: Postgres Primary Key Defaults always Pre-Execute" &>
- <p>Current Postgres support does not rely upon OID's to determine the identity of a row. This is because the usage of OIDs has been deprecated with Postgres and they are disabled by default for table creates as of PG version 8. Pyscopg2's "cursor.lastrowid" function only returns OIDs. Therefore, when inserting a new row which has passive defaults set on the primary key columns, the default function is <b>still pre-executed</b> since SQLAlchemy would otherwise have no way of retrieving the row just inserted.</p>
- </&>
- </&>
- <&|doclib.myt:item, name="sequences", description="Defining Sequences" &>
- <P>A table with a sequence looks like:</p>
- <&|formatting.myt:code&>
- table = Table("cartitems", db,
- Column("cart_id", Integer, Sequence('cart_id_seq'), primary_key=True),
- Column("description", String(40)),
- Column("createdate", DateTime())
- )
- </&>
- <p>The Sequence is used with Postgres or Oracle to indicate the name of a Sequence that will be used to create default values for a column. When a table with a Sequence on a column is created by SQLAlchemy, the Sequence object is also created. Similarly, the Sequence is dropped when the table is dropped. Sequences are typically used with primary key columns. When using Postgres, if an integer primary key column defines no explicit Sequence or other default method, SQLAlchemy will create the column with the SERIAL keyword, and will pre-execute a sequence named "tablename_columnname_seq" in order to retrieve new primary key values. Oracle, which has no "auto-increment" keyword, requires that a Sequence be created for a table if automatic primary key generation is desired. Note that for all databases, primary key values can always be explicitly stated within the bind parameters for any insert statement as well, removing the need for any kind of default generation function.</p>
-
- <p>A Sequence object can be defined on a Table that is then used for a non-sequence-supporting database. In that case, the Sequence object is simply ignored. Note that a Sequence object is <b>entirely optional for all databases except Oracle</b>, as other databases offer options for auto-creating primary key values, such as AUTOINCREMENT, SERIAL, etc. SQLAlchemy will use these default methods for creating primary key values if no Sequence is present on the table metadata.</p>
-
- <p>A sequence can also be specified with <span class="codeline">optional=True</span> which indicates the Sequence should only be used on a database that requires an explicit sequence, and not those that supply some other method of providing integer values. At the moment, it essentially means "use this sequence only with Oracle and not Postgres".</p>
- </&>
- </&>
- <&|doclib.myt:item, name="indexes", description="Defining Indexes" &>
- <p>Indexes can be defined on table columns, including named indexes, non-unique or unique, multiple column. Indexes are included along with table create and drop statements. They are not used for any kind of run-time constraint checking; SQLAlchemy leaves that job to the expert on constraint checking, the database itself.</p>
- <&|formatting.myt:code&>
- mytable = Table('mytable', engine,
-
- # define a unique index
- Column('col1', Integer, unique=True),
-
- # define a unique index with a specific name
- Column('col2', Integer, unique='mytab_idx_1'),
-
- # define a non-unique index
- Column('col3', Integer, index=True),
-
- # define a non-unique index with a specific name
- Column('col4', Integer, index='mytab_idx_2'),
-
- # pass the same name to multiple columns to add them to the same index
- Column('col5', Integer, index='mytab_idx_2'),
-
- Column('col6', Integer),
- Column('col7', Integer)
- )
-
- # create the table. all the indexes will be created along with it.
- mytable.create()
-
- # indexes can also be specified standalone
- i = Index('mytab_idx_3', mytable.c.col6, mytable.c.col7, unique=False)
-
- # which can then be created separately (will also get created with table creates)
- i.create()
-
- </&>
- </&>
- <&|doclib.myt:item, name="adapting", description="Adapting Tables to Alternate Engines" &>
- <p>A Table object created against a specific engine can be re-created against a new engine using the <span class="codeline">toengine</span> method:</p>
-
- <&|formatting.myt:code&>
- # create two engines
- sqlite_engine = create_engine('sqlite', {'filename':'querytest.db'})
- postgres_engine = create_engine('postgres',
- {'database':'test',
- 'host':'127.0.0.1', 'user':'scott', 'password':'tiger'})
-
- # load 'users' from the sqlite engine
- users = Table('users', sqlite_engine, autoload=True)
-
- # create the same Table object for the other engine
- pg_users = users.toengine(postgres_engine)
- </&>
-
- <p>Also available is the "database neutral" ansisql engine:</p>
- <&|formatting.myt:code&>
- import sqlalchemy.ansisql as ansisql
- generic_engine = ansisql.engine()
-
- users = Table('users', generic_engine,
- Column('user_id', Integer),
- Column('user_name', String(50))
- )
- </&>
- <p>Flexible "multi-engined" tables can also be achieved via the proxy engine, described in the section <&formatting.myt:link, path="dbengine_proxy"&>.</p>
-
- <&|doclib.myt:item, name="primitives", description="Non-engine primitives: TableClause/ColumnClause" &>
-
- <p>TableClause and ColumnClause are "primitive" versions of the Table and Column objects which dont use engines at all; applications that just want to generate SQL strings but not directly communicate with a database can use TableClause and ColumnClause objects (accessed via 'table' and 'column'), which are non-singleton and serve as the "lexical" base class of Table and Column:</p>
- <&|formatting.myt:code&>
- tab1 = table('table1',
- column('id'),
- column('name'))
-
- tab2 = table('table2',
- column('id'),
- column('email'))
-
- tab1.select(tab1.c.name == 'foo')
- </&>
-
- <p>TableClause and ColumnClause are strictly lexical. This means they are fully supported within the full range of SQL statement generation, but they don't support schema concepts like creates, drops, primary keys, defaults, nullable status, indexes, or foreign keys.</p>
- </&>
- </&>
-
-
-</&>
--- /dev/null
+Database Meta Data {@name=metadata}
+==================
+
+### Describing Tables with MetaData {@name=tables}
+
+The core of SQLAlchemy's query and object mapping operations is table metadata, which are Python objects that describe tables. Metadata objects can be created by explicitly naming the table and all its properties, using the Table, Column, ForeignKey, and Sequence objects imported from `sqlalchemy.schema`, and a database engine constructed as described in the previous section, or they can be automatically pulled from an existing database schema. First, the explicit version:
+
+ {python}
+ from sqlalchemy import *
+ engine = create_engine('sqlite', {'filename':':memory:'}, **opts)
+
+ users = Table('users', engine,
+ Column('user_id', Integer, primary_key = True),
+ Column('user_name', String(16), nullable = False),
+ Column('email_address', String(60), key='email'),
+ Column('password', String(20), nullable = False)
+ )
+
+ user_prefs = Table('user_prefs', engine,
+ Column('pref_id', Integer, primary_key=True),
+ Column('user_id', Integer, ForeignKey("users.user_id"), nullable=False),
+ Column('pref_name', String(40), nullable=False),
+ Column('pref_value', String(100))
+ )
+
+The specific datatypes, such as Integer, String, etc. are defined in [types](rel:types) and are automatically pulled in when you import * from `sqlalchemy`. Note that for Column objects, an altername name can be specified via the "key" parameter; if this parameter is given, then all programmatic references to this Column object will be based on its key, instead of its actual column name.
+
+Once constructed, the Table object provides a clean interface to the table's properties as well as that of its columns:
+
+ {python}
+ employees = Table('employees', engine,
+ Column('employee_id', Integer, primary_key=True),
+ Column('employee_name', String(60), nullable=False, key='name'),
+ Column('employee_dept', Integer, ForeignKey("departments.department_id"))
+ )
+
+ # access the column "EMPLOYEE_ID":
+ employees.columns.employee_id
+
+ # or just
+ employees.c.employee_id
+
+ # via string
+ employees.c['employee_id']
+
+ # iterate through all columns
+ for c in employees.c:
+ # ...
+
+ # get the table's primary key columns
+ for primary_key in employees.primary_key:
+ # ...
+
+ # get the table's foreign key objects:
+ for fkey in employees.foreign_keys:
+ # ...
+
+ # access the table's SQLEngine object:
+ employees.engine
+
+ # access a column's name, type, nullable, primary key, foreign key
+ employees.c.employee_id.name
+ employees.c.employee_id.type
+ employees.c.employee_id.nullable
+ employees.c.employee_id.primary_key
+ employees.c.employee_dept.foreign_key
+
+ # get the "key" of a column, which defaults to its name, but can
+ # be any user-defined string:
+ employees.c.name.key
+
+ # access a column's table:
+ employees.c.employee_id.table is employees
+ >>> True
+
+ # get the table related by a foreign key
+ fcolumn = employees.c.employee_dept.foreign_key.column.table
+
+Metadata objects can also be <b>reflected</b> from tables that already exist in the database. Reflection means based on a table name, the names, datatypes, and attributes of all columns, including foreign keys, will be loaded automatically. This feature is supported by all database engines:
+
+ {python}
+ >>> messages = Table('messages', engine, autoload = True)
+ >>> [c.name for c in messages.columns]
+ ['message_id', 'message_name', 'date']
+
+Note that if a reflected table has a foreign key referencing another table, then the metadata for the related table will be loaded as well, even if it has not been defined by the application:
+
+ {python}
+ >>> shopping_cart_items = Table('shopping_cart_items', engine, autoload = True)
+ >>> print shopping_cart_items.c.cart_id.table.name
+ shopping_carts
+
+To get direct access to 'shopping_carts', simply instantiate it via the Table constructor. You'll get the same instance of the shopping cart Table as the one that is attached to shopping_cart_items:
+
+ {python}
+ >>> shopping_carts = Table('shopping_carts', engine)
+ >>> shopping_carts is shopping_cart_items.c.cart_id.table.name
+ True
+
+This works because when the Table constructor is called for a particular name and database engine, if the table has already been created then the instance returned will be the same as the original. This is a <b>singleton</b> constructor:
+
+ {python}
+ >>> news_articles = Table('news', engine,
+ ... Column('article_id', Integer, primary_key = True),
+ ... Column('url', String(250), nullable = False)
+ ... )
+ >>> othertable = Table('news', engine)
+ >>> othertable is news_articles
+ True
+
+### Creating and Dropping Database Tables {@name=creating}
+
+Creating and dropping is easy, just use the `create()` and `drop()` methods:
+
+ {python}
+ employees = Table('employees', engine,
+ Column('employee_id', Integer, primary_key=True),
+ Column('employee_name', String(60), nullable=False, key='name'),
+ Column('employee_dept', Integer, ForeignKey("departments.department_id"))
+ )
+ {sql}employees.create()
+ CREATE TABLE employees(
+ employee_id SERIAL NOT NULL PRIMARY KEY,
+ employee_name VARCHAR(60) NOT NULL,
+ employee_dept INTEGER REFERENCES departments(department_id)
+ )
+ {}
+
+ {sql}employees.drop()
+ DROP TABLE employees
+ {}
+
+### Column Defaults and OnUpdates {@name=defaults}
+
+SQLAlchemy includes flexible constructs in which to create default values for columns upon the insertion of rows, as well as upon update. These defaults can take several forms: a constant, a Python callable to be pre-executed before the SQL is executed, a SQL expression or function to be pre-executed before the SQL is executed, a pre-executed Sequence (for databases that support sequences), or a "passive" default, which is a default function triggered by the database itself upon insert, the value of which can then be post-fetched by the engine, provided the row provides a primary key in which to call upon.
+
+#### Pre-Executed Insert Defaults {@name=oninsert}
+
+A basic default is most easily specified by the "default" keyword argument to Column. This defines a value, function, or SQL expression that will be pre-executed to produce the new value, before the row is inserted:
+
+ {python}
+ # a function to create primary key ids
+ i = 0
+ def mydefault():
+ global i
+ i += 1
+ return i
+
+ t = Table("mytable", db,
+ # function-based default
+ Column('id', Integer, primary_key=True, default=mydefault),
+
+ # a scalar default
+ Column('key', String(10), default="default")
+ )
+
+The "default" keyword can also take SQL expressions, including select statements or direct function calls:
+
+ {python}
+ t = Table("mytable", db,
+ Column('id', Integer, primary_key=True),
+
+ # define 'create_date' to default to now()
+ Column('create_date', DateTime, default=func.now()),
+
+ # define 'key' to pull its default from the 'keyvalues' table
+ Column('key', String(20), default=keyvalues.select(keyvalues.c.type='type1', limit=1))
+ )
+
+The "default" keyword argument is shorthand for using a ColumnDefault object in a column definition. This syntax is optional, but is required for other types of defaults, futher described below:
+
+ {python}
+ Column('mycolumn', String(30), ColumnDefault(func.get_data()))
+
+#### Pre-Executed OnUpdate Defaults {@name=onupdate}
+
+Similar to an on-insert default is an on-update default, which is most easily specified by the "onupdate" keyword to Column, which also can be a constant, plain Python function or SQL expression:
+
+ {python}
+ t = Table("mytable", db,
+ Column('id', Integer, primary_key=True),
+
+ # define 'last_updated' to be populated with current_timestamp (the ANSI-SQL version of now())
+ Column('last_updated', DateTime, onupdate=func.current_timestamp()),
+ )
+
+
+To use an explicit ColumnDefault object to specify an on-update, use the "for_update" keyword argument:
+
+ {python}
+ Column('mycolumn', String(30), ColumnDefault(func.get_data(), for_update=True))
+
+#### Inline Default Execution: PassiveDefault {@name=passive}
+
+A PassiveDefault indicates a column default or on-update value that is executed automatically by the database. This construct is used to specify a SQL function that will be specified as "DEFAULT" when creating tables, and also to indicate the presence of new data that is available to be "post-fetched" after an insert or update execution.
+
+ {python}
+ t = Table('test', e,
+ Column('mycolumn', DateTime, PassiveDefault("sysdate"))
+ )
+
+A create call for the above table will produce:
+
+ {code}
+ CREATE TABLE test (
+ mycolumn datetime default sysdate
+ )
+
+PassiveDefaults also send a message to the SQLEngine that data is available after update or insert. The object-relational mapper system uses this information to post-fetch rows after insert or update, so that instances can be refreshed with the new data. Below is a simplified version:
+
+ {python}
+ # table with passive defaults
+ mytable = Table('mytable', engine,
+ Column('my_id', Integer, primary_key=True),
+
+ # an on-insert database-side default
+ Column('data1', Integer, PassiveDefault("d1_func")),
+
+ # an on-update database-side default
+ Column('data2', Integer, PassiveDefault("d2_func", for_update=True))
+ )
+ # insert a row
+ mytable.insert().execute(name='fred')
+
+ # ask the engine: were there defaults fired off on that row ?
+ if table.engine.lastrow_has_defaults():
+ # postfetch the row based on primary key.
+ # this only works for a table with primary key columns defined
+ primary_key = table.engine.last_inserted_ids()
+ row = table.select(table.c.id == primary_key[0])
+
+When Tables are reflected from the database using <code>autoload=True</code>, any DEFAULT values set on the columns will be reflected in the Table object as PassiveDefault instances.
+
+##### The Catch: Postgres Primary Key Defaults always Pre-Execute {@name=postgres}
+
+Current Postgres support does not rely upon OID's to determine the identity of a row. This is because the usage of OIDs has been deprecated with Postgres and they are disabled by default for table creates as of PG version 8. Pyscopg2's "cursor.lastrowid" function only returns OIDs. Therefore, when inserting a new row which has passive defaults set on the primary key columns, the default function is <b>still pre-executed</b> since SQLAlchemy would otherwise have no way of retrieving the row just inserted.
+
+
+#### Defining Sequences {@name=sequences}
+
+A table with a sequence looks like:
+
+ {python}
+ table = Table("cartitems", db,
+ Column("cart_id", Integer, Sequence('cart_id_seq'), primary_key=True),
+ Column("description", String(40)),
+ Column("createdate", DateTime())
+ )
+
+The Sequence is used with Postgres or Oracle to indicate the name of a Sequence that will be used to create default values for a column. When a table with a Sequence on a column is created by SQLAlchemy, the Sequence object is also created. Similarly, the Sequence is dropped when the table is dropped. Sequences are typically used with primary key columns. When using Postgres, if an integer primary key column defines no explicit Sequence or other default method, SQLAlchemy will create the column with the SERIAL keyword, and will pre-execute a sequence named "tablename_columnname_seq" in order to retrieve new primary key values. Oracle, which has no "auto-increment" keyword, requires that a Sequence be created for a table if automatic primary key generation is desired. Note that for all databases, primary key values can always be explicitly stated within the bind parameters for any insert statement as well, removing the need for any kind of default generation function.
+
+A Sequence object can be defined on a Table that is then used for a non-sequence-supporting database. In that case, the Sequence object is simply ignored. Note that a Sequence object is <b>entirely optional for all databases except Oracle</b>, as other databases offer options for auto-creating primary key values, such as AUTOINCREMENT, SERIAL, etc. SQLAlchemy will use these default methods for creating primary key values if no Sequence is present on the table metadata.
+
+A sequence can also be specified with `optional=True` which indicates the Sequence should only be used on a database that requires an explicit sequence, and not those that supply some other method of providing integer values. At the moment, it essentially means "use this sequence only with Oracle and not Postgres".
+
+### Defining Indexes {@name=indexes}
+
+Indexes can be defined on table columns, including named indexes, non-unique or unique, multiple column. Indexes are included along with table create and drop statements. They are not used for any kind of run-time constraint checking; SQLAlchemy leaves that job to the expert on constraint checking, the database itself.
+
+ {python}
+ mytable = Table('mytable', engine,
+ # define a unique index
+ Column('col1', Integer, unique=True),
+
+ # define a unique index with a specific name
+ Column('col2', Integer, unique='mytab_idx_1'),
+
+ # define a non-unique index
+ Column('col3', Integer, index=True),
+
+ # define a non-unique index with a specific name
+ Column('col4', Integer, index='mytab_idx_2'),
+
+ # pass the same name to multiple columns to add them to the same index
+ Column('col5', Integer, index='mytab_idx_2'),
+
+ Column('col6', Integer),
+ Column('col7', Integer)
+ )
+
+ # create the table. all the indexes will be created along with it.
+ mytable.create()
+
+ # indexes can also be specified standalone
+ i = Index('mytab_idx_3', mytable.c.col6, mytable.c.col7, unique=False)
+
+ # which can then be created separately (will also get created with table creates)
+ i.create()
+
+### Adapting Tables to Alternate Engines {@name=adapting}
+
+A Table object created against a specific engine can be re-created against a new engine using the `toengine` method:
+
+ {python}
+ # create two engines
+ sqlite_engine = create_engine('sqlite', {'filename':'querytest.db'})
+ postgres_engine = create_engine('postgres',
+ {'database':'test',
+ 'host':'127.0.0.1', 'user':'scott', 'password':'tiger'})
+
+ # load 'users' from the sqlite engine
+ users = Table('users', sqlite_engine, autoload=True)
+
+ # create the same Table object for the other engine
+ pg_users = users.toengine(postgres_engine)
+
+Also available is the "database neutral" ansisql engine:
+
+ {python}
+ import sqlalchemy.ansisql as ansisql
+ generic_engine = ansisql.engine()
+
+ users = Table('users', generic_engine,
+ Column('user_id', Integer),
+ Column('user_name', String(50))
+ )
+
+Flexible "multi-engined" tables can also be achieved via the proxy engine, described in the section [dbengine_proxy](rel:dbengine_proxy).
+
+#### Non-engine primitives: TableClause/ColumnClause {@name=primitives}
+
+TableClause and ColumnClause are "primitive" versions of the Table and Column objects which dont use engines at all; applications that just want to generate SQL strings but not directly communicate with a database can use TableClause and ColumnClause objects (accessed via 'table' and 'column'), which are non-singleton and serve as the "lexical" base class of Table and Column:
+
+ {python}
+ tab1 = table('table1',
+ column('id'),
+ column('name'))
+
+ tab2 = table('table2',
+ column('id'),
+ column('email'))
+
+ tab1.select(tab1.c.name == 'foo')
+
+TableClause and ColumnClause are strictly lexical. This means they are fully supported within the full range of SQL statement generation, but they don't support schema concepts like creates, drops, primary keys, defaults, nullable status, indexes, or foreign keys.
+
#### Order By {@name=orderby}
-The ORDER BY clause of a select statement can be specified as individual columns to order by within an array specified via the `order_by` parameter, and optional usage of the asc() and desc() functions:
+The ORDER BY clause of a select statement can be specified as individual columns to order by within an array specified via the `order_by` parameter, and optional usage of the asc() and desc() functions:
{python}# straight order by
{sql}c = users.select(order_by=[users.c.user_name]).execute()
+++ /dev/null
-<%flags>inherit='document_base.myt'</%flags>
-<%attr>title='The Types System'</%attr>
-
-<&|doclib.myt:item, name="types", description="The Types System" &>
-<p>The package <span class="codeline">sqlalchemy.types</span> defines the datatype identifiers which may be used when defining <&formatting.myt:link, path="metadata", text="table metadata"&>. This package includes a set of generic types, a set of SQL-specific subclasses of those types, and a small extension system used by specific database connectors to adapt these generic types into database-specific type objects.
-</p>
-<&|doclib.myt:item, name="standard", description="Built-in Types" &>
-
-<p>SQLAlchemy comes with a set of standard generic datatypes, which are defined as classes.
-</p>
-<p>The standard set of generic types are:</p>
-<&|formatting.myt:code, title="package sqlalchemy.types"&>
- class String(TypeEngine):
- def __init__(self, length=None)
-
- class Integer(TypeEngine)
-
- class SmallInteger(Integer)
-
- class Numeric(TypeEngine):
- def __init__(self, precision=10, length=2)
-
- class Float(Numeric):
- def __init__(self, precision=10)
-
- class DateTime(TypeEngine)
-
- class Date(TypeEngine)
-
- class Time(TypeEngine)
-
- class Binary(TypeEngine):
- def __init__(self, length=None)
-
- class Boolean(TypeEngine)
-
- # converts unicode strings to raw bytes
- # as bind params, raw bytes to unicode as
- # rowset values, using the unicode encoding
- # setting on the engine (defaults to 'utf-8')
- class Unicode(String)
-
- # uses the pickle protocol to serialize data
- # in/out of Binary columns
- class PickleType(Binary)
-</&>
-<p>More specific subclasses of these types are available, which various database engines may choose to implement specifically, allowing finer grained control over types:</p>
-<&|formatting.myt:code&>
-class FLOAT(Numeric)
-class TEXT(String)
-class DECIMAL(Numeric)
-class INT(Integer)
-INTEGER = INT
-class TIMESTAMP(DateTime)
-class DATETIME(DateTime)
-class CLOB(String)
-class VARCHAR(String)
-class CHAR(String)
-class BLOB(Binary)
-class BOOLEAN(Boolean)
-</&>
-<p>When using a specific database engine, these types are adapted even further via a set of database-specific subclasses defined by the database engine.</p>
-<p>There may eventually be more type objects that are defined for specific databases. An example of this would be Postgres' Array type.
-</p>
-<p>Type objects are specified to table meta data using either the class itself, or an instance of the class. Creating an instance of the class allows you to specify parameters for the type, such as string length, numerical precision, etc.:</p>
-<&|formatting.myt:code&>
- mytable = Table('mytable', engine,
- # define type using a class
- Column('my_id', Integer, primary_key=True),
-
- # define type using an object instance
- Column('value', Number(7,4))
- )
-</&>
-</&>
-
-<&|doclib.myt:item, name="custom", description="Creating your Own Types" &>
-<p>User-defined types can be created, to support either database-specific types, or customized pre-processing of query parameters as well as post-processing of result set data. You can make your own classes to perform these operations. They are specified by subclassing the desired type class:</p>
-<&|formatting.myt:code, title="Basic Example"&>
- import sqlalchemy.types as types
-
- class MyType(types.String):
- """basic type that decorates String, prefixes values with "PREFIX:" on
- the way in and strips it off on the way out."""
- def convert_bind_param(self, value, engine):
- return "PREFIX:" + value
- def convert_result_value(self, value, engine):
- return value[7:]
-</&>
-<p>A common desire is for a "pickle" type, which overrides a Binary object to provide pickling behavior:
-<&|formatting.myt:code, title="Pickle Type"&>
- import cPickle
-
- class PickleType(Binary):
- def __init__(self, protocol=pickle.HIGHEST_PROTOCOL):
- """allows the pickle protocol to be specified"""
- self.protocol = protocol
- def convert_result_value(self, value, engine):
- if value is None:
- return None
- buf = Binary.convert_result_value(self, value, engine)
- return pickle.loads(str(buf))
- def convert_bind_param(self, value, engine):
- if value is None:
- return None
- return Binary.convert_bind_param(self, pickle.dumps(value, self.protocol), engine)
- def get_constructor_args(self):
- return {}
-</&>
-<p>Which can be used like:</p>
-<&|formatting.myt:code&>
- mytable = Table('mytable', engine,
- Column('id', Integer, primary_key=True),
- Column('data', PickleType()))
-
- my_object = MyObject()
- mytable.insert().execute(data=my_object)
-</&>
-<p>Another example, which illustrates a fully defined datatype. This just overrides the base type class TypeEngine:</p>
-<&|formatting.myt:code&>
- import sqlalchemy.types as types
-
- class MyType(types.TypeEngine):
- def __init__(self, precision = 8):
- self.precision = precision
- def get_col_spec(self):
- return "MYTYPE(%s)" % self.precision
- def convert_bind_param(self, value, engine):
- return value
- def convert_result_value(self, value, engine):
- return value
- def adapt_args(self):
- """allows for the adaptation of this TypeEngine object into a new kind of type depending on its arguments."""
- return self
-</&>
-</&>
-</&>
--- /dev/null
+The Types System {@name=types}
+================
+
+The package `sqlalchemy.types` defines the datatype identifiers which may be used when defining [metadata](rel:table metadata). This package includes a set of generic types, a set of SQL-specific subclasses of those types, and a small extension system used by specific database connectors to adapt these generic types into database-specific type objects.
+
+### Built-in Types {@name=standard}
+
+SQLAlchemy comes with a set of standard generic datatypes, which are defined as classes.
+
+The standard set of generic types are:
+
+ {python title="package sqlalchemy.types"}
+ class String(TypeEngine):
+ def __init__(self, length=None)
+
+ class Integer(TypeEngine)
+
+ class SmallInteger(Integer)
+
+ class Numeric(TypeEngine):
+ def __init__(self, precision=10, length=2)
+
+ class Float(Numeric):
+ def __init__(self, precision=10)
+
+ class DateTime(TypeEngine)
+
+ class Date(TypeEngine)
+
+ class Time(TypeEngine)
+
+ class Binary(TypeEngine):
+ def __init__(self, length=None)
+
+ class Boolean(TypeEngine)
+
+ # converts unicode strings to raw bytes
+ # as bind params, raw bytes to unicode as
+ # rowset values, using the unicode encoding
+ # setting on the engine (defaults to 'utf-8')
+ class Unicode(String)
+
+ # uses the pickle protocol to serialize data
+ # in/out of Binary columns
+ class PickleType(Binary)
+
+More specific subclasses of these types are available, which various database engines may choose to implement specifically, allowing finer grained control over types:
+
+ {python}
+ class FLOAT(Numeric)
+ class TEXT(String)
+ class DECIMAL(Numeric)
+ class INT(Integer)
+ INTEGER = INT
+ class TIMESTAMP(DateTime)
+ class DATETIME(DateTime)
+ class CLOB(String)
+ class VARCHAR(String)
+ class CHAR(String)
+ class BLOB(Binary)
+ class BOOLEAN(Boolean)
+
+When using a specific database engine, these types are adapted even further via a set of database-specific subclasses defined by the database engine.
+There may eventually be more type objects that are defined for specific databases. An example of this would be Postgres' Array type.
+
+Type objects are specified to table meta data using either the class itself, or an instance of the class. Creating an instance of the class allows you to specify parameters for the type, such as string length, numerical precision, etc.:
+
+ {python}
+ mytable = Table('mytable', engine,
+ # define type using a class
+ Column('my_id', Integer, primary_key=True),
+
+ # define type using an object instance
+ Column('value', Number(7,4))
+ )
+
+### Creating your Own Types {@name=custom}
+
+User-defined types can be created, to support either database-specific types, or customized pre-processing of query parameters as well as post-processing of result set data. You can make your own classes to perform these operations. They are specified by subclassing the desired type class:
+
+ {python title="Basic Example"}
+ import sqlalchemy.types as types
+
+ class MyType(types.String):
+ """basic type that decorates String, prefixes values with "PREFIX:" on
+ the way in and strips it off on the way out."""
+ def convert_bind_param(self, value, engine):
+ return "PREFIX:" + value
+ def convert_result_value(self, value, engine):
+ return value[7:]
+
+A common desire is for a "pickle" type, which overrides a Binary object to provide pickling behavior:
+
+ {python title="Pickle Type"}
+ import cPickle
+
+ class PickleType(Binary):
+ def __init__(self, protocol=pickle.HIGHEST_PROTOCOL):
+ """allows the pickle protocol to be specified"""
+ self.protocol = protocol
+ def convert_result_value(self, value, engine):
+ if value is None:
+ return None
+ buf = Binary.convert_result_value(self, value, engine)
+ return pickle.loads(str(buf))
+ def convert_bind_param(self, value, engine):
+ if value is None:
+ return None
+ return Binary.convert_bind_param(self, pickle.dumps(value, self.protocol), engine)
+ def get_constructor_args(self):
+ return {}
+
+Which can be used like:
+
+ {python}
+ mytable = Table('mytable', engine,
+ Column('id', Integer, primary_key=True),
+ Column('data', PickleType()))
+
+ my_object = MyObject()
+ mytable.insert().execute(data=my_object)
+
+Another example, which illustrates a fully defined datatype. This just overrides the base type class TypeEngine:
+
+ {python}
+ import sqlalchemy.types as types
+
+ class MyType(types.TypeEngine):
+ def __init__(self, precision = 8):
+ self.precision = precision
+ def get_col_spec(self):
+ return "MYTYPE(%s)" % self.precision
+ def convert_bind_param(self, value, engine):
+ return value
+ def convert_result_value(self, value, engine):
+ return value
+ def adapt_args(self):
+ """allows for the adaptation of this TypeEngine object into a new kind of type depending on its arguments."""
+ return self
+
+++ /dev/null
-<%flags>inherit='document_base.myt'</%flags>
-<%attr>title='Unit of Work'</%attr>
-
-<&|doclib.myt:item, name="unitofwork", description="Unit of Work" &>
- <&|doclib.myt:item, name="overview", description="Overview" &>
- <p>The concept behind Unit of Work is to track modifications to a field of objects, and then be able to commit those changes to the database in a single operation. Theres a lot of advantages to this, including that your application doesn't need to worry about individual save operations on objects, nor about the required order for those operations, nor about excessive repeated calls to save operations that would be more efficiently aggregated into one step. It also simplifies database transactions, providing a neat package with which to insert into the traditional database begin/commit phase.
- </p>
- <p>SQLAlchemy's unit of work includes these functions:
- <ul>
- <li>The ability to monitor scalar and list attributes on object instances, as well as object creates. This is handled via the attributes package.</li>
- <li>The ability to maintain and process a list of modified objects, and based on the relationships set up by the mappers for those objects as well as the foreign key relationships of the underlying tables, figure out the proper order of operations so that referential integrity is maintained, and also so that on-the-fly values such as newly created primary keys can be propigated to dependent objects that need them before they are saved. The central algorithm for this is the <b>topological sort</b>.</li>
- <li>The ability to define custom functionality that occurs within the unit-of-work commit phase, such as "before insert", "after insert", etc. This is accomplished via MapperExtension.</li>
- <li>an Identity Map, which is a dictionary storing the one and only instance of an object for a particular table/primary key combination. This allows many parts of an application to get a handle to a particular object without any chance of modifications going to two different places.</li>
- <li>Thread-local operation. the Identity map as well as its enclosing Unit of Work are normally instantiated and accessed in a manner that is local to the current thread, within an object called a Session. Another concurrently executing thread will therefore have its own Session, so unless an application explicitly shares objects between threads, the operation of the object relational mapping is automatically threadsafe. Session objects can also be constructed manually to allow any user-defined scoping.</li>
- </ul></p>
- </&>
- <&|doclib.myt:item, name="session", description="The Session Interface" &>
- <p>The current unit of work is accessed via a Session object. The Session is available in a thread-local context from the objectstore module as follows:</p>
- <&|formatting.myt:code&>
- # get the current thread's session
- session = objectstore.get_session()
- </&>
- <p>The Session object acts as a proxy to an underlying UnitOfWork object. Common methods include commit(), begin(), clear(), and delete(). Most of these methods are available at the module level in the objectstore module, which operate upon the Session returned by the get_session() function:
- </p>
- <&|formatting.myt:code&>
- # this...
- objectstore.get_session().commit()
-
- # is the same as this:
- objectstore.commit()
- </&>
-
- <p>A description of the most important methods and concepts follows.</p>
-
- <&|doclib.myt:item, name="identitymap", description="Identity Map" &>
- <p>The first concept to understand about the Unit of Work is that it is keeping track of all mapped objects which have been loaded from the database, as well as all mapped objects which have been saved to the database in the current session. This means that everytime you issue a <code>select</code> call to a mapper which returns results, all of those objects are now installed within the current Session, mapped to their identity.</p>
-
- <p>In particular, it is insuring that only <b>one</b> instance of a particular object, corresponding to a particular database identity, exists within the Session at one time. By "database identity" we mean a table or relational concept in the database combined with a particular primary key in that table. The session accomplishes this task using a dictionary known as an <b>Identity Map</b>. When <code>select</code> or <code>get</code> calls on mappers issue queries to the database, they will in nearly all cases go out to the database on each call to fetch results. However, when the mapper <b>instantiates</b> objects corresponding to the result set rows it receives, it will <b>check the current identity map first</b> before instantating a new object, and return <b>the same instance</b> already present in the identiy map if it already exists.</p>
-
- <p>Example:</p>
- <&|formatting.myt:code&>
- mymapper = mapper(MyClass, mytable)
-
- obj1 = mymapper.selectfirst(mytable.c.id==15)
- obj2 = mymapper.selectfirst(mytable.c.id==15)
-
- >>> obj1 is obj2
- True
- </&>
- <p>The Identity Map is an instance of <code>weakref.WeakValueDictionary</code>, so that when an in-memory object falls out of scope, it will be removed automatically. However, this may not be instant if there are circular references upon the object. The current SA attributes implementation places some circular refs upon objects, although this may change in the future. There are other ways to remove object instances from the current session, as well as to clear the current session entirely, which are described later in this section.</p>
- <p>To view the Session's identity map, it is accessible via the <code>identity_map</code> accessor, and is an instance of <code>weakref.WeakValueDictionary</code>:</p>
- <&|formatting.myt:code&><% """
- >>> objectstore.get_session().identity_map.values()
- [<__main__.User object at 0x712630>, <__main__.Address object at 0x712a70>]
- """ %>
- </&>
-
- <p>The identity of each object instance is available via the _instance_key property attached to each object instance, and is a tuple consisting of the object's class and an additional tuple of primary key values, in the order that they appear within the table definition:</p>
- <&|formatting.myt:code&>
- >>> obj._instance_key
- (<class 'test.tables.User'>, (7,))
- </&>
-
- <p>
- At the moment that an object is assigned this key, it is also added to the current thread's unit-of-work's identity map.
- </p>
-
- <p>The get() method on a mapper, which retrieves an object based on primary key identity, also checks in the current identity map first to save a database round-trip if possible. In the case of an object lazy-loading a single child object, the get() method is used as well, so scalar-based lazy loads may in some cases not query the database; this is particularly important for backreference relationships as it can save a lot of queries.</p>
-
- <p>Methods on mappers and the objectstore module, which are relevant to identity include the following:</p>
- <&|formatting.myt:code&>
- # assume 'm' is a mapper
- m = mapper(User, users)
-
- # get the identity key corresponding to a primary key
- key = m.identity_key(7)
-
- # for composite key, list out the values in the order they
- # appear in the table
- key = m.identity_key(12, 'rev2')
-
- # get the identity key given a primary key
- # value as a tuple and a class
- key = objectstore.get_id_key((12, 'rev2'), User)
-
- # get the identity key for an object, whether or not it actually
- # has one attached to it (m is the mapper for obj's class)
- key = m.instance_key(obj)
-
- # is this key in the current identity map?
- session.has_key(key)
-
- # is this object in the current identity map?
- session.has_instance(obj)
-
- # get this object from the current identity map based on
- # singular/composite primary key, or if not go
- # and load from the database
- obj = m.get(12, 'rev2')
- </&>
-
- </&>
-
- <&|doclib.myt:item, name="changed", description="Whats Changed ?" &>
- <p>The next concept is that in addition to the Session storing a record of all objects loaded or saved, it also stores records of all <b>newly created</b> objects, records of all objects whose attributes have been <b>modified</b>, records of all objects that have been marked as <b>deleted</b>, and records of all <b>modified list-based attributes</b> where additions or deletions have occurred. These lists are used when a <code>commit()</code> call is issued to save all changes. After the commit occurs, these lists are all cleared out.</p>
-
- <p>These records are all tracked by a collection of <code>Set</code> objects (which are a SQLAlchemy-specific instance called a <code>HashSet</code>) that are also viewable off the Session:</p>
- <&|formatting.myt:code&>
- # new objects that were just constructed
- session.new
-
- # objects that exist in the database, that were modified
- session.dirty
-
- # objects that have been marked as deleted via session.delete(obj)
- session.deleted
-
- # list-based attributes thave been appended
- session.modified_lists
- </&>
- <p>Heres an interactive example, assuming the <code>User</code> and <code>Address</code> mapper setup first outlined in <&formatting.myt:link, path="datamapping_relations"&>:</p>
- <&|formatting.myt:code&>
- ">>>" # get the current thread's session
- ">>>" session = objectstore.get_session()
-
- ">>>" # create a new object, with a list-based attribute
- ">>>" # containing two more new objects
- ">>>" u = User(user_name='Fred')
- ">>>" u.addresses.append(Address(city='New York'))
- ">>>" u.addresses.append(Address(city='Boston'))
-
- ">>>" # objects are in the "new" list
- ">>>" session.new
- [<__main__.User object at 0x713630>,
- <__main__.Address object at 0x713a70>,
- <__main__.Address object at 0x713b30>]
-
- ">>>" # view the "modified lists" member,
- ">>>" # reveals our two Address objects as well, inside of a list
- ">>>" session.modified_lists
- [[<__main__.Address object at 0x713a70>, <__main__.Address object at 0x713b30>]]
-
- ">>>" # lets view what the class/ID is for the list object
- ">>>" ["%s %s" % (l.__class__, id(l)) for l in session.modified_lists]
- ['sqlalchemy.mapping.unitofwork.UOWListElement 7391872']
-
- ">>>" # now commit
- ">>>" session.commit()
-
- ">>>" # the "new" list is now empty
- ">>>" session.new
- []
-
- ">>>" # the "modified lists" list is now empty
- ">>>" session.modified_lists
- []
-
- ">>>" # now lets modify an object
- ">>>" u.user_name='Ed'
-
- ">>>" # it gets placed in the "dirty" list
- ">>>" session.dirty
- [<__main__.User object at 0x713630>]
-
- ">>>" # delete one of the addresses
- ">>>" session.delete(u.addresses[0])
-
- ">>>" # and also delete it off the User object, note that
- ">>>" # this is *not automatic* when using session.delete()
- ">>>" del u.addresses[0]
- ">>>" session.deleted
- [<__main__.Address object at 0x713a70>]
-
- ">>>" # commit
- ">>>" session.commit()
-
- ">>>" # all lists are cleared out
- ">>>" session.new, session.dirty, session.modified_lists, session.deleted
- ([], [], [], [])
-
- ">>>" # identity map has the User and the one remaining Address
- ">>>" session.identity_map.values()
- [<__main__.Address object at 0x713b30>, <__main__.User object at 0x713630>]
- </&>
- <p>Unlike the identity map, the <code>new</code>, <code>dirty</code>, <code>modified_lists</code>, and <code>deleted</code> lists are <b>not weak referencing.</b> This means if you abandon all references to new or modified objects within a session, <b>they are still present</b> and will be saved on the next commit operation, unless they are removed from the Session explicitly (more on that later). The <code>new</code> list may change in a future release to be weak-referencing, however for the <code>deleted</code> list, one can see that its quite natural for a an object marked as deleted to have no references in the application, yet a DELETE operation is still required.</p>
- </&>
-
- <&|doclib.myt:item, name="commit", description="Commit" &>
- <p>This is the main gateway to what the Unit of Work does best, which is save everything ! It should be clear by now that a commit looks like:
- </p>
- <&|formatting.myt:code&>
- objectstore.get_session().commit()
- </&>
- <p>It also can be called with a list of objects; in this form, the commit operation will be limited only to the objects specified in the list, as well as any child objects within <code>private</code> relationships for a delete operation:</p>
- <&|formatting.myt:code&>
- # saves only user1 and address2. all other modified
- # objects remain present in the session.
- objectstore.get_session().commit(user1, address2)
- </&>
- <p>This second form of commit should be used more carefully as it will not necessarily locate other dependent objects within the session, whose database representation may have foreign constraint relationships with the objects being operated upon.</p>
-
- <&|doclib.myt:item, name="whatis", description="What Commit is, and Isn't" &>
- <p>The purpose of the Commit operation, as defined by the <code>objectstore</code> package, is to instruct the Unit of Work to analyze its lists of modified objects, assemble them into a dependency graph, fire off the appopriate INSERT, UPDATE, and DELETE statements via the mappers related to those objects, and to synchronize column-based object attributes that correspond directly to updated/inserted database columns.</p>
- <p>Its important to note that the <b>objectstore.get_session().commit() operation is not the same as the commit() operation on SQLEngine.</b> A <code>SQLEngine</code>, described in <&formatting.myt:link, path="database"&>, has its own <code>begin</code> and <code>commit</code> statements which deal directly with transactions opened on DBAPI connections. While the <code>session.commit()</code> makes use of these calls in order to issue its own SQL within a database transaction, it is only dealing with "committing" its own in-memory changes and only has an indirect relationship with database connection objects.
- </p>
- <p>The <code>session.commit()</code> operation also does not affect any <code>relation</code>-based object attributes, that is attributes that reference other objects or lists of other objects, in any way. A brief list of what will <b>not</b> happen includes:</p>
- <ul>
- <li>It will not append or delete any object instances to/from any list-based object attributes. Any objects that have been created or marked as deleted will be updated as such in the database, but if a newly deleted object instance is still attached to a parent object's list, the object itself will remain in that list.</li>
- <li>It will not set or remove any scalar references to other objects, even if the corresponding database identifier columns have been committed.</li>
- </ul>
- <p>This means, if you set <code>address.user_id</code> to 5, that integer attribute will be saved, but it will not place an <code>Address</code> object in the <code>addresses</code> attribute of the corresponding <code>User</code> object. In some cases there may be a lazy-loader still attached to an object attribute which when first accesed performs a fresh load from the database and creates the appearance of this behavior, but this behavior should not be relied upon as it is specific to lazy loading and also may disappear in a future release. Similarly, if the <code>Address</code> object is marked as deleted and a commit is issued, the correct DELETE statements will be issued, but if the object instance itself is still attached to the <code>User</code>, it will remain.</p>
- <P>So the primary guideline for dealing with commit() is, <b>the developer is responsible for maintaining in-memory objects and their relationships to each other, the unit of work is responsible for maintaining the database representation of the in-memory objects.</b> The typical pattern is that the manipulation of objects *is* the way that changes get communicated to the unit of work, so that when the commit occurs, the objects are already in their correct in-memory representation and problems dont arise. The manipulation of identifier attributes like integer key values as well as deletes in particular are a frequent source of confusion.</p>
-
- <p>A terrific feature of SQLAlchemy which is also a supreme source of confusion is the backreference feature, described in <&formatting.myt:link, path="datamapping_relations_backreferences"&>. This feature allows two types of objects to maintain attributes that reference each other, typically one object maintaining a list of elements of the other side, which contains a scalar reference to the list-holding object. When you append an element to the list, the element gets a "backreference" back to the object which has the list. When you attach the list-holding element to the child element, the child element gets attached to the list. <b>This feature has nothing to do whatsoever with the Unit of Work.*</b> It is strictly a small convenience feature intended to support the developer's manual manipulation of in-memory objects, and the backreference operation happens at the moment objects are attached or removed to/from each other, independent of any kind of database operation. It does not change the golden rule, that the developer is reponsible for maintaining in-memory object relationships.</p>
- <p>* there is an internal relationship between two <code>relations</code> that have a backreference, which state that a change operation is only logged once to the unit of work instead of two separate changes since the two changes are "equivalent", so a backreference does affect the information that is sent to the Unit of Work. But the Unit of Work itself has no knowledge of this arrangement and has no ability to affect it.</p>
- </&>
- </&>
-
- <&|doclib.myt:item, name="delete", description="Delete" &>
- <P>The delete call places an object or objects into the Unit of Work's list of objects to be marked as deleted:</p>
- <&|formatting.myt:code&>
- # mark three objects to be deleted
- objectstore.get_session().delete(obj1, obj2, obj3)
-
- # commit
- objectstore.get_session().commit()
- </&>
- <p>When objects which contain references to other objects are deleted, the mappers for those related objects will issue UPDATE statements for those objects that should no longer contain references to the deleted object, setting foreign key identifiers to NULL. Similarly, when a mapper contains relations with the <code>private=True</code> option, DELETE statements will be issued for objects within that relationship in addition to that of the primary deleted object; this is called a <b>cascading delete</b>.</p>
- <p>As stated before, the purpose of delete is strictly to issue DELETE statements to the database. It does not affect the in-memory structure of objects, other than changing the identifying attributes on objects, such as setting foreign key identifiers on updated rows to None. It has no effect on the status of references between object instances, nor any effect on the Python garbage-collection status of objects.</p>
- </&>
-
- <&|doclib.myt:item, name="clear", description="Clear" &>
- <p>To clear out the current thread's UnitOfWork, which has the effect of discarding the Identity Map and the lists of all objects that have been modified, just issue a clear:
- </p>
- <&|formatting.myt:code&>
- # via module
- objectstore.clear()
-
- # or via Session
- objectstore.get_session().clear()
- </&>
- <p>This is the easiest way to "start fresh", as in a web application that wants to have a newly loaded graph of objects on each request. Any object instances created before the clear operation should either be discarded or at least not used with any Mapper or Unit Of Work operations (with the exception of <code>import_instance()</code>), as they no longer have any relationship to the current Unit of Work, and their behavior with regards to the current session is undefined.</p>
- </&>
-
- <&|doclib.myt:item, name="refreshexpire", description="Refresh / Expire" &>
- <p>To assist with the Unit of Work's "sticky" behavior, individual objects can have all of their attributes immediately re-loaded from the database, or marked as "expired" which will cause a re-load to occur upon the next access of any of the object's mapped attributes. This includes all relationships, so lazy-loaders will be re-initialized, eager relationships will be repopulated. Any changes marked on the object are discarded:</p>
- <&|formatting.myt:code&>
- # immediately re-load attributes on obj1, obj2
- session.refresh(obj1, obj2)
-
- # expire objects obj1, obj2, attributes will be reloaded
- # on the next access:
- session.expire(obj1, obj2, obj3)
- </&>
- </&>
-
- <&|doclib.myt:item, name="expunge", description="Expunge" &>
- <P>Expunge simply removes all record of an object from the current Session. This includes the identity map, and all history-tracking lists:</p>
- <&|formatting.myt:code&>
- session.expunge(obj1)
- </&>
- <p>Use <code>expunge</code> when youd like to remove an object altogether from memory, such as before calling <code>del</code> on it, which will prevent any "ghost" operations occuring when the session is committed.</p>
- </&>
-
- <&|doclib.myt:item, name="import", description="Import Instance" &>
- <p>The _instance_key attribute placed on object instances is designed to work with objects that are serialized into strings and brought back again. As it contains no references to internal structures or database connections, applications that use caches or session storage which require serialization (i.e. pickling) can store SQLAlchemy-loaded objects. However, as mentioned earlier, an object with a particular database identity is only allowed to exist uniquely within the current unit-of-work scope. So, upon deserializing such an object, it has to "check in" with the current Session. This is achieved via the <code>import_instance()</code> method:</p>
- <&|formatting.myt:code&>
- # deserialize an object
- myobj = pickle.loads(mystring)
-
- # "import" it. if the objectstore already had this object in the
- # identity map, then you get back the one from the current session.
- myobj = session.import_instance(myobj)
- </&>
- <p>Note that the import_instance() function will either mark the deserialized object as the official copy in the current identity map, which includes updating its _instance_key with the current application's class instance, or it will discard it and return the corresponding object that was already present. Thats why its important to receive the return results from the method and use the result as the official object instance.</p>
- </&>
-
- <&|doclib.myt:item, name="begin", description="Begin" &>
- <p>The "scope" of the unit of work commit can be controlled further by issuing a begin(). A begin operation constructs a new UnitOfWork object and sets it as the currently used UOW. It maintains a reference to the original UnitOfWork as its "parent", and shares the same identity map of objects that have been loaded from the database within the scope of the parent UnitOfWork. However, the "new", "dirty", and "deleted" lists are empty. This has the effect that only changes that take place after the begin() operation get logged to the current UnitOfWork, and therefore those are the only changes that get commit()ted. When the commit is complete, the "begun" UnitOfWork removes itself and places the parent UnitOfWork as the current one again.</p>
-<p>The begin() method returns a transactional object, upon which you can call commit() or rollback(). <b>Only this transactional object controls the transaction</b> - commit() upon the Session will do nothing until commit() or rollback() is called upon the transactional object.</p>
- <&|formatting.myt:code&>
- # modify an object
- myobj1.foo = "something new"
-
- # begin
- trans = session.begin()
-
- # modify another object
- myobj2.lala = "something new"
-
- # only 'myobj2' is saved
- trans.commit()
- </&>
- <p>begin/commit supports the same "nesting" behavior as the SQLEngine (note this behavior is not the original "nested" behavior), meaning that many begin() calls can be made, but only the outermost transactional object will actually perform a commit(). Similarly, calls to the commit() method on the Session, which might occur in function calls within the transaction, will not do anything; this allows an external function caller to control the scope of transactions used within the functions.</p>
- </&>
-
- </&>
-
- <&|doclib.myt:item, name="advscope", description="Advanced UnitOfWork Management"&>
-
- <&|doclib.myt:item, name="transactionnesting", description="Nesting UnitOfWork in a Database Transaction" &>
- <p>The UOW commit operation places its INSERT/UPDATE/DELETE operations within the scope of a database transaction controlled by a SQLEngine:
- <&|formatting.myt:code&>
- engine.begin()
- try:
- # run objectstore update operations
- except:
- engine.rollback()
- raise
- engine.commit()
- </&>
- <p>If you recall from the <&formatting.myt:link, path="dbengine_transactions"&> section, the engine's begin()/commit() methods support reentrant behavior. This means you can nest begin and commits and only have the outermost begin/commit pair actually take effect (rollbacks however, abort the whole operation at any stage). From this it follows that the UnitOfWork commit operation can be nested within a transaction as well:</p>
- <&|formatting.myt:code&>
- engine.begin()
- try:
- # perform custom SQL operations
- objectstore.commit()
- # perform custom SQL operations
- except:
- engine.rollback()
- raise
- engine.commit()
- </&>
-
- </&>
-
-
- <&|doclib.myt:item, name="object", description="Per-Object Sessions" &>
- <p>Sessions can be created on an ad-hoc basis and used for individual groups of objects and operations. This has the effect of bypassing the normal thread-local Session and explicitly using a particular Session:</p>
- <&|formatting.myt:code&>
- # make a new Session with a global UnitOfWork
- s = objectstore.Session()
-
- # make objects bound to this Session
- x = MyObj(_sa_session=s)
-
- # perform mapper operations bound to this Session
- # (this function coming soon)
- r = MyObj.mapper.using(s).select_by(id=12)
-
- # get the session that corresponds to an instance
- s = objectstore.get_session(x)
-
- # commit
- s.commit()
-
- # perform a block of operations with this session set within the current scope
- objectstore.push_session(s)
- try:
- r = mapper.select_by(id=12)
- x = new MyObj()
- objectstore.commit()
- finally:
- objectstore.pop_session()
- </&>
- <&|doclib.myt:item, name="nested", description="Nested Transaction Sessions" &>
- <p>Sessions also now support a "nested transaction" feature whereby a second Session can use a different database connection. This can be used inside of a larger database transaction to issue commits to the database that will be committed independently of the larger transaction's status:</p>
- <&|formatting.myt:code&>
- engine.begin()
- try:
- a = MyObj()
- b = MyObj()
-
- sess = Session(nest_on=engine)
- objectstore.push_session(sess)
- try:
- c = MyObj()
- objectstore.commit() # will commit "c" to the database,
- # even if the external transaction rolls back
- finally:
- objectstore.pop_session()
-
- objectstore.commit() # commit "a" and "b" to the database
- engine.commit()
- except:
- engine.rollback()
- raise
- </&>
- </&>
- </&>
-
- <&|doclib.myt:item, name="scope", description="Custom Session Objects/Custom Scopes" &>
-
- <p>For users who want to make their own Session subclass, or replace the algorithm used to return scoped Session objects (i.e. the objectstore.get_session() method):</p>
- <&|formatting.myt:code, title="Create a Session"&>
- # make a new Session
- s = objectstore.Session()
-
- # set it as the current thread-local session
- objectstore.session_registry.set(s)
- </&>
- <&|formatting.myt:code, title="Create a custom Registry Algorithm"&>
- # set the objectstore's session registry to a different algorithm
-
- def create_session():
- """creates new sessions"""
- return objectstore.Session()
- def mykey():
- """creates contextual keys to store scoped sessions"""
- return "mykey"
-
- objectstore.session_registry = sqlalchemy.util.ScopedRegistry(createfunc=create_session, scopefunc=mykey)
- </&>
- </&>
-
- <&|doclib.myt:item, name="logging", description="Analyzing Object Commits" &>
- <p>The objectstore module can log an extensive display of its "commit plans", which is a graph of its internal representation of objects before they are committed to the database. To turn this logging on:
- <&|formatting.myt:code&>
- # make an engine with echo_uow
- engine = create_engine('myengine...', echo_uow=True)
- </&>
- <p>Commits will then dump to the standard output displays like the following:</p>
- <&|formatting.myt:code, syntaxtype=None&>
- Task dump:
- UOWTask(6034768) 'User/users/6015696'
- |
- |- Save elements
- |- Save: UOWTaskElement(6034800): User(6016624) (save)
- |
- |- Save dependencies
- |- UOWDependencyProcessor(6035024) 'addresses' attribute on saved User's (UOWTask(6034768) 'User/users/6015696')
- | |-UOWTaskElement(6034800): User(6016624) (save)
- |
- |- Delete dependencies
- |- UOWDependencyProcessor(6035056) 'addresses' attribute on User's to be deleted (UOWTask(6034768) 'User/users/6015696')
- | |-(no objects)
- |
- |- Child tasks
- |- UOWTask(6034832) 'Address/email_addresses/6015344'
- | |
- | |- Save elements
- | |- Save: UOWTaskElement(6034864): Address(6034384) (save)
- | |- Save: UOWTaskElement(6034896): Address(6034256) (save)
- | |----
- |
- |----
- </&>
- <p>The above graph can be read straight downwards to determine the order of operations. It indicates "save User 6016624, process each element in the 'addresses' list on User 6016624, save Address 6034384, Address 6034256".
- </&>
-
- </&>
-</&>
--- /dev/null
+Unit of Work
+============
+
+### Overview {@name=overview}
+
+The concept behind Unit of Work is to track modifications to a field of objects, and then be able to commit those changes to the database in a single operation. Theres a lot of advantages to this, including that your application doesn't need to worry about individual save operations on objects, nor about the required order for those operations, nor about excessive repeated calls to save operations that would be more efficiently aggregated into one step. It also simplifies database transactions, providing a neat package with which to insert into the traditional database begin/commit phase.
+
+SQLAlchemy's unit of work includes these functions:
+
+* The ability to monitor scalar and list attributes on object instances, as well as object creates. This is handled via the attributes package.
+* The ability to maintain and process a list of modified objects, and based on the relationships set up by the mappers for those objects as well as the foreign key relationships of the underlying tables, figure out the proper order of operations so that referential integrity is maintained, and also so that on-the-fly values such as newly created primary keys can be propigated to dependent objects that need them before they are saved. The central algorithm for this is the *topological sort*.
+* The ability to define custom functionality that occurs within the unit-of-work commit phase, such as "before insert", "after insert", etc. This is accomplished via MapperExtension.
+* an Identity Map, which is a dictionary storing the one and only instance of an object for a particular table/primary key combination. This allows many parts of an application to get a handle to a particular object without any chance of modifications going to two different places.
+* Thread-local operation. the Identity map as well as its enclosing Unit of Work are normally instantiated and accessed in a manner that is local to the current thread, within an object called a Session. Another concurrently executing thread will therefore have its own Session, so unless an application explicitly shares objects between threads, the operation of the object relational mapping is automatically threadsafe. Session objects can also be constructed manually to allow any user-defined scoping.
+
+### The Session Interface {@name=session}
+
+The current unit of work is accessed via a Session object. The Session is available in a thread-local context from the objectstore module as follows:
+
+ {python}
+ # get the current thread's session
+ session = objectstore.get_session()
+
+The Session object acts as a proxy to an underlying UnitOfWork object. Common methods include commit(), begin(), clear(), and delete(). Most of these methods are available at the module level in the objectstore module, which operate upon the Session returned by the get_session() function:
+
+ {python}# this...
+ objectstore.get_session().commit()
+
+ # is the same as this:
+ objectstore.commit()
+
+A description of the most important methods and concepts follows.
+
+#### Identity Map {@name=identitymap}
+
+The first concept to understand about the Unit of Work is that it is keeping track of all mapped objects which have been loaded from the database, as well as all mapped objects which have been saved to the database in the current session. This means that everytime you issue a `select` call to a mapper which returns results, all of those objects are now installed within the current Session, mapped to their identity.
+
+In particular, it is insuring that only *one* instance of a particular object, corresponding to a particular database identity, exists within the Session at one time. By "database identity" we mean a table or relational concept in the database combined with a particular primary key in that table. The session accomplishes this task using a dictionary known as an *Identity Map*. When `select` or `get` calls on mappers issue queries to the database, they will in nearly all cases go out to the database on each call to fetch results. However, when the mapper *instantiates* objects corresponding to the result set rows it receives, it will *check the current identity map first* before instantating a new object, and return *the same instance* already present in the identiy map if it already exists.
+
+Example:
+
+ {python}mymapper = mapper(MyClass, mytable)
+
+ obj1 = mymapper.selectfirst(mytable.c.id==15)
+ obj2 = mymapper.selectfirst(mytable.c.id==15)
+
+ >>> obj1 is obj2
+ True
+
+The Identity Map is an instance of `weakref.WeakValueDictionary`, so that when an in-memory object falls out of scope, it will be removed automatically. However, this may not be instant if there are circular references upon the object. The current SA attributes implementation places some circular refs upon objects, although this may change in the future. There are other ways to remove object instances from the current session, as well as to clear the current session entirely, which are described later in this section.
+
+To view the Session's identity map, it is accessible via the `identity_map` accessor, and is an instance of `weakref.WeakValueDictionary`:
+
+ {python}
+ >>> objectstore.get_session().identity_map.values()
+ [<__main__.User object at 0x712630>, <__main__.Address object at 0x712a70>]
+
+The identity of each object instance is available via the _instance_key property attached to each object instance, and is a tuple consisting of the object's class and an additional tuple of primary key values, in the order that they appear within the table definition:
+
+ {python}
+ >>> obj._instance_key
+ (<class 'test.tables.User'>, (7,))
+
+At the moment that an object is assigned this key, it is also added to the current thread's unit-of-work's identity map.
+
+The get() method on a mapper, which retrieves an object based on primary key identity, also checks in the current identity map first to save a database round-trip if possible. In the case of an object lazy-loading a single child object, the get() method is used as well, so scalar-based lazy loads may in some cases not query the database; this is particularly important for backreference relationships as it can save a lot of queries.
+
+Methods on mappers and the objectstore module, which are relevant to identity include the following:
+
+ {python}
+ # assume 'm' is a mapper
+ m = mapper(User, users)
+
+ # get the identity key corresponding to a primary key
+ key = m.identity_key(7)
+
+ # for composite key, list out the values in the order they
+ # appear in the table
+ key = m.identity_key(12, 'rev2')
+
+ # get the identity key given a primary key
+ # value as a tuple and a class
+ key = objectstore.get_id_key((12, 'rev2'), User)
+
+ # get the identity key for an object, whether or not it actually
+ # has one attached to it (m is the mapper for obj's class)
+ key = m.instance_key(obj)
+
+ # is this key in the current identity map?
+ session.has_key(key)
+
+ # is this object in the current identity map?
+ session.has_instance(obj)
+
+ # get this object from the current identity map based on
+ # singular/composite primary key, or if not go
+ # and load from the database
+ obj = m.get(12, 'rev2')
+
+#### Whats Changed ? {@name=changed}
+
+The next concept is that in addition to the Session storing a record of all objects loaded or saved, it also stores records of all *newly created* objects, records of all objects whose attributes have been *modified*, records of all objects that have been marked as *deleted*, and records of all *modified list-based attributes* where additions or deletions have occurred. These lists are used when a `commit()` call is issued to save all changes. After the commit occurs, these lists are all cleared out.
+
+These records are all tracked by a collection of `Set` objects (which are a SQLAlchemy-specific instance called a `HashSet`) that are also viewable off the Session:
+
+ {python}
+ # new objects that were just constructed
+ session.new
+
+ # objects that exist in the database, that were modified
+ session.dirty
+
+ # objects that have been marked as deleted via session.delete(obj)
+ session.deleted
+
+ # list-based attributes thave been appended
+ session.modified_lists
+
+Heres an interactive example, assuming the `User` and `Address` mapper setup first outlined in [datamapping_relations](rel:datamapping_relations):
+
+ {python}
+ >>> # get the current thread's session
+ >>> session = objectstore.get_session()
+
+ >>> # create a new object, with a list-based attribute
+ >>> # containing two more new objects
+ >>> u = User(user_name='Fred')
+ >>> u.addresses.append(Address(city='New York'))
+ >>> u.addresses.append(Address(city='Boston'))
+
+ >>> # objects are in the "new" list
+ >>> session.new
+ [<__main__.User object at 0x713630>,
+ <__main__.Address object at 0x713a70>,
+ <__main__.Address object at 0x713b30>]
+
+ >>> # view the "modified lists" member,
+ >>> # reveals our two Address objects as well, inside of a list
+ >>> session.modified_lists
+ [[<__main__.Address object at 0x713a70>, <__main__.Address object at 0x713b30>]]
+
+ >>> # lets view what the class/ID is for the list object
+ >>> ["%s %s" % (l.__class__, id(l)) for l in session.modified_lists]
+ ['sqlalchemy.mapping.unitofwork.UOWListElement 7391872']
+
+ >>> # now commit
+ >>> session.commit()
+
+ >>> # the "new" list is now empty
+ >>> session.new
+ []
+
+ >>> # the "modified lists" list is now empty
+ >>> session.modified_lists
+ []
+
+ >>> # now lets modify an object
+ >>> u.user_name='Ed'
+
+ >>> # it gets placed in the "dirty" list
+ >>> session.dirty
+ [<__main__.User object at 0x713630>]
+
+ >>> # delete one of the addresses
+ >>> session.delete(u.addresses[0])
+
+ >>> # and also delete it off the User object, note that
+ >>> # this is *not automatic* when using session.delete()
+ >>> del u.addresses[0]
+ >>> session.deleted
+ [<__main__.Address object at 0x713a70>]
+
+ >>> # commit
+ >>> session.commit()
+
+ >>> # all lists are cleared out
+ >>> session.new, session.dirty, session.modified_lists, session.deleted
+ ([], [], [], [])
+
+ >>> # identity map has the User and the one remaining Address
+ >>> session.identity_map.values()
+ [<__main__.Address object at 0x713b30>, <__main__.User object at 0x713630>]
+
+Unlike the identity map, the `new`, `dirty`, `modified_lists`, and `deleted` lists are *not weak referencing.* This means if you abandon all references to new or modified objects within a session, *they are still present* and will be saved on the next commit operation, unless they are removed from the Session explicitly (more on that later). The `new` list may change in a future release to be weak-referencing, however for the `deleted` list, one can see that its quite natural for a an object marked as deleted to have no references in the application, yet a DELETE operation is still required.
+
+#### Commit {@name=commit}
+
+This is the main gateway to what the Unit of Work does best, which is save everything ! It should be clear by now that a commit looks like:
+
+ {python}
+ objectstore.get_session().commit()
+
+It also can be called with a list of objects; in this form, the commit operation will be limited only to the objects specified in the list, as well as any child objects within `private` relationships for a delete operation:
+
+ {python}
+ # saves only user1 and address2. all other modified
+ # objects remain present in the session.
+ objectstore.get_session().commit(user1, address2)
+
+This second form of commit should be used more carefully as it will not necessarily locate other dependent objects within the session, whose database representation may have foreign constraint relationships with the objects being operated upon.
+
+##### What Commit is, and Isn't {@name=whatis}
+
+The purpose of the Commit operation, as defined by the `objectstore` package, is to instruct the Unit of Work to analyze its lists of modified objects, assemble them into a dependency graph, fire off the appopriate INSERT, UPDATE, and DELETE statements via the mappers related to those objects, and to synchronize column-based object attributes that correspond directly to updated/inserted database columns.
+
+Its important to note that the *objectstore.get_session().commit() operation is not the same as the commit() operation on SQLEngine.* A `SQLEngine`, described in [database](rel:database), has its own `begin` and `commit` statements which deal directly with transactions opened on DBAPI connections. While the `session.commit()` makes use of these calls in order to issue its own SQL within a database transaction, it is only dealing with "committing" its own in-memory changes and only has an indirect relationship with database connection objects.
+
+The `session.commit()` operation also does not affect any `relation`-based object attributes, that is attributes that reference other objects or lists of other objects, in any way. A brief list of what will *not* happen includes:
+
+* It will not append or delete any object instances to/from any list-based object attributes. Any objects that have been created or marked as deleted will be updated as such in the database, but if a newly deleted object instance is still attached to a parent object's list, the object itself will remain in that list.
+* It will not set or remove any scalar references to other objects, even if the corresponding database identifier columns have been committed.
+
+This means, if you set `address.user_id` to 5, that integer attribute will be saved, but it will not place an `Address` object in the `addresses` attribute of the corresponding `User` object. In some cases there may be a lazy-loader still attached to an object attribute which when first accesed performs a fresh load from the database and creates the appearance of this behavior, but this behavior should not be relied upon as it is specific to lazy loading and also may disappear in a future release. Similarly, if the `Address` object is marked as deleted and a commit is issued, the correct DELETE statements will be issued, but if the object instance itself is still attached to the `User`, it will remain.
+
+So the primary guideline for dealing with commit() is, *the developer is responsible for maintaining in-memory objects and their relationships to each other, the unit of work is responsible for maintaining the database representation of the in-memory objects.* The typical pattern is that the manipulation of objects *is* the way that changes get communicated to the unit of work, so that when the commit occurs, the objects are already in their correct in-memory representation and problems dont arise. The manipulation of identifier attributes like integer key values as well as deletes in particular are a frequent source of confusion.
+
+A terrific feature of SQLAlchemy which is also a supreme source of confusion is the backreference feature, described in [datamapping_relations_backreferences](rel:datamapping_relations_backreferences). This feature allows two types of objects to maintain attributes that reference each other, typically one object maintaining a list of elements of the other side, which contains a scalar reference to the list-holding object. When you append an element to the list, the element gets a "backreference" back to the object which has the list. When you attach the list-holding element to the child element, the child element gets attached to the list. *This feature has nothing to do whatsoever with the Unit of Work.*`*` It is strictly a small convenience feature intended to support the developer's manual manipulation of in-memory objects, and the backreference operation happens at the moment objects are attached or removed to/from each other, independent of any kind of database operation. It does not change the golden rule, that the developer is reponsible for maintaining in-memory object relationships.
+
+`*` there is an internal relationship between two `relations` that have a backreference, which state that a change operation is only logged once to the unit of work instead of two separate changes since the two changes are "equivalent", so a backreference does affect the information that is sent to the Unit of Work. But the Unit of Work itself has no knowledge of this arrangement and has no ability to affect it.
+
+#### Delete {@name=delete}
+
+The delete call places an object or objects into the Unit of Work's list of objects to be marked as deleted:
+
+ {python}
+ # mark three objects to be deleted
+ objectstore.get_session().delete(obj1, obj2, obj3)
+
+ # commit
+ objectstore.get_session().commit()
+
+When objects which contain references to other objects are deleted, the mappers for those related objects will issue UPDATE statements for those objects that should no longer contain references to the deleted object, setting foreign key identifiers to NULL. Similarly, when a mapper contains relations with the `private=True` option, DELETE statements will be issued for objects within that relationship in addition to that of the primary deleted object; this is called a *cascading delete*.
+
+As stated before, the purpose of delete is strictly to issue DELETE statements to the database. It does not affect the in-memory structure of objects, other than changing the identifying attributes on objects, such as setting foreign key identifiers on updated rows to None. It has no effect on the status of references between object instances, nor any effect on the Python garbage-collection status of objects.
+
+#### Clear {@name=clear}
+
+To clear out the current thread's UnitOfWork, which has the effect of discarding the Identity Map and the lists of all objects that have been modified, just issue a clear:
+
+ {python}
+ # via module
+ objectstore.clear()
+
+ # or via Session
+ objectstore.get_session().clear()
+
+This is the easiest way to "start fresh", as in a web application that wants to have a newly loaded graph of objects on each request. Any object instances created before the clear operation should either be discarded or at least not used with any Mapper or Unit Of Work operations (with the exception of `import_instance()`), as they no longer have any relationship to the current Unit of Work, and their behavior with regards to the current session is undefined.
+
+#### Refresh / Expire {@name=refreshexpire}
+
+To assist with the Unit of Work's "sticky" behavior, individual objects can have all of their attributes immediately re-loaded from the database, or marked as "expired" which will cause a re-load to occur upon the next access of any of the object's mapped attributes. This includes all relationships, so lazy-loaders will be re-initialized, eager relationships will be repopulated. Any changes marked on the object are discarded:
+
+ {python}
+ # immediately re-load attributes on obj1, obj2
+ session.refresh(obj1, obj2)
+
+ # expire objects obj1, obj2, attributes will be reloaded
+ # on the next access:
+ session.expire(obj1, obj2, obj3)
+
+#### Expunge {@name=expunge}
+
+Expunge simply removes all record of an object from the current Session. This includes the identity map, and all history-tracking lists:
+
+ {python}
+ session.expunge(obj1)
+
+Use `expunge` when youd like to remove an object altogether from memory, such as before calling `del` on it, which will prevent any "ghost" operations occuring when the session is committed.
+
+#### Import Instance {@name=import}
+
+The _instance_key attribute placed on object instances is designed to work with objects that are serialized into strings and brought back again. As it contains no references to internal structures or database connections, applications that use caches or session storage which require serialization (i.e. pickling) can store SQLAlchemy-loaded objects. However, as mentioned earlier, an object with a particular database identity is only allowed to exist uniquely within the current unit-of-work scope. So, upon deserializing such an object, it has to "check in" with the current Session. This is achieved via the `import_instance()` method:
+
+ {python}
+ # deserialize an object
+ myobj = pickle.loads(mystring)
+
+ # "import" it. if the objectstore already had this object in the
+ # identity map, then you get back the one from the current session.
+ myobj = session.import_instance(myobj)
+
+Note that the import_instance() function will either mark the deserialized object as the official copy in the current identity map, which includes updating its _instance_key with the current application's class instance, or it will discard it and return the corresponding object that was already present. Thats why its important to receive the return results from the method and use the result as the official object instance.
+
+#### Begin {@name=begin}
+
+The "scope" of the unit of work commit can be controlled further by issuing a begin(). A begin operation constructs a new UnitOfWork object and sets it as the currently used UOW. It maintains a reference to the original UnitOfWork as its "parent", and shares the same identity map of objects that have been loaded from the database within the scope of the parent UnitOfWork. However, the "new", "dirty", and "deleted" lists are empty. This has the effect that only changes that take place after the begin() operation get logged to the current UnitOfWork, and therefore those are the only changes that get commit()ted. When the commit is complete, the "begun" UnitOfWork removes itself and places the parent UnitOfWork as the current one again.
+The begin() method returns a transactional object, upon which you can call commit() or rollback(). *Only this transactional object controls the transaction* - commit() upon the Session will do nothing until commit() or rollback() is called upon the transactional object.
+
+ {python}
+ # modify an object
+ myobj1.foo = "something new"
+
+ # begin
+ trans = session.begin()
+
+ # modify another object
+ myobj2.lala = "something new"
+
+ # only 'myobj2' is saved
+ trans.commit()
+
+begin/commit supports the same "nesting" behavior as the SQLEngine (note this behavior is not the original "nested" behavior), meaning that many begin() calls can be made, but only the outermost transactional object will actually perform a commit(). Similarly, calls to the commit() method on the Session, which might occur in function calls within the transaction, will not do anything; this allows an external function caller to control the scope of transactions used within the functions.
+
+### Advanced UnitOfWork Management {@name=advscope}
+
+#### Nesting UnitOfWork in a Database Transaction {@name=transactionnesting}
+
+The UOW commit operation places its INSERT/UPDATE/DELETE operations within the scope of a database transaction controlled by a SQLEngine:
+
+ {python}
+ engine.begin()
+ try:
+ # run objectstore update operations
+ except:
+ engine.rollback()
+ raise
+ engine.commit()
+
+If you recall from the [dbengine_transactions](rel:dbengine_transactions) section, the engine's begin()/commit() methods support reentrant behavior. This means you can nest begin and commits and only have the outermost begin/commit pair actually take effect (rollbacks however, abort the whole operation at any stage). From this it follows that the UnitOfWork commit operation can be nested within a transaction as well:
+
+ {python}
+ engine.begin()
+ try:
+ # perform custom SQL operations
+ objectstore.commit()
+ # perform custom SQL operations
+ except:
+ engine.rollback()
+ raise
+ engine.commit()
+
+#### Per-Object Sessions {@name=object}
+
+Sessions can be created on an ad-hoc basis and used for individual groups of objects and operations. This has the effect of bypassing the normal thread-local Session and explicitly using a particular Session:
+
+ {python}
+ # make a new Session with a global UnitOfWork
+ s = objectstore.Session()
+
+ # make objects bound to this Session
+ x = MyObj(_sa_session=s)
+
+ # perform mapper operations bound to this Session
+ # (this function coming soon)
+ r = MyObj.mapper.using(s).select_by(id=12)
+
+ # get the session that corresponds to an instance
+ s = objectstore.get_session(x)
+
+ # commit
+ s.commit()
+
+ # perform a block of operations with this session set within the current scope
+ objectstore.push_session(s)
+ try:
+ r = mapper.select_by(id=12)
+ x = new MyObj()
+ objectstore.commit()
+ finally:
+ objectstore.pop_session()
+
+##### Nested Transaction Sessions {@name=nested}
+
+Sessions also now support a "nested transaction" feature whereby a second Session can use a different database connection. This can be used inside of a larger database transaction to issue commits to the database that will be committed independently of the larger transaction's status:
+
+ {python}
+ engine.begin()
+ try:
+ a = MyObj()
+ b = MyObj()
+
+ sess = Session(nest_on=engine)
+ objectstore.push_session(sess)
+ try:
+ c = MyObj()
+ objectstore.commit() # will commit "c" to the database,
+ # even if the external transaction rolls back
+ finally:
+ objectstore.pop_session()
+
+ objectstore.commit() # commit "a" and "b" to the database
+ engine.commit()
+ except:
+ engine.rollback()
+ raise
+
+#### Custom Session Objects/Custom Scopes {@name=scope}
+
+For users who want to make their own Session subclass, or replace the algorithm used to return scoped Session objects (i.e. the objectstore.get_session() method):
+
+ {python title="Create a Session"}
+ # make a new Session
+ s = objectstore.Session()
+
+ # set it as the current thread-local session
+ objectstore.session_registry.set(s)
+
+ {python title="Create a custom Registry Algorithm"}
+ # set the objectstore's session registry to a different algorithm
+
+ def create_session():
+ """creates new sessions"""
+ return objectstore.Session()
+ def mykey():
+ """creates contextual keys to store scoped sessions"""
+ return "mykey"
+
+ objectstore.session_registry = sqlalchemy.util.ScopedRegistry(createfunc=create_session, scopefunc=mykey)
+
+#### Analyzing Object Commits {@name=logging}
+
+The objectstore module can log an extensive display of its "commit plans", which is a graph of its internal representation of objects before they are committed to the database. To turn this logging on:
+
+ {python}
+ # make an engine with echo_uow
+ engine = create_engine('myengine...', echo_uow=True)
+
+Commits will then dump to the standard output displays like the following:
+
+ {code}
+ Task dump:
+ UOWTask(6034768) 'User/users/6015696'
+ |
+ |- Save elements
+ |- Save: UOWTaskElement(6034800): User(6016624) (save)
+ |
+ |- Save dependencies
+ |- UOWDependencyProcessor(6035024) 'addresses' attribute on saved User's (UOWTask(6034768) 'User/users/6015696')
+ | |-UOWTaskElement(6034800): User(6016624) (save)
+ |
+ |- Delete dependencies
+ |- UOWDependencyProcessor(6035056) 'addresses' attribute on User's to be deleted (UOWTask(6034768) 'User/users/6015696')
+ | |-(no objects)
+ |
+ |- Child tasks
+ |- UOWTask(6034832) 'Address/email_addresses/6015344'
+ | |
+ | |- Save elements
+ | |- Save: UOWTaskElement(6034864): Address(6034384) (save)
+ | |- Save: UOWTaskElement(6034896): Address(6034256) (save)
+ | |----
+ |
+ |----
+
+The above graph can be read straight downwards to determine the order of operations. It indicates "save User 6016624, process each element in the 'addresses' list on User 6016624, save Address 6034384, Address 6034256".
+
return element\r
\r
CODE_BLOCK = 'formatting.myt:code'\r
-CODE_BLOCK_MARKER = '{python}'\r
DOCTEST_DIRECTIVES = re.compile(r'#\s*doctest:\s*[+-]\w+(,[+-]\w+)*\s*$', re.M)\r
LINK = 'formatting.myt:link'\r
LINK_MARKER = 'rel:'\r
Note: also remove all doctest directives\r
"""\r
parent = get_parent_map(tree)\r
- keyword = CODE_BLOCK_MARKER\r
\r
- def replace_pre_with_myt(pre, text):\r
+ def replace_pre_with_myt(pre, text, type=None, title=None):\r
text = re.sub(DOCTEST_DIRECTIVES, '', text)\r
# process '>>>' to have quotes around it, to work with the myghty python\r
# syntax highlighter which uses the tokenize module\r
sqlre = re.compile(r'{sql}(.*?)((?:SELECT|INSERT|DELETE|UPDATE|CREATE|DROP).*?)\n\s*(\n|$)', re.S)\r
text = sqlre.sub(r"<&formatting.myt:poplink&>\1\n<&|formatting.myt:codepopper, link='sql'&>\2</&>\n\n", text)\r
\r
+ sqlre2 = re.compile(r'{opensql}(.*?)((?:SELECT|INSERT|DELETE|UPDATE|CREATE|DROP).*?)\n\s*(\n|$)', re.S)\r
+ text = sqlre2.sub(r"<&|formatting.myt:poppedcode &>\1\n\2</&>\n\n", text)\r
+\r
pre_parent = parent[pre]\r
- tag = MyghtyTag(CODE_BLOCK)\r
+ if type == 'python':\r
+ syntype = 'python'\r
+ else:\r
+ syntype = None\r
+ if title is not None:\r
+ tag = MyghtyTag(CODE_BLOCK, {'title':title, 'syntaxtype':syntype})\r
+ else:\r
+ tag = MyghtyTag(CODE_BLOCK, {'syntaxtype':syntype})\r
tag.text = text\r
tag.tail = pre.tail\r
pre_parent[index(pre_parent, pre)] = tag\r
\r
for precode in tree.findall('.//pre/code'):\r
- if precode.text.lstrip().startswith(keyword):\r
+ m = re.match(r'\{(python|code)(?: title="(.*?)"){0,1}\}', precode.text.lstrip())\r
+ if m:\r
+ code = m.group(1)\r
+ title = m.group(2)\r
text = precode.text.lstrip()\r
- text = text.replace(keyword, '', 1).lstrip()\r
- replace_pre_with_myt(parent[precode], text)\r
+ text = re.sub(r'{(python|code).*?}(\n\s*)?', '', text)\r
+ replace_pre_with_myt(parent[precode], text, type=code, title=title)\r
elif precode.text.lstrip().startswith('>>> '):\r
replace_pre_with_myt(parent[precode], precode.text)\r
\r
print inname, '->', outname\r
input = file(inname).read()\r
html = markdown.markdown(input)\r
+ file(inname[:-3] + "html", 'w').write(html)\r
myt = html2myghtydoc(html)\r
- #file(inname[:-3] + "html", 'w').write(html)\r
file(outname, 'w').write(myt)\r