Horizontal Sharding — SQLAlchemy 1.1 Documentation

Release: 1.1.16 | Release Date: February 16, 2018

# SQLAlchemy 1.1 Documentation

# Horizontal Sharding

Horizontal sharding support.

Defines a rudimental ‘horizontal sharding’ system which allows a Session to distribute queries and persistence operations across multiple databases.

For a usage example, see the Horizontal Sharding example included in the source distribution.

## API Documentation

class `sqlalchemy.ext.horizontal_shard.` `ShardedSession`(shard_chooser, id_chooser, query_chooser, shards=None, query_cls=<class 'sqlalchemy.ext.horizontal_shard.ShardedQuery'>, **kwargs)

  Bases: `sqlalchemy.orm.session.Session`

  `__init__`(shard_chooser, id_chooser, query_chooser, shards=None, query_cls=<class 'sqlalchemy.ext.horizontal_shard.ShardedQuery'>, **kwargs)

    Construct a ShardedSession.

    Parameters: | shard_chooser – A callable which, passed a Mapper, a mapped instance, and possibly a SQL clause, returns a shard ID. This id may be based off of the attributes present within the object, or on some round-robin scheme. If the scheme is based on a selection, it should set whatever state on the instance to mark it in the future as participating in that shard. id_chooser – A callable, passed a query and a tuple of identity values, which should return a list of shard ids where the ID might reside. The databases will be queried in the order of this listing. query_chooser – For a given Query, returns the list of shard_ids where the query should be issued. Results from all shards returned will be combined together into a single listing. shards – A dictionary of string shard names to `Engine` objects.

  `connection`(mapper=None, instance=None, shard_id=None, **kwargs)

    Return a `Connection` object corresponding to this `Session` object’s transactional state.

    If this `Session` is configured with `autocommit=False`, either the `Connection` corresponding to the current transaction is returned, or if no transaction is in progress, a new one is begun and the `Connection` returned (note that no transactional state is established with the DBAPI until the first SQL statement is emitted).

    Alternatively, if this `Session` is configured with `autocommit=True`, an ad-hoc `Connection` is returned using `Engine.contextual_connect()` on the underlying `Engine`.

    Ambiguity in multi-bind or unbound `Session` objects can be resolved through any of the optional keyword arguments. This ultimately makes usage of the `get_bind()` method for resolution.

    Parameters: | bind – Optional `Engine` to be used as the bind. If this engine is already involved in an ongoing transaction, that connection will be used. This argument takes precedence over `mapper`, `clause`. mapper – Optional `mapper()` mapped class, used to identify the appropriate bind. This argument takes precedence over `clause`. clause – A `ClauseElement` (i.e. `select()`, `text()`, etc.) which will be used to locate a bind, if a bind cannot otherwise be identified. close_with_result – Passed to `Engine.connect()`, indicating the `Connection` should be considered “single use”, automatically closing when the first result set is closed. This flag only has an effect if this `Session` is configured with `autocommit=True` and does not already have a transaction in progress. execution_options – a dictionary of execution options that will be passed to `Connection.execution_options()`, when the connection is first procured only. If the connection is already present within the `Session`, a warning is emitted and the arguments are ignored. New in version 0.9.9. See also Setting Transaction Isolation Levels **kw – Additional keyword arguments are sent to `get_bind()`, allowing additional arguments to be passed to custom implementations of `get_bind()`.

  `get_bind`(mapper, shard_id=None, instance=None, clause=None, **kw)

    Return a “bind” to which this `Session` is bound.

    The “bind” is usually an instance of `Engine`, except in the case where the `Session` has been explicitly bound directly to a `Connection`.

    For a multiply-bound or unbound `Session`, the `mapper` or `clause` arguments are used to determine the appropriate bind to return.

    Note that the “mapper” argument is usually present when `Session.get_bind()` is called via an ORM operation such as a `Session.query()`, each individual INSERT/UPDATE/DELETE operation within a `Session.flush()`, call, etc.

    The order of resolution is:

    1. if mapper given and session.binds is present, locate a bind based on mapper.
    2. if clause given and session.binds is present, locate a bind based on `Table` objects found in the given clause present in session.binds.
    3. if session.bind is present, return that.
    4. if clause given, attempt to return a bind linked to the `MetaData` ultimately associated with the clause.
    5. if mapper given, attempt to return a bind linked to the `MetaData` ultimately associated with the `Table` or other selectable to which the mapper is mapped.
    6. No bind can be found, `UnboundExecutionError` is raised.

    Parameters: | mapper – Optional `mapper()` mapped class or instance of `Mapper`. The bind can be derived from a `Mapper` first by consulting the “binds” map associated with this `Session`, and secondly by consulting the `MetaData` associated with the `Table` to which the `Mapper` is mapped for a bind. clause – A `ClauseElement` (i.e. `select()`, `text()`, etc.). If the `mapper` argument is not present or could not produce a bind, the given expression construct will be searched for a bound element, typically a `Table` associated with bound `MetaData`.

class `sqlalchemy.ext.horizontal_shard.` `ShardedQuery`(*args, **kwargs)

  Bases: `sqlalchemy.orm.query.Query`

  `get`(ident, **kwargs)

    Return an instance based on the given primary key identifier, or `None` if not found.

    E.g.:

        my_user = session.query(User).get(5)

        some_object = session.query(VersionedFoo).get((5, 10))

    `get()` is special in that it provides direct access to the identity map of the owning `Session`. If the given primary key identifier is present in the local identity map, the object is returned directly from this collection and no SQL is emitted, unless the object has been marked fully expired. If not present, a SELECT is performed in order to locate the object.

    `get()` also will perform a check if the object is present in the identity map and marked as expired - a SELECT is emitted to refresh the object as well as to ensure that the row is still present. If not, `ObjectDeletedError` is raised.

    `get()` is only used to return a single mapped instance, not multiple instances or individual column constructs, and strictly on a single primary key value. The originating `Query` must be constructed in this way, i.e. against a single mapped entity, with no additional filtering criterion. Loading options via `options()` may be applied however, and will be used if the object is not yet locally present.

    A lazy-loading, many-to-one attribute configured by `relationship()`, using a simple foreign-key-to-primary-key criterion, will also use an operation equivalent to `get()` in order to retrieve the target value from the local identity map before querying the database. See Relationship Loading Techniques for further details on relationship loading.

    Parameters: | ident – A scalar or tuple value representing the primary key. For a composite primary key, the order of identifiers corresponds in most cases to that of the mapped `Table` object’s primary key columns. For a `mapper()` that was given the `primary key` argument during construction, the order of identifiers corresponds to the elements present in this collection.
    Returns: | The object instance, or `None`.

  `set_shard`(shard_id)

    return a new query, limited to a single shard ID.

    all subsequent operations with the returned query will be against the single shard regardless of other state.