From: Daniele Varrazzo Date: Thu, 31 Dec 2020 03:27:49 +0000 (+0100) Subject: Added AdaptersMap.get_loader(), get_dumper() methods X-Git-Tag: 3.0.dev0~214 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ae2bc955658ee1c65eb5669fdbc14b5137477171;p=thirdparty%2Fpsycopg.git Added AdaptersMap.get_loader(), get_dumper() methods Internal refactoring to keep separate maps for different formats rather than using tuples as keys, both in the AdaptersMap and Transformer. Dumper.src renamed to cls. --- diff --git a/docs/adaptation.rst b/docs/adaptation.rst index 0a560b70c..54f7be1d9 100644 --- a/docs/adaptation.rst +++ b/docs/adaptation.rst @@ -42,18 +42,16 @@ be customised for specific needs within the same application: in order to do so you can use the *context* parameter of `Dumper.register()` and `Loader.register()`. +When a `!Connection` is created, it inherits the global adapters +configuration; when a `!Cursor` is created it inherits its `!Connection` +configuration. + .. note:: `!register()` is a class method on the base class, so if you subclass `!Dumper` or `!Loader` you should call the ``.register()`` on the class you created. -If the data type has also a distinct binary format which you may want to use -from psycopg (as documented in :ref:`binary-data`) you may want to implement -binary loaders and dumpers, whose only difference from text dumper is that -they must be registered using ``format=Format.BINARY`` in `Dumper.register()` -and `Loader.register()`. - .. admonition:: TODO - Example: infinity date customisation @@ -66,19 +64,22 @@ Dumpers and loaders life cycle Registering dumpers and loaders will instruct `!psycopg3` to use them in the queries to follow, in the context where they have been registered. -When a query is performed, a `Transformer` object will be used to instantiate -dumpers and loaders as requested and to dispatch the values to convert -to the right instance: +When a query is performed on a `!Cursor`, a `Transformer` object is created +as a local context to manage conversions during the query, instantiating the +required dumpers and loaders and dispatching the values to convert to the +right instance. -- The `!Trasformer` will look up the most specific adapter: one registered on - the `~psycopg3.Cursor` if available, then one registered on the - `~psycopg3.Connection`, finally a global one. +- The `!Trasformer` copies the adapters configuration from the `!Cursor`, thus + inheriting all the changes made to the global configuration, the current + `!Connection`, the `!Cursor`. -- For every Python type passed as query argument there will be a `!Dumper` - instantiated. All the objects of the same type will use the same loader. +- For every Python type passed as query argument, the `!Transformer` will + instantiate a `!Dumper`. All the objects of the same type will be converted + by the same dumper. -- For every OID returned by a query there will be a `!Loader` instantiated. - All the values with the same OID will be converted by the same loader. +- For every OID returned by the query, the `!Transformer` will instantiate a + `!Loader`. All the values with the same OID will be converted by the same + loader. - Recursive types (e.g. Python lists, PostgreSQL arrays and composite types) will use the same adaptation rules. @@ -100,10 +101,16 @@ Objects involved in types adaptation move to API section + +.. autoclass:: Format + :members: + + .. autoclass:: Dumper(src, context=None) This is an abstract base class: subclasses *must* implement the `dump()` - method. They *may* implement `oid` (as attribute or property) in order to + method and specify the `format`. + They *may* implement `oid` (as attribute or property) in order to override the oid type oid; if not PostgreSQL will try to infer the type from the context, but this may fail in some contexts and may require a cast. @@ -115,6 +122,13 @@ Objects involved in types adaptation be possible to know the connection encoding or the server date format. :type context: `~psycopg3.Connection`, `~psycopg3.Cursor`, or `Transformer` + .. attribute:: format + :type: Format + + The format this class dumps, `~Format.TEXT` or `~Format.BINARY`. + This is a class attribute. + + .. automethod:: dump The format returned by dump shouldn't contain quotes or escaped @@ -143,7 +157,7 @@ Objects involved in types adaptation Document how to find type OIDs in a database. - .. automethod:: register(src, context=None, format=Format.TEXT) + .. automethod:: register(src, context=None) You should call this method on the `Dumper` subclass you create, passing the Python type you want to dump as *src*. @@ -153,8 +167,6 @@ Objects involved in types adaptation :param context: Where the dumper should be used. If `!None` the dumper will be used globally. :type context: `~psycopg3.Connection`, `~psycopg3.Cursor`, or `Transformer` - :param format: Register the dumper for text or binary adaptation - :type format: `~psycopg3.pq.Format` If *src* is specified as string it will be lazy-loaded, so that it will be possible to register it without importing it before. In this @@ -165,7 +177,7 @@ Objects involved in types adaptation .. autoclass:: Loader(oid, context=None) This is an abstract base class: subclasses *must* implement the `load()` - method. + method and specify a `format`. :param oid: The type that will be managed by this dumper. :type oid: int @@ -174,9 +186,15 @@ Objects involved in types adaptation be possible to know the connection encoding or the server date format. :type context: `~psycopg3.Connection`, `~psycopg3.Cursor`, or `Transformer` + .. attribute:: format + :type: Format + + The format this class can load, `~Format.TEXT` or `~Format.BINARY`. + This is a class attribute. + .. automethod:: load - .. automethod:: register(oid, context=None, format=Format.TEXT) + .. automethod:: register(oid, context=None) You should call this method on the `Loader` subclass you create, passing the OID of the type you want to load as *oid* parameter. @@ -186,8 +204,6 @@ Objects involved in types adaptation :param context: Where the loader should be used. If `!None` the loader will be used globally. :type context: `~psycopg3.Connection`, `~psycopg3.Cursor`, or `Transformer` - :param format: Register the loader for text or binary adaptation - :type format: `~psycopg3.pq.Format` .. autoclass:: Transformer(context=None) diff --git a/docs/pq.rst b/docs/pq.rst index a0d3265d4..ff574a650 100644 --- a/docs/pq.rst +++ b/docs/pq.rst @@ -116,10 +116,6 @@ Enumerations .. seealso:: :pq:`PQtransactionStatus` for a description of these states. -.. autoclass:: Format - :members: - - .. autoclass:: ExecStatus :members: diff --git a/psycopg3/psycopg3/_transform.py b/psycopg3/psycopg3/_transform.py index a73881496..4926a0558 100644 --- a/psycopg3/psycopg3/_transform.py +++ b/psycopg3/psycopg3/_transform.py @@ -44,10 +44,12 @@ class Transformer(AdaptContext): self._connection = None # mapping class, fmt -> Dumper instance - self._dumpers_cache: Dict[Tuple[type, Format], "Dumper"] = {} + self._dumpers_cache: Tuple[Dict[type, "Dumper"], Dict[type, "Dumper"]] + self._dumpers_cache = ({}, {}) # mapping oid, fmt -> Loader instance - self._loaders_cache: Dict[Tuple[int, Format], "Loader"] = {} + self._loaders_cache: Tuple[Dict[int, "Loader"], Dict[int, "Loader"]] + self._loaders_cache = ({}, {}) # sequence of load functions from value to python # the length of the result columns @@ -97,37 +99,17 @@ class Transformer(AdaptContext): # Fast path: return a Dumper class already instantiated from the same type cls = type(obj) try: - return self._dumpers_cache[cls, format] + return self._dumpers_cache[format][cls] except KeyError: pass - # We haven't seen this type in this query yet. Look for an adapter - # in the current context (which was grown from more generic ones). - # Also look for superclasses: if you can adapt a type you should be - # able to adapt its subtypes, otherwise Liskov is sad. - dmap = self._adapters._dumpers - for scls in cls.__mro__: - dumper_class = dmap.get((scls, format)) - if not dumper_class: - continue - - dumper = self._dumpers_cache[cls, format] = dumper_class(cls, self) - return dumper - - # If the adapter is not found, look for its name as a string - for scls in cls.__mro__: - fqn = f"{cls.__module__}.{scls.__qualname__}" - dumper_class = dmap.get((fqn, format)) - if dumper_class is None: - continue - - dmap[cls, format] = dumper_class - dumper = self._dumpers_cache[cls, format] = dumper_class(cls, self) - return dumper + dumper_class = self._adapters.get_dumper(cls, format) + if dumper_class: + d = self._dumpers_cache[format][cls] = dumper_class(cls, self) + return d raise e.ProgrammingError( - f"cannot adapt type {type(obj).__name__}" - f" to format {Format(format).name}" + f"cannot adapt type {cls.__name__} to format {Format(format).name}" ) def load_rows(self, row0: int, row1: int) -> Sequence[Tuple[Any, ...]]: @@ -177,14 +159,15 @@ class Transformer(AdaptContext): ) def get_loader(self, oid: int, format: Format) -> "Loader": - key = (oid, format) try: - return self._loaders_cache[key] + return self._loaders_cache[format][oid] except KeyError: pass - loader_cls = self._adapters._loaders.get(key) + loader_cls = self._adapters.get_loader(oid, format) if not loader_cls: - loader_cls = self._adapters._loaders[INVALID_OID, format] - loader = self._loaders_cache[key] = loader_cls(oid, self) + loader_cls = self._adapters.get_loader(INVALID_OID, format) + if not loader_cls: + raise e.InterfaceError("unknown oid loader not found") + loader = self._loaders_cache[format][oid] = loader_cls(oid, self) return loader diff --git a/psycopg3/psycopg3/adapt.py b/psycopg3/psycopg3/adapt.py index d9f6e1754..888fcdfeb 100644 --- a/psycopg3/psycopg3/adapt.py +++ b/psycopg3/psycopg3/adapt.py @@ -5,13 +5,13 @@ Entry point into the adaptation system. # Copyright (C) 2020 The Psycopg Team from abc import ABC, abstractmethod -from typing import Any, Callable, Optional, Type, TYPE_CHECKING, Union - +from typing import Any, Dict, Callable, List, Optional, Type, Union +from typing import TYPE_CHECKING from . import pq from . import proto from .pq import Format as Format from .oids import builtins, TEXT_OID -from .proto import DumpersMap, DumperType, LoadersMap, LoaderType, AdaptContext +from .proto import AdaptContext if TYPE_CHECKING: from .connection import BaseConnection @@ -19,7 +19,7 @@ if TYPE_CHECKING: class Dumper(ABC): """ - Convert Python object of the type *src* to PostgreSQL representation. + Convert Python object of the type *cls* to PostgreSQL representation. """ format: Format @@ -29,8 +29,8 @@ class Dumper(ABC): # the subclass overrides it in init. _oid: int = 0 - def __init__(self, src: type, context: Optional[AdaptContext] = None): - self.src = src + def __init__(self, cls: type, context: Optional[AdaptContext] = None): + self.cls = cls self.connection = context.connection if context else None self.oid = self._oid @@ -64,25 +64,25 @@ class Dumper(ABC): @classmethod def register( - cls, src: Union[type, str], context: Optional[AdaptContext] = None + this_cls, cls: Union[type, str], context: Optional[AdaptContext] = None ) -> None: """ - Configure *context* to use this dumper to convert object of type *src*. + Configure *context* to use this dumper to convert object of type *cls*. """ adapters = context.adapters if context else global_adapters - adapters.register_dumper(src, cls) + adapters.register_dumper(cls, this_cls) @classmethod def builtin( cls, *types: Union[type, str] - ) -> Callable[[DumperType], DumperType]: + ) -> Callable[[Type["Dumper"]], Type["Dumper"]]: """ Decorator to mark a dumper class as default for a builtin type. """ - def builtin_(dumper: DumperType) -> DumperType: - for src in types: - dumper.register(src) + def builtin_(dumper: Type["Dumper"]) -> Type["Dumper"]: + for cls in types: + dumper.register(cls) return dumper return builtin_ @@ -118,16 +118,16 @@ class Loader(ABC): @classmethod def builtin( cls, *types: Union[int, str] - ) -> Callable[[LoaderType], LoaderType]: + ) -> Callable[[Type["Loader"]], Type["Loader"]]: """ Decorator to mark a loader class as default for a builtin type. """ - def builtin_(loader: LoaderType) -> LoaderType: - for src in types: - if isinstance(src, str): - src = builtins[src].oid - loader.register(src) + def builtin_(loader: Type["Loader"]) -> Type["Loader"]: + for cls in types: + if isinstance(cls, str): + cls = builtins[cls].oid + loader.register(cls) return loader return builtin_ @@ -143,37 +143,38 @@ class AdaptersMap: is cheap: a copy is made only on customisation. """ - _dumpers: DumpersMap - _loaders: LoadersMap + _dumpers: List[Dict[Union[type, str], Type["Dumper"]]] + _loaders: List[Dict[int, Type["Loader"]]] def __init__(self, extend: Optional["AdaptersMap"] = None): if extend: - self._dumpers = extend._dumpers - self._own_dumpers = False - self._loaders = extend._loaders - self._own_loaders = False + self._dumpers = extend._dumpers[:] + self._own_dumpers = [False, False] + self._loaders = extend._loaders[:] + self._own_loaders = [False, False] else: - self._dumpers = {} - self._own_dumpers = True - self._loaders = {} - self._own_loaders = True + self._dumpers = [{}, {}] + self._own_dumpers = [True, True] + self._loaders = [{}, {}] + self._own_loaders = [True, True] def register_dumper( - self, src: Union[type, str], dumper: Type[Dumper] + self, cls: Union[type, str], dumper: Type[Dumper] ) -> None: """ - Configure the context to use *dumper* to convert object of type *src*. + Configure the context to use *dumper* to convert object of type *cls*. """ - if not isinstance(src, (str, type)): + if not isinstance(cls, (str, type)): raise TypeError( - f"dumpers should be registered on classes, got {src} instead" + f"dumpers should be registered on classes, got {cls} instead" ) - if not self._own_dumpers: - self._dumpers = self._dumpers.copy() - self._own_dumpers = True + fmt = dumper.format + if not self._own_dumpers[fmt]: + self._dumpers[fmt] = self._dumpers[fmt].copy() + self._own_dumpers[fmt] = True - self._dumpers[src, dumper.format] = dumper + self._dumpers[fmt][cls] = dumper def register_loader(self, oid: int, loader: Type[Loader]) -> None: """ @@ -184,11 +185,43 @@ class AdaptersMap: f"loaders should be registered on oid, got {oid} instead" ) - if not self._own_loaders: - self._loaders = self._loaders.copy() - self._own_loaders = True + fmt = loader.format + if not self._own_loaders[fmt]: + self._loaders[fmt] = self._loaders[fmt].copy() + self._own_loaders[fmt] = True + + self._loaders[fmt][oid] = loader + + def get_dumper(self, cls: type, format: Format) -> Optional[Type[Dumper]]: + """ + Return the dumper class for the given type and format. + + Return None if not found. + """ + dumpers = self._dumpers[format] + + # Look for the right class, including looking at superclasses + for scls in cls.__mro__: + if scls in dumpers: + return dumpers[scls] + + # If the adapter is not found, look for its name as a string + for scls in cls.__mro__: + fqn = scls.__module__ + "." + scls.__qualname__ + if fqn in dumpers: + # Replace the class name with the class itself + d = dumpers[scls] = dumpers.pop(fqn) + return d + + return None - self._loaders[oid, loader.format] = loader + def get_loader(self, oid: int, format: Format) -> Optional[Type[Loader]]: + """ + Return the loader class for the given oid and format. + + Return None if not found. + """ + return self._loaders[format].get(oid) global_adapters = AdaptersMap() diff --git a/psycopg3/psycopg3/pq/_enums.py b/psycopg3/psycopg3/pq/_enums.py index e850e8b31..d02f940b8 100644 --- a/psycopg3/psycopg3/pq/_enums.py +++ b/psycopg3/psycopg3/pq/_enums.py @@ -182,10 +182,10 @@ class DiagnosticField(IntEnum): class Format(IntEnum): """ - The format of a query argument or return value. + Enum representing the format of a query argument or return value. """ - __module__ = "psycopg3.pq" + __module__ = "psycopg3.adapt" TEXT = 0 """Text parameter.""" diff --git a/psycopg3/psycopg3/proto.py b/psycopg3/psycopg3/proto.py index 9234c7211..f7a717297 100644 --- a/psycopg3/psycopg3/proto.py +++ b/psycopg3/psycopg3/proto.py @@ -4,8 +4,8 @@ Protocol objects representing different implementations of the same classes. # Copyright (C) 2020 The Psycopg Team -from typing import Any, Callable, Dict, Generator, Mapping -from typing import Optional, Sequence, Tuple, Type, TypeVar, Union +from typing import Any, Callable, Generator, Mapping +from typing import Optional, Sequence, Tuple, TypeVar, Union from typing import TYPE_CHECKING from typing_extensions import Protocol @@ -44,12 +44,7 @@ Wait states. # Adaptation types DumpFunc = Callable[[Any], bytes] -DumperType = Type["Dumper"] -DumpersMap = Dict[Tuple[Union[type, str], Format], DumperType] - LoadFunc = Callable[[bytes], Any] -LoaderType = Type["Loader"] -LoadersMap = Dict[Tuple[int, Format], LoaderType] # TODO: Loader, Dumper should probably become protocols # as there are both C and a Python implementation diff --git a/psycopg3/psycopg3/types/array.py b/psycopg3/psycopg3/types/array.py index ac302d3af..c2dc45eaf 100644 --- a/psycopg3/psycopg3/types/array.py +++ b/psycopg3/psycopg3/types/array.py @@ -18,8 +18,8 @@ class BaseListDumper(Dumper): _oid = TEXT_ARRAY_OID - def __init__(self, src: type, context: Optional[AdaptContext] = None): - super().__init__(src, context) + def __init__(self, cls: type, context: Optional[AdaptContext] = None): + super().__init__(cls, context) self._tx = Transformer(context) def _get_array_oid(self, base_oid: int) -> int: @@ -117,7 +117,7 @@ class ListBinaryDumper(BaseListDumper): oid = 0 def calc_dims(L: List[Any]) -> None: - if isinstance(L, self.src): + if isinstance(L, self.cls): if not L: raise e.DataError("lists cannot contain empty lists") dims.append(len(L)) @@ -144,7 +144,7 @@ class ListBinaryDumper(BaseListDumper): data.append(b"\xff\xff\xff\xff") else: for item in L: - if not isinstance(item, self.src): + if not isinstance(item, self.cls): raise e.DataError( "nested lists have inconsistent depths" ) diff --git a/psycopg3_c/psycopg3_c/_psycopg3/transform.pyx b/psycopg3_c/psycopg3_c/_psycopg3/transform.pyx index 50b3d47d3..8ac9fa4b7 100644 --- a/psycopg3_c/psycopg3_c/_psycopg3/transform.pyx +++ b/psycopg3_c/psycopg3_c/_psycopg3/transform.pyx @@ -52,10 +52,8 @@ cdef class Transformer: cdef readonly object connection cdef readonly object adapters - # TODO: the dumpers should have the same tweaking of the loaders - cdef dict _adapters_loaders - cdef dict _adapters_dumpers - cdef dict _dumpers_cache + cdef dict _text_dumpers + cdef dict _binary_dumpers cdef dict _text_loaders cdef dict _binary_loaders cdef pq.PGresult _pgresult @@ -71,11 +69,9 @@ cdef class Transformer: self.adapters = global_adapters self.connection = None - self._adapters_loaders = self.adapters._loaders - self._adapters_dumpers = self.adapters._dumpers - - # mapping class, fmt -> Dumper instance - self._dumpers_cache: Dict[Tuple[type, Format], "Dumper"] = {} + # mapping class -> Dumper instance (text, binary) + self._text_dumpers = {} + self._binary_dumpers = {} # mapping oid -> Loader instance (text, binary) self._text_loaders = {} @@ -166,39 +162,23 @@ cdef class Transformer: def get_dumper(self, obj: Any, format: Format) -> "Dumper": # Fast path: return a Dumper class already instantiated from the same type + cdef dict cache + cdef PyObject *ptr + cls = type(obj) - key = (cls, format) - cdef PyObject *ptr = PyDict_GetItem(self._dumpers_cache, key) + cache = self._binary_dumpers if format else self._text_dumpers + ptr = PyDict_GetItem(cache, cls) if ptr != NULL: return ptr - # We haven't seen this type in this query yet. Look for an adapter - # in the current context (which was grown from more generic ones). - # Also look for superclasses: if you can adapt a type you should be - # able to adapt its subtypes, otherwise Liskov is sad. - cdef dict dmap = self.adapters._dumpers - for scls in cls.__mro__: - dumper_class = dmap.get((scls, format)) - if not dumper_class: - continue - - self._dumpers_cache[key] = dumper = dumper_class(cls, self) - return dumper - - # If the adapter is not found, look for its name as a string - for scls in cls.__mro__: - fqn = f"{cls.__module__}.{scls.__qualname__}" - dumper_class = dmap.get((fqn, format)) - if dumper_class is None: - continue - - dmap[key] = dumper_class - self._dumpers_cache[key] = dumper = dumper_class(cls, self) - return dumper + dumper_class = self.adapters.get_dumper(cls, format) + if dumper_class: + d = dumper_class(cls, self) + cache[cls] = d + return d raise e.ProgrammingError( - f"cannot adapt type {type(obj).__name__}" - f" to format {Format(format).name}" + f"cannot adapt type {cls.__name__} to format {Format(format).name}" ) def load_rows(self, int row0, int row1) -> Sequence[Tuple[Any, ...]]: @@ -338,25 +318,17 @@ cdef class Transformer: cdef dict cache cdef PyObject *ptr - if format == 0: - cache = self._text_loaders - else: - cache = self._binary_loaders - + cache = self._binary_loaders if format else self._text_loaders ptr = PyDict_GetItem(cache, oid) if ptr != NULL: return ptr - key = (oid, format) - ptr = PyDict_GetItem(self._adapters_loaders, (oid, format)) - if ptr != NULL: - loader_cls = ptr - else: - ptr = PyDict_GetItem( - self.adapters._loaders, (oids.INVALID_OID, format)) - if ptr == NULL: - raise e.InterfaceError(f"unknown oid loader not found") - loader_cls = ptr + loader_cls = self.adapters.get_loader(oid, format) + if not loader_cls: + loader_cls = self.adapters.get_loader(oids.INVALID_OID, format) + if not loader_cls: + raise e.InterfaceError("unknown oid loader not found") + loader = loader_cls(oid, self) PyDict_SetItem(cache, oid, loader) return loader diff --git a/tests/types/test_composite.py b/tests/types/test_composite.py index dc0bf156e..2d781ef09 100644 --- a/tests/types/test_composite.py +++ b/tests/types/test_composite.py @@ -211,25 +211,23 @@ def test_load_composite_factory(conn, testcomp, fmt_out): assert isinstance(res[0].baz, float) -def test_register_scope(conn): +def test_register_scope(conn, testcomp): info = CompositeInfo.fetch(conn, "testcomp") info.register() for fmt in (Format.TEXT, Format.BINARY): for oid in (info.oid, info.array_oid): - assert global_adapters._loaders.pop((oid, fmt)) + assert global_adapters._loaders[fmt].pop(oid) cur = conn.cursor() info.register(cur) for fmt in (Format.TEXT, Format.BINARY): for oid in (info.oid, info.array_oid): - key = oid, fmt - assert key not in global_adapters._loaders - assert key not in conn.adapters._loaders - assert key in cur.adapters._loaders + assert oid not in global_adapters._loaders[fmt] + assert oid not in conn.adapters._loaders[fmt] + assert oid in cur.adapters._loaders[fmt] info.register(conn) for fmt in (Format.TEXT, Format.BINARY): for oid in (info.oid, info.array_oid): - key = oid, fmt - assert key not in global_adapters._loaders - assert key in conn.adapters._loaders + assert oid not in global_adapters._loaders[fmt] + assert oid in conn.adapters._loaders[fmt]