itemadapter

The ItemAdapter class is a wrapper for data container objects, providing a
common interface to handle objects of different types in an uniform manner,
regardless of their underlying implementation.

Currently supported types are:

scrapy.item.Item
dict
dataclass-based classes
attrs-based classes
pydantic-based classes

Additionally, interaction with arbitrary types is supported, by implementing
a pre-defined interface (see extending itemadapter).

Requirements

Python 3.9+, either the CPython implementation (default) or the PyPy
implementation
scrapy 2.2+: optional, needed to interact with
scrapy items
attrs 18.1.0+: optional, needed to
interact with attrs-based items
pydantic 1.8+: optional, needed to
interact with pydantic-based items

Installation

itemadapter is available on PyPI, it can be installed with pip:

pip install itemadapter

For attrs, pydantic and scrapy support, install the corresponding extra
to ensure that a supported version of the corresponding dependencies is
installed. For example:

pip install itemadapter[scrapy]

Mind that you can install multiple extras as needed. For example:

pip install itemadapter[attrs,pydantic,scrapy]

License

itemadapter is distributed under a BSD-3 license.

Basic usage

The following is a simple example using a dataclass object.
Consider the following type definition:

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: float
...     stock: int
>>>

An ItemAdapter object can be treated much like a dictionary:

>>> obj = InventoryItem(name='foo', price=20.5, stock=10)
>>> ItemAdapter.is_item(obj)
True
>>> adapter = ItemAdapter(obj)
>>> len(adapter)
3
>>> adapter["name"]
'foo'
>>> adapter.get("price")
20.5
>>>

The wrapped object is modified in-place:

>>> adapter["name"] = "bar"
>>> adapter.update({"price": 12.7, "stock": 9})
>>> adapter.item
InventoryItem(name='bar', price=12.7, stock=9)
>>> adapter.item is obj
True
>>>

Converting to dict

The ItemAdapter class provides the asdict method, which converts
nested items recursively. Consider the following example:

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class Price:
...     value: int
...     currency: str
>>> @dataclass
... class Product:
...     name: str
...     price: Price
>>>

>>> item = Product("Stuff", Price(42, "UYU"))
>>> adapter = ItemAdapter(item)
>>> adapter.asdict()
{'name': 'Stuff', 'price': {'value': 42, 'currency': 'UYU'}}
>>>

Note that just passing an adapter object to the dict built-in also works,
but it doesn't traverse the object recursively converting nested items:

>>> dict(adapter)
{'name': 'Stuff', 'price': Price(value=42, currency='UYU')}
>>>

API reference

Built-in adapters

The following adapters are included by default:

itemadapter.adapter.ScrapyItemAdapter: handles Scrapy items
itemadapter.adapter.DictAdapter: handles Python dictionaries
itemadapter.adapter.DataclassAdapter: handles dataclass objects
itemadapter.adapter.AttrsAdapter: handles attrs objects
itemadapter.adapter.PydanticAdapter: handles pydantic objects

class `itemadapter.adapter.ItemAdapter(item: Any)`

This is the main entrypoint for the package. Tipically, user code
wraps an item using this class, and proceeds to handle it with the provided interface.
ItemAdapter implements the
MutableMapping
interface, providing a dict-like API to manipulate data for the object it wraps
(which is modified in-place).

Attributes

class attribute `ADAPTER_CLASSES: Iterable`

Stores the currently registered adapter classes.

The order in which the adapters are registered is important. When an ItemAdapter object is
created for a specific item, the registered adapters are traversed in order and the first
adapter class to return True for the is_item class method is used for all subsequent
operations. The default order is the one defined in the
built-in adapters section.

The default implementation uses a
collections.deque
to support efficient addition/deletion of adapters classes to both ends, but if you are
deriving a subclass (see the section on extending itemadapter
for additional information), any other iterable (e.g. list, tuple) will work.

Methods

class method `is_item(item: Any) -> bool`

Return True if any of the registed adapters can handle the item
(i.e. if any of them returns True for its is_item method with
item as argument), False otherwise.

class method `is_item_class(item_class: type) -> bool`

Return True if any of the registered adapters can handle the item class
(i.e. if any of them returns True for its is_item_class method with
item_class as argument), False otherwise.

class method `get_field_meta_from_class(item_class: type, field_name: str) -> MappingProxyType`

Return a types.MappingProxyType
object, which is a read-only mapping with metadata about the given field. If the item class does not
support field metadata, or there is no metadata for the given field, an empty object is returned.

The returned value is taken from the following sources, depending on the item type:

scrapy.item.Field
for scrapy.item.Items
dataclasses.field.metadata
for dataclass-based items
attr.Attribute.metadata
for attrs-based items
pydantic.fields.FieldInfo
for pydantic-based items

class method `get_field_names_from_class(item_class: type) -> Optional[list[str]]`

Return a list with the names of all the fields defined for the item class.
If an item class doesn't support defining fields upfront, None is returned.

`get_field_meta(field_name: str) -> MappingProxyType`

Return metadata for the given field, if available. Unless overriden in a custom adapter class, by default
this method calls the adapter's get_field_meta_from_class method, passing the wrapped item's class.

`field_names() -> collections.abc.KeysView`

Return a keys view
with the names of all the defined fields for the item.

`asdict() -> dict`

Return a dict object with the contents of the adapter. This works slightly different than
calling dict(adapter), because it's applied recursively to nested items (if there are any).

function `itemadapter.utils.is_item(obj: Any) -> bool`

Return True if the given object belongs to (at least) one of the supported types,
False otherwise. This is an alias, using the itemadapter.adapter.ItemAdapter.is_item
class method is encouraged for better performance.

function `itemadapter.utils.get_field_meta_from_class(item_class: type, field_name: str) -> types.MappingProxyType`

Alias for itemadapter.adapter.ItemAdapter.get_field_meta_from_class

Metadata support

scrapy.item.Item, dataclass, attrs, and pydantic objects allow the definition of
arbitrary field metadata. This can be accessed through a
MappingProxyType
object, which can be retrieved from an item instance with
itemadapter.adapter.ItemAdapter.get_field_meta, or from an item class
with the itemadapter.adapter.ItemAdapter.get_field_meta_from_class
method (or its alias itemadapter.utils.get_field_meta_from_class).
The source of the data depends on the underlying type (see the docs for
ItemAdapter.get_field_meta_from_class).

`scrapy.item.Item` objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field(serializer=str)
...     value = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

`dataclass` objects

>>> from dataclasses import dataclass, field
>>> @dataclass
... class InventoryItem:
...     name: str = field(metadata={"serializer": str})
...     value: int = field(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

`attrs` objects

>>> import attr
>>> @attr.s
... class InventoryItem:
...     name = attr.ib(metadata={"serializer": str})
...     value = attr.ib(metadata={"serializer": int, "limit": 100})
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'serializer': <class 'str'>})
>>> adapter.get_field_meta("value")
mappingproxy({'serializer': <class 'int'>, 'limit': 100})
>>>

`pydantic` objects

>>> from pydantic import BaseModel, Field
>>> class InventoryItem(BaseModel):
...     name: str = Field(serializer=str)
...     value: int = Field(serializer=int, limit=100)
...
>>> adapter = ItemAdapter(InventoryItem(name="foo", value=10))
>>> adapter.get_field_meta("name")
mappingproxy({'default': PydanticUndefined, 'json_schema_extra': {'serializer': <class 'str'>}, 'repr': True})
>>> adapter.get_field_meta("value")
mappingproxy({'default': PydanticUndefined, 'json_schema_extra': {'serializer': <class 'int'>, 'limit': 100}, 'repr': True})
>>>

Extending `itemadapter`

This package allows to handle arbitrary item classes, by implementing an adapter interface:

class itemadapter.adapter.AdapterInterface(item: Any)

Abstract Base Class for adapters. An adapter that handles a specific type of item must
inherit from this class and implement the abstract methods defined on it. AdapterInterface
inherits from collections.abc.MutableMapping,
so all methods from the MutableMapping interface must be implemented as well.

class method is_item_class(cls, item_class: type) -> bool

Return True if the adapter can handle the given item class, False otherwise. Abstract (mandatory).
class method is_item(cls, item: Any) -> bool

Return True if the adapter can handle the given item, False otherwise.
The default implementation calls cls.is_item_class(item.__class__).
class method get_field_meta_from_class(cls, item_class: type) -> bool

Return metadata for the given item class and field name, if available.
By default, this method returns an empty MappingProxyType object. Please supply your
own method definition if you want to handle field metadata based on custom logic.
See the section on metadata support for additional information.
method get_field_meta(self, field_name: str) -> types.MappingProxyType

Return metadata for the given field name, if available. It's usually not necessary to
override this method, since the itemadapter.adapter.AdapterInterface base class
provides a default implementation that calls ItemAdapter.get_field_meta_from_class
with the wrapped item's class as argument.
See the section on metadata support for additional information.
method field_names(self) -> collections.abc.KeysView:

Return a dynamic view
of the item's field names. By default, this method returns the result of calling keys() on
the current adapter, i.e., its return value depends on the implementation of the methods from the
MutableMapping interface (more specifically, it depends on the return value of __iter__).

You might want to override this method if you want a way to get all fields for an item, whether or not
they are populated. For instance, Scrapy uses this method to define column names when exporting items to CSV.

Registering an adapter

Add your custom adapter class to the
itemadapter.adapter.ItemAdapter.ADAPTER_CLASSES class attribute in order to
handle custom item classes.

Example

pip install zyte-common-items

>>> from itemadapter.adapter import ItemAdapter
>>> from zyte_common_items import Item, ZyteItemAdapter
>>>
>>> ItemAdapter.ADAPTER_CLASSES.appendleft(ZyteItemAdapter)
>>> item = Item()
>>> adapter = ItemAdapter(item)
>>> adapter
<ItemAdapter for Item()>
>>>

Multiple adapter classes

If you need to have different handlers and/or priorities for different cases
you can subclass the ItemAdapter class and set the ADAPTER_CLASSES
attribute as needed:

Example

>>> from itemadapter.adapter import (
...     ItemAdapter,
...     AttrsAdapter,
...     DataclassAdapter,
...     DictAdapter,
...     PydanticAdapter,
...     ScrapyItemAdapter,
... )
>>> from scrapy.item import Item, Field
>>>
>>> class BuiltinTypesItemAdapter(ItemAdapter):
...     ADAPTER_CLASSES = [DictAdapter, DataclassAdapter]
...
>>> class ThirdPartyTypesItemAdapter(ItemAdapter):
...     ADAPTER_CLASSES = [AttrsAdapter, PydanticAdapter, ScrapyItemAdapter]
...
>>> class ScrapyItem(Item):
...     foo = Field()
...
>>> BuiltinTypesItemAdapter.is_item(dict())
True
>>> ThirdPartyTypesItemAdapter.is_item(dict())
False
>>> BuiltinTypesItemAdapter.is_item(ScrapyItem(foo="bar"))
False
>>> ThirdPartyTypesItemAdapter.is_item(ScrapyItem(foo="bar"))
True
>>>

More examples

`scrapy.item.Item` objects

>>> from scrapy.item import Item, Field
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(Item):
...     name = Field()
...     price = Field()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>

`dict`

>>> from itemadapter import ItemAdapter
>>> item = dict(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
{'name': 'bar', 'price': 5}
>>>

`dataclass` objects

>>> from dataclasses import dataclass
>>> from itemadapter import ItemAdapter
>>> @dataclass
... class InventoryItem:
...     name: str
...     price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>

`attrs` objects

>>> import attr
>>> from itemadapter import ItemAdapter
>>> @attr.s
... class InventoryItem:
...     name = attr.ib()
...     price = attr.ib()
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>

`pydantic` objects

>>> from pydantic import BaseModel
>>> from itemadapter import ItemAdapter
>>> class InventoryItem(BaseModel):
...     name: str
...     price: int
...
>>> item = InventoryItem(name="foo", price=10)
>>> adapter = ItemAdapter(item)
>>> adapter.item is item
True
>>> adapter["name"]
'foo'
>>> adapter["name"] = "bar"
>>> adapter["price"] = 5
>>> item
InventoryItem(name='bar', price=5)
>>>

Changelog

See the full changelog

Dependencies of itemadapter (3).
Dependency	Optional
attrs	true
pydantic	true
scrapy	true

Package profile

itemadapter

Releases

Release table

PyPI Downloads

Dependencies

Details

Project URLs

Dependencies

Package description