======
Parsel

.. image:: https://github.com/scrapy/parsel/actions/workflows/tests.yml/badge.svg
:target: https://github.com/scrapy/parsel/actions/workflows/tests.yml
:alt: Tests

.. image:: https://img.shields.io/pypi/pyversions/parsel.svg
:target: https://github.com/scrapy/parsel/actions/workflows/tests.yml
:alt: Supported Python versions

.. image:: https://img.shields.io/pypi/v/parsel.svg
:target: https://pypi.python.org/pypi/parsel
:alt: PyPI Version

.. image:: https://img.shields.io/codecov/c/github/scrapy/parsel/master.svg
:target: https://codecov.io/github/scrapy/parsel?branch=master
:alt: Coverage report

Parsel is a BSD-licensed Python_ library to extract data from HTML_, JSON_, and
XML_ documents.

It supports:

CSS_ and XPath_ expressions for HTML and XML documents
JMESPath_ expressions for JSON documents
Regular expressions_

Find the Parsel online documentation at https://parsel.readthedocs.org.

Example (open online demo_):

.. code-block:: python

>>> from parsel import Selector
>>> text = """
        <html>
            <body>
                <h1>Hello, Parsel!</h1>
                <ul>
                    <li><a href="http://example.com">Link 1</a></li>
                    <li><a href="http://scrapy.org">Link 2</a></li>
                </ul>
                <script type="application/json">{"a": ["b", "c"]}</script>
            </body>
        </html>"""
>>> selector = Selector(text=text)
>>> selector.css('h1::text').get()
'Hello, Parsel!'
>>> selector.xpath('//h1/text()').re(r'\w+')
['Hello', 'Parsel']
>>> for li in selector.css('ul > li'):
...     print(li.xpath('.//@href').get())
http://example.com
http://scrapy.org
>>> selector.css('script::text').jmespath("a").get()
'b'
>>> selector.css('script::text').jmespath("a").getall()
['b', 'c']

.. _CSS: https://en.wikipedia.org/wiki/Cascading_Style_Sheets
.. _HTML: https://en.wikipedia.org/wiki/HTML
.. _JMESPath: https://jmespath.org/
.. _JSON: https://en.wikipedia.org/wiki/JSON
.. _open online demo: https://colab.research.google.com/drive/149VFa6Px3wg7S3SEnUqk--TyBrKplxCN#forceEdit=true&sandboxMode=true
.. _Python: https://www.python.org/
.. _regular expressions: https://docs.python.org/library/re.html
.. _XML: https://en.wikipedia.org/wiki/XML
.. _XPath: https://en.wikipedia.org/wiki/XPath

History

1.10.0 (2024-12-16)


* Removed support for Python 3.8.
* Added support for Python 3.13.
* Changed the default encoding name from ``"utf8"`` to ``"utf-8"`` everywhere.
  The former name is not supported in certain environments.
* CI fixes and improvements.

1.9.1 (2024-04-08)
~~~~~~~~~~~~~~~~~~

* Removed the dependency on ``pytest-runner``.
* Removed the obsolete ``Makefile``.

1.9.0 (2024-03-14)
~~~~~~~~~~~~~~~~~~

* Now requires ``cssselect >= 1.2.0`` (this minimum version was required since
  1.8.0 but that wasn't properly recorded)
* Removed support for Python 3.7
* Added support for Python 3.12 and PyPy 3.10
* Fixed an exception when calling ``__str__`` or ``__repr__`` on some JSON
  selectors
* Code formatted with ``black``
* CI fixes and improvements

1.8.1 (2023-04-18)
~~~~~~~~~~~~~~~~~~

* Remove a Sphinx reference from NEWS to fix the PyPI description
* Add a ``twine check`` CI check to detect such problems

1.8.0 (2023-04-18)
~~~~~~~~~~~~~~~~~~

* Add support for JMESPath: you can now create a selector for a JSON document
  and call ``Selector.jmespath()``. See `the documentation`_ for more
  information and examples.
* Selectors can now be constructed from ``bytes`` (using the ``body`` and
  ``encoding`` arguments) instead of ``str`` (using the ``text`` argument), so
  that there is no internal conversion from ``str`` to ``bytes`` and the memory
  usage is lower.
* Typing improvements
* The ``pkg_resources`` module (which was absent from the requirements) is no
  longer used
* Documentation build fixes
* New requirements:

  * ``jmespath``
  * ``typing_extensions`` (on Python 3.7)

 .. _the documentation: https://parsel.readthedocs.io/en/latest/usage.html

1.7.0 (2022-11-01)
~~~~~~~~~~~~~~~~~~

* Add PEP 561-style type information
* Support for Python 2.7, 3.5 and 3.6 is removed
* Support for Python 3.9-3.11 is added
* Very large documents (with deep nesting or long tag content) can now be
  parsed, and ``Selector`` now takes a new argument ``huge_tree`` to disable
  this
* Support for new features of cssselect 1.2.0 is added
* The ``Selector.remove()`` and ``SelectorList.remove()`` methods are
  deprecated and replaced with the new ``Selector.drop()`` and
  ``SelectorList.drop()`` methods which don't delete text after the dropped
  elements when used in the HTML mode.


1.6.0 (2020-05-07)
~~~~~~~~~~~~~~~~~~

* Python 3.4 is no longer supported
* New ``Selector.remove()`` and ``SelectorList.remove()`` methods to remove
  selected elements from the parsed document tree
* Improvements to error reporting, test coverage and documentation, and code
  cleanup


1.5.2 (2019-08-09)
~~~~~~~~~~~~~~~~~~

* ``Selector.remove_namespaces`` received a significant performance improvement
* The value of ``data`` within the printable representation of a selector
  (``repr(selector)``) now ends in ``...`` when truncated, to make the
  truncation obvious.
* Minor documentation improvements.


1.5.1 (2018-10-25)
~~~~~~~~~~~~~~~~~~

* ``has-class`` XPath function handles newlines and other separators
  in class names properly;
* fixed parsing of HTML documents with null bytes;
* documentation improvements;
* Python 3.7 tests are run on CI; other test improvements.


1.5.0 (2018-07-04)
~~~~~~~~~~~~~~~~~~

* New ``Selector.attrib`` and ``SelectorList.attrib`` properties which make
  it easier to get attributes of HTML elements.
* CSS selectors became faster: compilation results are cached
  (LRU cache is used for ``css2xpath``), so there is
  less overhead when the same CSS expression is used several times.
* ``.get()`` and ``.getall()`` selector methods are documented and recommended
  over ``.extract_first()`` and ``.extract()``.
* Various documentation tweaks and improvements.

One more change is that ``.extract()`` and  ``.extract_first()`` methods
are now implemented using ``.get()`` and ``.getall()``, not the other
way around, and instead of calling ``Selector.extract`` all other methods
now call ``Selector.get`` internally. It can be **backwards incompatible**
in case of custom Selector subclasses which override ``Selector.extract``
without doing the same for ``Selector.get``. If you have such Selector
subclass, make sure ``get`` method is also overridden. For example, this::

    class MySelector(parsel.Selector):
        def extract(self):
            return super().extract() + " foo"

should be changed to this::

    class MySelector(parsel.Selector):
        def get(self):
            return super().get() + " foo"
        extract = get


1.4.0 (2018-02-08)
~~~~~~~~~~~~~~~~~~

* ``Selector`` and ``SelectorList`` can't be pickled because
  pickling/unpickling doesn't work for ``lxml.html.HtmlElement``;
  parsel now raises TypeError explicitly instead of allowing pickle to
  silently produce wrong output. This is technically backwards-incompatible
  if you're using Python < 3.6.


1.3.1 (2017-12-28)
~~~~~~~~~~~~~~~~~~

* Fix artifact uploads to pypi.


1.3.0 (2017-12-28)
~~~~~~~~~~~~~~~~~~

* ``has-class`` XPath extension function;
* ``parsel.xpathfuncs.set_xpathfunc`` is a simplified way to register
  XPath extensions;
* ``Selector.remove_namespaces`` now removes namespace declarations;
* Python 3.3 support is dropped;
* ``make htmlview`` command for easier Parsel docs development.
* CI: PyPy installation is fixed; parsel now runs tests for PyPy3 as well.


1.2.0 (2017-05-17)
~~~~~~~~~~~~~~~~~~

* Add ``SelectorList.get`` and ``SelectorList.getall``
  methods as aliases for ``SelectorList.extract_first``
  and ``SelectorList.extract`` respectively
* Add default value parameter to ``SelectorList.re_first`` method
* Add ``Selector.re_first`` method
* Add ``replace_entities`` argument on ``.re()`` and ``.re_first()``
  to turn off replacing of character entity references
* Bug fix: detect ``None`` result from lxml parsing and fallback with an empty document
* Rearrange XML/HTML examples in the selectors usage docs
* Travis CI:

  * Test against Python 3.6
  * Test against PyPy using "Portable PyPy for Linux" distribution


1.1.0 (2016-11-22)
~~~~~~~~~~~~~~~~~~

* Change default HTML parser to `lxml.html.HTMLParser <https://lxml.de/api/lxml.html.HTMLParser-class.html>`_,
  which makes easier to use some HTML specific features
* Add css2xpath function to translate CSS to XPath
* Add support for ad-hoc namespaces declarations
* Add support for XPath variables
* Documentation improvements and updates


1.0.3 (2016-07-29)
~~~~~~~~~~~~~~~~~~

* Add BSD-3-Clause license file
* Re-enable PyPy tests
* Integrate py.test runs with setuptools (needed for Debian packaging)
* Changelog is now called ``NEWS``


1.0.2 (2016-04-26)
~~~~~~~~~~~~~~~~~~

* Fix bug in exception handling causing original traceback to be lost
* Added docstrings and other doc fixes


1.0.1 (2015-08-24)
~~~~~~~~~~~~~~~~~~

* Updated PyPI classifiers
* Added docstrings for csstranslator module and other doc fixes


1.0.0 (2015-08-22)
~~~~~~~~~~~~~~~~~~

* Documentation fixes


0.9.6 (2015-08-14)
~~~~~~~~~~~~~~~~~~

* Updated documentation
* Extended test coverage


0.9.5 (2015-08-11)
~~~~~~~~~~~~~~~~~~

* Support for extending SelectorList


0.9.4 (2015-08-10)
~~~~~~~~~~~~~~~~~~

* Try workaround for travis-ci/dpl#253


0.9.3 (2015-08-07)
~~~~~~~~~~~~~~~~~~

* Add base_url argument


0.9.2 (2015-08-07)
~~~~~~~~~~~~~~~~~~

* Rename module unified -> selector and promoted root attribute
* Add create_root_node function


0.9.1 (2015-08-04)
~~~~~~~~~~~~~~~~~~

* Setup Sphinx build and docs structure
* Build universal wheels
* Rename some leftovers from package extraction


0.9.0 (2015-07-30)
~~~~~~~~~~~~~~~~~~

* First release on PyPI.

Dependency	Optional
cssselect	false
jmespath	false
lxml	false
packaging	false
w3lib	false

Package profile

parsel

Releases

Release table

PyPI Downloads

Dependencies

Details

Project URLs

Dependencies

Package description