PyYAML-ft

A full-featured YAML processing framework for Python with support for free-threading

Why a fork?

PEP 703 introduced free-threaded Python as a
separate build of CPython 3.13. Thread-safety issues that might have otherwise gone
unnoticed are now much easier to trigger because of the absence of protection from
the GIL. Also, because the free-threaded build is ABI-incompatible, extension
modules need to be separate, free-threaded wheels and declare support for
it.

The PyYAML maintainers
decided to not port PyYAML to the free-threaded build
before the latter, along with Cython support for it, has been tested more extensively
in real-world applications. Our rationale with this fork is to implement support for
the free-threaded build, so that PyYAML can be tested with it by its users, even before
the port is merged upstream.

Differences compared with upstream

This fork uses Cython 3.1 which supports the free-threaded build, but whose support
is still experimental.

add_constructor, add_representer and add_*resolver now all use thread-local
registries, so you will have to explicitly register your custom constrcutors,
representers and resolvers in each thread. Here's a small test showcasing that:

import io
import threading
import yaml_ft as yaml


class Dice:
    def __init__(self, first, second):
        self.first = first
        self.second = second
    def __repr__(self):
        return f"Dice({self.first}, {self.second})"


def construct_dice(constructor, node):
    mapping = constructor.construct_mapping(node)
    return Dice(**mapping)


def load_dice():
    yamlcode = io.StringIO("""\
- !dice
  first: 1
  second: 6
- !dice
  first: 4
  second: 4
""")
    print(f"Thread {threading.current_thread().name}")
    try:
        objs = yaml.load(yamlcode, Loader=yaml.CLoader)
        print(f"\t{objs=}")
    except Exception as e:
        print(f"\tException occurred: {e!s}")

yaml.add_constructor("!dice", construct_dice, Loader=yaml.CLoader)
load_dice()

t = threading.Thread(target=load_dice)
t.start()
t.join()

Running the above script gives the following:

❯ python3.13t tmp/t.py
Thread MainThread
        objs=[Dice(1, 6), Dice(4, 4)]
Thread Thread-1 (load_dice)
        Exception occurred: could not determine a constructor for the tag '!dice'
  in "<file>", line 1, column 3

If you see new errors in multithreaded programs using PyYAML-ft that work with
PyYAML, you may need to add calls to yaml_ft.add_constructor, yaml_ft.add_representer
yaml_ft.add_implicit_resolver or yaml_ft.add_path_resolver in your thread worker
function or worker initialization function.

Python versions support

Because PyYAML-ft is only aiming to exist for as long as upstream PyYAML
does not support the free-threaded build, we recommend that users only
conditionally switch to PyYAML-ft.

At this time, PyYAML-ft only supports Python 3.13 and 3.13t (i.e. the
free-threaded build of 3.13). To switch to it, you can do the following
in your requirements.txt file:

...
PyYAML; python_version < '3.13'
PyYAML-ft; python_version >= '3.13'

If you're developing a library that depends on PyYAML and you're using
pyproject.toml to specify your dependencies, you can do the following:

dependencies = [
  ...,
  "PyYAML; python_version<'3.13'",
  "PyYAML-ft; python_version>='3.13'",
]

Different module name

PyYAML-ft uses a different module name (namely yaml_ft) than upstream PyYAML
on purpose, so that both can be installed in an environment at the same time.

If your library depends on both for different Python versions, you can do
the following for ease of use:

try:
    import yaml_ft as yaml
except ModuleNotFoundError:
    import yaml

Installation

To install, type pip install PyYAML-ft.

When LibYAML bindings are installed (enabled by default), you may use the fast
LibYAML-based parser and emitter as follows:

>>> yaml.load(stream, Loader=yaml.CLoader)
>>> yaml.dump(data, Dumper=yaml.CDumper)

If you don't trust the input YAML stream, you should use:

>>> yaml.safe_load(stream)

Building from source and testing

To build PyYAML-ft from source:

Clone the fork repository with
git clone https://github.com/Quansight-Labs/pyyaml-ft.
Create a new virtual environment with python3.13 -m venv .venv.
Run source .venv/bin/activate to activate your new virtual environment.
Run python3.13 -m pip install .. This will build PyYAML-ft
from source and install it in the newly created virtual environment. By default, the
installation process will try to install the LibYAML bindings as well, but will
fail silently if it cannot and instead use the pure-Python version.

To get the latest version of the LibYAML bindings, you can use the libyaml.sh
installation script:
```
./packaging/build/libyaml.sh  # This will install the bindings
```
After the bindings are installed, you can force installing the bindings with the
following invocation:
```
PYYAML_FORCE_CYTHON=1 \
PYYAML_FORCE_LIBYAML=1 \
CFLAGS="-Ilibyaml/include" \
LDFLAGS="-Llibyaml/src/.libs" \
python -m pip install -v .
```

To test PyYAML-ft:

Install pytest with python3.13 -m pip install pytest.
Run the tests by just invoking pytest in the root directory of the project.

Optionally, you can also run the free-threading tests with pytest-run-parallel:

python3.13 -m pip install pytest-run-parallel
python3.13 -m pytest --parallel-threads=auto --iterations=20 tests/free_threading

Further Information

For more information, check the
PyYAML homepage.
PyYAML tutorial and reference.
Discuss PyYAML with the maintainers on
Matrix at https://matrix.to/#/#pyyaml:yaml.io or
IRC #pyyaml irc.libera.chat
Submit bug reports and feature requests to the
PyYAML-ft bug tracker.

License

The PyYAML module was written by Kirill Simonov xi@resolvent.net.
It is currently maintained by the YAML and Python communities.

PyYAML-ft was forked by Quansight-Labs and is currently maintained by
a team of engineers working on ecosystem-wide support for free-threading.
It is released under the MIT license.

See the file LICENSE for more details.

Package profile

PyYAML-ft

Releases

Release table

PyPI Downloads

Dependencies

Details

Project URLs

Dependencies

Package description