Package profile
tokenizers
- Summary:
- Author: Nicolas Patry <patry.nicolas@protonmail.com>, Anthony Moi <anthony@huggingface.co>
- Homepage: https://github.com/huggingface/tokenizers
- Source: https://github.com/huggingface/tokenizers (Repo profile)
- Number of releases: 97
- First release: 0.0.1 on 2019-11-01
- Latest release: 0.22.0 on 2025-08-29
Releases
PyPI Downloads
Loading PyPI statistics...
Dependencies
Tokenizers has 11 dependencies, 10 of which optional.| Dependency | Optional |
|---|---|
| huggingface-hub | false |
| black | true |
| datasets | true |
| numpy | true |
| pytest | true |
| pytest-asyncio | true |
| requests | true |
| ruff | true |
| setuptools-rust | true |
| sphinx | true |
| sphinx-rtd-theme | true |