Package profile
tokenizers
- Summary:
- Author: Anthony MOI <m.anthony.moi@gmail.com>
- Homepage: https://github.com/huggingface/tokenizers
- Source: https://github.com/huggingface/tokenizers (Repo profile)
- Number of releases: 98
- First release: 0.0.1 on 2019-11-01
- Latest release: 0.21.4.dev0 on 2025-07-04
Releases
PyPI Downloads
Dependencies
Tokenizers has 10 dependencies, 9 of which optional.| Dependency | Optional |
|---|---|
| huggingface-hub | false |
| black | true |
| datasets | true |
| numpy | true |
| pytest | true |
| requests | true |
| ruff | true |
| setuptools-rust | true |
| sphinx | true |
| sphinx-rtd-theme | true |