PySpect

Home

Lists

Top stars

Rising stars

← Back to tokenizers

Dependencies for tokenizers

Frequently asked questions

© 2025-2026 PySpect

Dependencies of tokenizers (11).
Dependency	Optional	Group
huggingface-hub	false
datasets	true	testing
numpy	true	testing
pytest	true	testing

Dependent packages

Packages depending on tokenizers (43).
Package	Optional	Group
ai2-olmo	false
ai2-olmo-eval	false
aleph-alpha-client	false
chromadb	false
cohere	false
crewai	false
curated-transformers	false
fastembed	false
fastembed-gpu	false
faster-whisper	false
fastrtc-moonshine-onnx	false
flashrank	false
keras-hub	false
langchain-huggingface	false
langchain-mistralai	false
litellm	false
model2vec	false
optimum-graphcore	false
pytorch-tokenizers	false
skeletoken	false
transformers	false
unclecode-litellm	false
verdict	false
vllm	false
voyageai	false
whisper-s2t-reborn	false
agno	true	tokenizers
apache-beam	true	ml-test
arize	true	embeddings
chonkie	true	tokenizers
crawl4ai	true	transformer
crfm-helm	true	aleph-alpha
guidance	true	unittest
hf-doc-builder	true	all
modelscope	true	nlp
neural-compressor	true	jax
paddlex	true	ie
pyopenjtalk-plus	true	tsqyomi
semantic-router	true	all
shinychat	true	providers
skorch	true	extended
snowflake-ml-python	true	all
sudachipy	true	tests