Details

Similar packages

• segtoksentence segmentation and word tokenization tools
• untokenizeTransforms tokens into original source code (while preserving whitespace).
• textparserText parser.
• pytokensA Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
• tokenize-rtA wrapper around the stdlib `tokenize` which roundtrips.
• langchain-text-splittersLangChain text splitting utilities
• chardetUniversal character encoding detector

Releases

1 known vulnerabilityLatest: GHSA-38vq-g6vr-w8wf — Sentencepiece has a a heap overflow issue View all →

Dependencies

Dependent packages