Repository info

Tokenizer

• Summary: a summary of Tokenizer
• Description: Fast and customizable text tokenization library with BPE and SentencePiece support
• Stars: 304
• Number of forks: 74