Package profile
tokenize-rt
- Summary: A wrapper around the stdlib `tokenize` which roundtrips.
- Author: Anthony Sottile
- License: MIT
- Homepage: https://github.com/asottile/tokenize-rt
- Source: https://github.com/asottile/tokenize-rt
- Number of releases: 19
- First release: 1.0.0 on 2017-06-02
- Latest release: 6.2.0 on 2025-05-23
- Latest release size: 5.9 KB (pure Python wheel)
Dependencies
Tokenize-rt has no dependency.Dependent packages
| Package | Optional | Group |
|---|---|---|
| cython-lint | false | |
| future-fstrings | false | |
| nbqa | false | |
| pyupgrade | false | |
| unasync | false |
Similar packages
- pytokensA Fast, spec compliant Python 3.14+ tokenizer that runs on older Pythons.
- untokenizeTransforms tokens into original source code (while preserving whitespace).
- jaraco-functoolsFunctools like those found in stdlib
- sentencepieceUnsupervised text tokenizer and detokenizer.
- segtoksentence segmentation and word tokenization tools
- myst-parserAn extended [CommonMark](https://spec.commonmark.org/) compliant parser,
- glfwA ctypes-based wrapper for GLFW3.