{"author":"","author_email":"","bugtrack_url":null,"classifiers":["License :: OSI Approved :: MIT License","Operating System :: OS Independent","Programming Language :: Python :: 3"],"description":"# Sacremoses\n\n# License\n\n[MIT License](LICENSE).\n\n# Install\n\n```\npip install -U sacremoses\n```\n\nNOTE: Sacremoses only supports Python 3 now (`sacremoses>=0.0.41`). If you're using Python 2, the last possible version is `sacremoses==0.0.40`.\n\n# Usage (Python)\n\n## Tokenizer and Detokenizer\n\n```python\n>>> from sacremoses import MosesTokenizer, MosesDetokenizer\n>>> mt = MosesTokenizer(lang='en')\n>>> text = 'This, is a sentence with weird\\xbb symbols\\u2026 appearing everywhere\\xbf'\n>>> expected_tokenized = 'This , is a sentence with weird \\xbb symbols \\u2026 appearing everywhere \\xbf'\n>>> tokenized_text = mt.tokenize(text, return_str=True)\n>>> tokenized_text == expected_tokenized\nTrue\n\n\n>>> mt, md = MosesTokenizer(lang='en'), MosesDetokenizer(lang='en')\n>>> sent = \"This ain't funny. It's actually hillarious, yet double Ls. | [] < > [ ] & You're gonna shake it off? Don't?\"\n>>> expected_tokens = ['This', 'ain', ''t', 'funny', '.', 'It', ''s', 'actually', 'hillarious', ',', 'yet', 'double', 'Ls', '.', '|', '[', ']', '<', '>', '[', ']', '&', 'You', ''re', 'gonna', 'shake', 'it', 'off', '?', 'Don', ''t', '?']\n>>> expected_detokens = \"This ain't funny. It's actually hillarious, yet double Ls. | [] < > [] & You're gonna shake it off? Don't?\"\n>>> mt.tokenize(sent) == expected_tokens\nTrue\n>>> md.detokenize(tokens) == expected_detokens\nTrue\n```\n\n\n## Truecaser\n\n```python\n>>> from sacremoses import MosesTruecaser, MosesTokenizer\n\n# Train a new truecaser from a 'big.txt' file.\n>>> mtr = MosesTruecaser()\n>>> mtok = MosesTokenizer(lang='en')\n\n# Save the truecase model to 'big.truecasemodel' using `save_to`\n>> tokenized_docs = [mtok.tokenize(line) for line in open('big.txt')]\n>>> mtr.train(tokenized_docs, save_to='big.truecasemodel')\n\n# Save the truecase model to 'big.truecasemodel' after training\n# (just in case you forgot to use `save_to`)\n>>> mtr = MosesTruecaser()\n>>> mtr.train('big.txt')\n>>> mtr.save_model('big.truecasemodel')\n\n# Truecase a string after training a model.\n>>> mtr = MosesTruecaser()\n>>> mtr.train('big.txt')\n>>> mtr.truecase(\"THE ADVENTURES OF SHERLOCK HOLMES\")\n['the', 'adventures', 'of', 'Sherlock', 'Holmes']\n\n# Loads a model and truecase a string using trained model.\n>>> mtr = MosesTruecaser('big.truecasemodel')\n>>> mtr.truecase(\"THE ADVENTURES OF SHERLOCK HOLMES\")\n['the', 'adventures', 'of', 'Sherlock', 'Holmes']\n>>> mtr.truecase(\"THE ADVENTURES OF SHERLOCK HOLMES\", use_known=True)\n['the', 'ADVENTURES', 'OF', 'SHERLOCK', 'HOLMES']\n>>> mtr.truecase(\"THE ADVENTURES OF SHERLOCK HOLMES\", return_str=True)\n'the adventures of Sherlock Holmes'\n```\n\n## Normalizer\n\n```python\n>>> from sacremoses import MosesPunctNormalizer\n>>> mpn = MosesPunctNormalizer()\n>>> mpn.normalize('THIS EBOOK IS OTHERWISE PROVIDED TO YOU \"AS-IS.\"')\n'THIS EBOOK IS OTHERWISE PROVIDED TO YOU \"AS-IS.\"'\n```\n\n# Usage (CLI)\n\nSince version `0.0.42`, the pipeline feature for CLI is introduced, thus there\nare global options that should be set first before calling the commands:\n\n - language\n - processes\n - encoding\n - quiet\n\n```shell\n$ pip install -U sacremoses>=0.1\n\n$ sacremoses --help\nUsage: sacremoses [OPTIONS] COMMAND1 [ARGS]... [COMMAND2 [ARGS]...]...\n\nOptions:\n -l, --language TEXT Use language specific rules when tokenizing\n -j, --processes INTEGER No. of processes.\n -e, --encoding TEXT Specify encoding of file.\n -q, --quiet Disable progress bar.\n --version Show the version and exit.\n -h, --help Show this message and exit.\n\nCommands:\n detokenize\n detruecase\n normalize\n tokenize\n train-truecase\n truecase\n```\n\n## Pipeline\n\nExample to chain the following commands:\n\n - `normalize` with `-c` option to remove control characters.\n - `tokenize` with `-a` option for aggressive dash split rules.\n - `truecase` with `-a` option to indicate that model is for ASR \n - if `big.truemodel` exists, load the model with `-m` option,\n - otherwise train a model and save it with `-m` option to `big.truemodel` file.\n - save the output to console to the `big.txt.norm.tok.true` file.\n\n```shell\ncat big.txt | sacremoses -l en -j 4 \\\n normalize -c tokenize -a truecase -a -m big.truemodel \\\n > big.txt.norm.tok.true\n```\n\n## Tokenizer\n\n```shell\n$ sacremoses tokenize --help\nUsage: sacremoses tokenize [OPTIONS]\n\nOptions:\n -a, --aggressive-dash-splits Triggers dash split rules.\n -x, --xml-escape Escape special characters for XML.\n -p, --protected-patterns TEXT Specify file with patters to be protected in\n tokenisation.\n -c, --custom-nb-prefixes TEXT Specify a custom non-breaking prefixes file,\n add prefixes to the default ones from the\n specified language.\n -h, --help Show this message and exit.\n\n\n $ sacremoses -l en -j 4 tokenize < big.txt > big.txt.tok\n100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 24363.39it/s\n\n $ wget https://raw.githubusercontent.com/moses-smt/mosesdecoder/master/scripts/tokenizer/basic-protected-patterns\n $ sacremoses -l en -j 4 tokenize -p basic-protected-patterns < big.txt > big.txt.tok\n100%|██████████████████████████████████| 128457/128457 [00:05<00:00, 22183.94it/s\n```\n\n## Detokenizer\n\n```shell\n$ sacremoses detokenize --help\nUsage: sacremoses detokenize [OPTIONS]\n\nOptions:\n -x, --xml-unescape Unescape special characters for XML.\n -h, --help Show this message and exit.\n\n $ sacremoses -l en -j 4 detokenize < big.txt.tok > big.txt.tok.detok\n100%|██████████████████████████████████| 128457/128457 [00:16<00:00, 7931.26it/s]\n```\n\n## Truecase\n\n```shell\n$ sacremoses truecase --help\nUsage: sacremoses truecase [OPTIONS]\n\nOptions:\n -m, --modelfile TEXT Filename to save/load the modelfile.\n [required]\n -a, --is-asr A flag to indicate that model is for ASR.\n -p, --possibly-use-first-token Use the first token as part of truecase\n training.\n -h, --help Show this message and exit.\n\n$ sacremoses -j 4 truecase -m big.model < big.txt.tok > big.txt.tok.true\n100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 14257.27it/s]\n```\n\n## Detruecase\n\n```shell\n$ sacremoses detruecase --help\nUsage: sacremoses detruecase [OPTIONS]\n\nOptions:\n -j, --processes INTEGER No. of processes.\n -a, --is-headline Whether the file are headlines.\n -e, --encoding TEXT Specify encoding of file.\n -h, --help Show this message and exit.\n\n$ sacremoses -j 4 detruecase < big.txt.tok.true > big.txt.tok.true.detrue\n100%|█████████████████████████████████| 128457/128457 [00:04<00:00, 26945.16it/s]\n```\n\n## Normalize\n\n```shell\n$ sacremoses normalize --help\nUsage: sacremoses normalize [OPTIONS]\n\nOptions:\n -q, --normalize-quote-commas Normalize quotations and commas.\n -d, --normalize-numbers Normalize number.\n -p, --replace-unicode-puncts Replace unicode punctuations BEFORE\n normalization.\n -c, --remove-control-chars Remove control characters AFTER normalization.\n -h, --help Show this message and exit.\n\n$ sacremoses -j 4 normalize < big.txt > big.txt.norm\n100%|██████████████████████████████████| 128457/128457 [00:09<00:00, 13096.23it/s]\n```\n","description_content_type":"text/markdown","docs_url":null,"download_url":"","downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/hplt-project/sacremoses","keywords":"","license":"","license_expression":null,"license_files":null,"maintainer":"","maintainer_email":"","name":"sacremoses","package_url":"https://pypi.org/project/sacremoses/","platform":null,"project_url":"https://pypi.org/project/sacremoses/","project_urls":{"Homepage":"https://github.com/hplt-project/sacremoses"},"provides_extra":null,"release_url":"https://pypi.org/project/sacremoses/0.1.1/","requires_dist":["regex","click","joblib","tqdm"],"requires_python":">=3.8","summary":"SacreMoses","version":"0.1.1","yanked":false,"yanked_reason":null}

Dependencies of sacremoses (4).
Dependency	Optional
click	false
joblib	false
regex	false
tqdm	false

Package search

sacremoses

project_urls

Dependencies

All package metadata

Releases