{"author":"Pattarawat Chormai et al.","author_email":"foomail@foo.com","bugtrack_url":null,"classifiers":["License :: OSI Approved :: MIT License","Natural Language :: Thai","Operating System :: OS Independent","Programming Language :: Python :: 3"],"description":"# AttaCut: Fast and Reasonably Accurate Word Tokenizer for Thai\n\n[![Build Status](https://travis-ci.org/PyThaiNLP/attacut.svg?branch=master)](https://travis-ci.org/PyThaiNLP/attacut)\n[![Build status](https://ci.appveyor.com/api/projects/status/ittfnb2pyg95kpxk/branch/master?svg=true)](https://ci.appveyor.com/project/wannaphongcom/attacut/branch/master)\n[![](https://img.shields.io/badge/-presentation-informational)](https://drive.google.com/file/d/16AUNZv1HXVmERgryfBf4JpCo1QrQyHHE/view?usp=sharing)\n[![](http://img.shields.io/badge/arXiv-1911.07056-yellow.svg?style=flat)](https://arxiv.org/abs/1911.07056)\n\n## How does AttaCut look like?\n\n

\n
\n TL;DR: \n3-Layer Dilated CNN on syllable and character features. It’s 6x faster than DeepCut (SOTA) while its WL-f1 on BEST is 91%, only 2% lower.\n

\n\n## Installation\n\n```\n$ pip install attacut\n```\n\n**Remarks:** Windows users need to install **PyTorch** before the command above.\nPlease consult [PyTorch.org](https://pytorch.org) for more details.\n\n## Usage\n\n### Command-Line Interface\n\n```\n$ attacut-cli -h\nAttaCut: Fast and Reasonably Accurate Word Tokenizer for Thai\n\nUsage:\n attacut-cli [--dest=] [--model=]\n attacut-cli [-v | --version]\n attacut-cli [-h | --help]\n\nArguments:\n Path to input text file to be tokenized\n\nOptions:\n -h --help Show this screen.\n --model= Model to be used [default: attacut-sc].\n --dest= If not specified, it'll be -tokenized-by-.txt\n -v --version Show version\n```\n\n### High-Level API \n```\nfrom attacut import tokenize, Tokenizer\n\n# tokenize `txt` using our best model `attacut-sc`\nwords = tokenize(txt)\n\n# alternatively, an AttaCut tokenizer might be instantiated directly, allowing\n# one to specify whether to use `attacut-sc` or `attacut-c`.\natta = Tokenizer(model=\"attacut-sc\")\nwords = atta.tokenize(txt)\n```\n\n## Benchmark Results\n\nBelows are brief summaries. More details can be found on [our benchmarking page](https://pythainlp.github.io/attacut/benchmark.html).\n\n\n### Tokenization Quality\n![](https://pythainlp.github.io/attacut/_images/quality-benchmark-in-of-domain.png)\n\n### Speed\n![](https://pythainlp.github.io/attacut/_images/speed-benchmark-ec2.png)\n\n\n## Retraining on Custom Dataset\nPlease refer to [our retraining page](https://pythainlp.github.io/attacut/)\n\n## Related Resources\n- [Tokenization Visualization][tovis]\n- [Thai Tokenizer Dockers][docker]\n\n## Acknowledgements\nThis repository was initially done by [Pattarawat Chormai][pat], while interning at [Dr. Attapol Thamrongrattanarit's NLP Lab][ate], Chulalongkorn University, Bangkok, Thailand.\nMany people have involed in this project. Complete list of names can be found on [Acknowledgement](https://pythainlp.github.io/attacut/acknowledgement.html).\n\n\n[pat]: http://pat.chormai.org\n[ate]: https://attapol.github.io/lab.html\n[noom]: https://github.com/Ekkalak-T\n[can]: https://github.com/c4n\n[ake]: https://github.com/ekapolc\n[tovis]: https://pythainlp.github.io/tokenization-benchmark-visualization/\n[docker]: https://github.com/PyThaiNLP/docker-thai-tokenizers\n\n\n","description_content_type":"text/markdown","docs_url":null,"download_url":"","downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/PyThaiNLP/attacut","keywords":"","license":"","license_expression":null,"license_files":null,"maintainer":"","maintainer_email":"","name":"attacut","package_url":"https://pypi.org/project/attacut/","platform":"","project_url":"https://pypi.org/project/attacut/","project_urls":{"Homepage":"https://github.com/PyThaiNLP/attacut"},"provides_extra":null,"release_url":"https://pypi.org/project/attacut/1.0.6/","requires_dist":["docopt (>=0.6.2)","fire (>=0.1.3)","nptyping (>=0.2.0)","numpy (>=1.17.0)","pyyaml (>=5.1.2)","six (>=1.12.0)","ssg (>=0.0.4)","torch (>=1.2.0)"],"requires_python":"","summary":"Fast and Reasonably Accurate Word Tokenizer for Thai","version":"1.0.6","yanked":false,"yanked_reason":null}

Dependency	Optional
docopt	false
fire	false
nptyping	false
numpy	false
pyyaml	false
six	false
ssg	false
torch	false

Package search

attacut

project_urls

Dependencies

All package metadata

Releases