{"author":"Tomaz Solc","author_email":"tomaz.solc@tablix.org","bugtrack_url":null,"classifiers":["License :: OSI Approved :: GNU General Public License v2 or later (GPLv2+)","Programming Language :: Python","Programming Language :: Python :: 3","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.7","Programming Language :: Python :: 3.8","Programming Language :: Python :: 3.9","Programming Language :: Python :: Implementation :: CPython","Programming Language :: Python :: Implementation :: PyPy","Topic :: Text Processing","Topic :: Text Processing :: Filters"],"description":"Unidecode, lossy ASCII transliterations of Unicode text\n=======================================================\n\nIt often happens that you have text data in Unicode, but you need to\nrepresent it in ASCII. For example when integrating with legacy code that\ndoesn't support Unicode, or for ease of entry of non-Roman names on a US\nkeyboard, or when constructing ASCII machine identifiers from human-readable\nUnicode strings that should still be somewhat intelligible. A popular example\nof this is when making an URL slug from an article title.\n\n**Unidecode is not a replacement for fully supporting Unicode for strings in\nyour program. There are a number of caveats that come with its use,\nespecially when its output is directly visible to users. Please read the rest\nof this README before using Unidecode in your project.**\n\nIn most of examples listed above you could represent Unicode characters as\n``???`` or ``\\\\15BA\\\\15A0\\\\1610``, to mention two extreme cases. But that's\nnearly useless to someone who actually wants to read what the text says.\n\nWhat Unidecode provides is a middle road: the function ``unidecode()`` takes\nUnicode data and tries to represent it in ASCII characters (i.e., the\nuniversally displayable characters between 0x00 and 0x7F), where the\ncompromises taken when mapping between two character sets are chosen to be\nnear what a human with a US keyboard would choose.\n\nThe quality of resulting ASCII representation varies. For languages of\nwestern origin it should be between perfect and good. On the other hand\ntransliteration (i.e., conveying, in Roman letters, the pronunciation\nexpressed by the text in some other writing system) of languages like\nChinese, Japanese or Korean is a very complex issue and this library does\nnot even attempt to address it. It draws the line at context-free\ncharacter-by-character mapping. So a good rule of thumb is that the further\nthe script you are transliterating is from Latin alphabet, the worse the\ntransliteration will be.\n\nGenerally Unidecode produces better results than simply stripping accents from\ncharacters (which can be done in Python with built-in functions). It is based\non hand-tuned character mappings that for example also contain ASCII\napproximations for symbols and non-Latin alphabets.\n\n**Note that some people might find certain transliterations offending.** Most\ncommon examples include characters that are used in multiple languages. A user\nexpects a character to be transliterated in their language but Unidecode uses a\ntransliteration for a different language. It's best to not use Unidecode for\nstrings that are directly visible to users of your application. See also the\n*Frequently Asked Questions* section for more info on common problems.\n\nThis is a Python port of ``Text::Unidecode`` Perl module by Sean M. Burke\n.\n\n\nModule content\n--------------\n\nThis library contains a function that takes a string object, possibly\ncontaining non-ASCII characters, and returns a string that can be safely\nencoded to ASCII::\n\n >>> from unidecode import unidecode\n >>> unidecode('kožušček')\n 'kozuscek'\n >>> unidecode('30 \\U0001d5c4\\U0001d5c6/\\U0001d5c1')\n '30 km/h'\n >>> unidecode('\\u5317\\u4EB0')\n 'Bei Jing '\n\nYou can also specify an *errors* argument to ``unidecode()`` that determines\nwhat Unidecode does with characters that are not present in its transliteration\ntables. The default is ``'ignore'`` meaning that Unidecode will ignore those\ncharacters (replace them with an empty string). ``'strict'`` will raise a\n``UnidecodeError``. The exception object will contain an *index* attribute that\ncan be used to find the offending character. ``'replace'`` will replace them\nwith ``'?'`` (or another string, specified in the *replace_str* argument).\n``'preserve'`` will keep the original, non-ASCII character in the string. Note\nthat if ``'preserve'`` is used the string returned by ``unidecode()`` will not\nbe ASCII-encodable!::\n\n >>> unidecode('\\ue000') # unidecode does not have replacements for Private Use Area characters\n ''\n >>> unidecode('\\ue000', errors='strict')\n Traceback (most recent call last):\n ...\n unidecode.UnidecodeError: no replacement found for character '\\ue000' in position 0\n\nA utility is also included that allows you to transliterate text from the\ncommand line in several ways. Reading from standard input::\n\n $ echo hello | unidecode\n hello\n\nfrom a command line argument::\n\n $ unidecode -c hello\n hello\n\nor from a file::\n\n $ unidecode hello.txt\n hello\n\nThe default encoding used by the utility depends on your system locale. You can\nspecify another encoding with the ``-e`` argument. See ``unidecode --help`` for\na full list of available options.\n\nRequirements\n------------\n\nNothing except Python itself. Unidecode supports Python 3.7 or later.\n\nYou need a Python build with \"wide\" Unicode characters (also called \"UCS-4\nbuild\") in order for Unidecode to work correctly with characters outside of\nBasic Multilingual Plane (BMP). Common characters outside BMP are bold, italic,\nscript, etc. variants of the Latin alphabet intended for mathematical notation.\nSurrogate pair encoding of \"narrow\" builds is not supported in Unidecode.\n\nIf your Python build supports \"wide\" Unicode the following expression will\nreturn True::\n\n >>> import sys\n >>> sys.maxunicode > 0xffff\n True\n\nSee `PEP 261 `_ for details\nregarding support for \"wide\" Unicode characters in Python.\n\n\nInstallation\n------------\n\nTo install the latest version of Unidecode from the Python package index, use\nthese commands::\n\n $ pip install unidecode\n\nTo install Unidecode from the source distribution and run unit tests, use::\n\n $ python setup.py install\n $ python setup.py test\n\nFrequently asked questions\n--------------------------\n\nGerman umlauts are transliterated incorrectly\n Latin letters \"a\", \"o\" and \"u\" with diaeresis are transliterated by\n Unidecode as \"a\", \"o\", \"u\", *not* according to German rules \"ae\", \"oe\",\n \"ue\". This is intentional and will not be changed. Rationale is that these\n letters are used in languages other than German (for example, Finnish and\n Turkish). German text transliterated without the extra \"e\" is much more\n readable than other languages transliterated using German rules. A\n workaround is to do your own replacements of these characters before\n passing the string to ``unidecode()``.\n\nJapanese Kanji is transliterated as Chinese\n Same as with Latin letters with accents discussed in the answer above, the\n Unicode standard encodes letters, not letters in a certain language or\n their meaning. With Japanese and Chinese this is even more evident because\n the same letter can have very different transliterations depending on the\n language it is used in. Since Unidecode does not do language-specific\n transliteration (see next question), it must decide on one. For certain\n characters that are used in both Japanese and Chinese the decision was to\n use Chinese transliterations. If you intend to transliterate Japanese,\n Chinese or Korean text please consider using other libraries which do\n language-specific transliteration, such as `Unihandecode\n `_.\n\nUnidecode should support localization (e.g. a language or country parameter, inspecting system locale, etc.)\n Language-specific transliteration is a complicated problem and beyond the\n scope of this library. Changes related to this will not be accepted. Please\n consider using other libraries which do provide this capability, such as\n `Unihandecode `_.\n\nUnidecode should automatically detect the language of the text being transliterated\n Language detection is a completely separate problem and beyond the scope of\n this library.\n\nUnidecode should use a permissive license such as MIT or the BSD license.\n The maintainer of Unidecode believes that providing access to source code\n on redistribution is a fair and reasonable request when basing products on\n voluntary work of many contributors. If the license is not suitable for\n you, please consider using other libraries, such as `text-unidecode\n `_.\n\nUnidecode produces completely wrong results (e.g. \"u\" with diaeresis transliterating as \"A 1/4 \")\n The strings you are passing to Unidecode have been wrongly decoded\n somewhere in your program. For example, you might be decoding utf-8 encoded\n strings as latin1. With a misconfigured terminal, locale and/or a text\n editor this might not be immediately apparent. Inspect your strings with\n ``repr()`` and consult the\n `Unicode HOWTO `_.\n\nWhy does Unidecode not replace \\\\u and \\\\U backslash escapes in my strings?\n Unidecode knows nothing about escape sequences. Interpreting these sequences\n and replacing them with actual Unicode characters in string literals is the\n task of the Python interpreter. If you are asking this question you are\n very likely misunderstanding the purpose of this library. Consult the\n `Unicode HOWTO `_ and possibly\n the ``unicode_escape`` encoding in the standard library.\n\nI've upgraded Unidecode and now some URLs on my website return 404 Not Found.\n This is an issue with the software that is running your website, not\n Unidecode. Occasionally, new versions of Unidecode library are released\n which contain improvements to the transliteration tables. This means that\n you cannot rely that ``unidecode()`` output will not change across\n different versions of Unidecode library. If you use ``unidecode()`` to\n generate URLs for your website, either generate the URL slug once and store\n it in the database or lock your dependency of Unidecode to one specific\n version.\n\nSome of the issues in this section are discussed in more detail in `this blog\npost `_.\n\n\nPerformance notes\n-----------------\n\nBy default, ``unidecode()`` optimizes for the use case where most of the strings\npassed to it are already ASCII-only and no transliteration is necessary (this\ndefault might change in future versions).\n\nFor performance critical applications, two additional functions are exposed:\n\n``unidecode_expect_ascii()`` is optimized for ASCII-only inputs (approximately\n5 times faster than ``unidecode_expect_nonascii()`` on 10 character strings,\nmore on longer strings), but slightly slower for non-ASCII inputs.\n\n``unidecode_expect_nonascii()`` takes approximately the same amount of time on\nASCII and non-ASCII inputs, but is slightly faster for non-ASCII inputs than\n``unidecode_expect_ascii()``.\n\nApart from differences in run time, both functions produce identical results.\nFor most users of Unidecode, the difference in performance should be\nnegligible.\n\n\nSource\n------\n\nYou can get the latest development version of Unidecode with::\n\n $ git clone https://www.tablix.org/~avian/git/unidecode.git\n\nThere is also an official mirror of this repository on GitHub at\nhttps://github.com/avian2/unidecode\n\n\nContact\n-------\n\nPlease make sure to read the `Frequently asked questions`_ section above before\ncontacting the maintainer.\n\nBug reports, patches and suggestions for Unidecode can be sent to\ntomaz.solc@tablix.org.\n\nAlternatively, you can also open a ticket or pull request at\nhttps://github.com/avian2/unidecode\n\n\nCopyright\n---------\n\nOriginal character transliteration tables:\n\nCopyright 2001, Sean M. Burke , all rights reserved.\n\nPython code and later additions:\n\nCopyright 2025, Tomaž Šolc \n\nThis program is free software; you can redistribute it and/or modify it\nunder the terms of the GNU General Public License as published by the Free\nSoftware Foundation; either version 2 of the License, or (at your option)\nany later version.\n\nThis program is distributed in the hope that it will be useful, but WITHOUT\nANY WARRANTY; without even the implied warranty of MERCHANTABILITY or\nFITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for\nmore details.\n\nYou should have received a copy of the GNU General Public License along\nwith this program; if not, write to the Free Software Foundation, Inc., 51\nFranklin Street, Fifth Floor, Boston, MA 02110-1301 USA. The programs and\ndocumentation in this dist are distributed in the hope that they will be\nuseful, but without any warranty; without even the implied warranty of\nmerchantability or fitness for a particular purpose.\n\n..\n vim: set filetype=rst:\n\n\n","description_content_type":null,"docs_url":null,"download_url":null,"downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":null,"keywords":null,"license":"GPL","license_expression":null,"license_files":null,"maintainer":null,"maintainer_email":null,"name":"Unidecode","package_url":"https://pypi.org/project/Unidecode/","platform":null,"project_url":"https://pypi.org/project/Unidecode/","project_urls":null,"provides_extra":null,"release_url":"https://pypi.org/project/Unidecode/1.4.0/","requires_dist":null,"requires_python":">=3.7","summary":"ASCII transliterations of Unicode text","version":"1.4.0","yanked":false,"yanked_reason":null}

Package search

Unidecode

project_urls

Dependencies

All package metadata

Releases