{"author":"Onur Celebi","author_email":"celebio@fb.com","bugtrack_url":null,"classifiers":["Development Status :: 3 - Alpha","Intended Audience :: Developers","Intended Audience :: Science/Research","License :: OSI Approved :: MIT License","Operating System :: MacOS","Operating System :: Microsoft :: Windows","Operating System :: POSIX","Operating System :: Unix","Programming Language :: Python :: 2.7","Programming Language :: Python :: 3.4","Programming Language :: Python :: 3.5","Programming Language :: Python :: 3.6","Topic :: Scientific/Engineering","Topic :: Software Development"],"description":"fastText |CircleCI|\n===================\n\n`fastText `__ is a library for efficient learning\nof word representations and sentence classification.\n\nIn this document we present how to use fastText in python.\n\nTable of contents\n-----------------\n\n- `Requirements <#requirements>`__\n- `Installation <#installation>`__\n- `Usage overview <#usage-overview>`__\n- `Word representation model <#word-representation-model>`__\n- `Text classification model <#text-classification-model>`__\n- `IMPORTANT: Preprocessing data / encoding\n conventions <#important-preprocessing-data-encoding-conventions>`__\n- `More examples <#more-examples>`__\n- `API <#api>`__\n- `train_unsupervised parameters <#train_unsupervised-parameters>`__\n- `train_supervised parameters <#train_supervised-parameters>`__\n- `model object <#model-object>`__\n\nRequirements\n============\n\n`fastText `__ builds on modern Mac OS and Linux\ndistributions. Since it uses C++11 features, it requires a compiler with\ngood C++11 support. You will need `Python `__\n(version 2.7 or ≥ 3.4), `NumPy `__ &\n`SciPy `__ and\n`pybind11 `__.\n\nInstallation\n============\n\nTo install the latest release, you can do :\n\n.. code:: bash\n\n $ pip install fasttext\n\nor, to get the latest development version of fasttext, you can install\nfrom our github repository :\n\n.. code:: bash\n\n $ git clone https://github.com/facebookresearch/fastText.git\n $ cd fastText\n $ sudo pip install .\n $ # or :\n $ sudo python setup.py install\n\nUsage overview\n==============\n\nWord representation model\n-------------------------\n\nIn order to learn word vectors, as `described\nhere `__,\nwe can use ``fasttext.train_unsupervised`` function like this:\n\n.. code:: py\n\n import fasttext\n\n # Skipgram model :\n model = fasttext.train_unsupervised('data.txt', model='skipgram')\n\n # or, cbow model :\n model = fasttext.train_unsupervised('data.txt', model='cbow')\n\nwhere ``data.txt`` is a training file containing utf-8 encoded text.\n\nThe returned ``model`` object represents your learned model, and you can\nuse it to retrieve information.\n\n.. code:: py\n\n print(model.words) # list of words in dictionary\n print(model['king']) # get the vector of the word 'king'\n\nSaving and loading a model object\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nYou can save your trained model object by calling the function\n``save_model``.\n\n.. code:: py\n\n model.save_model(\"model_filename.bin\")\n\nand retrieve it later thanks to the function ``load_model`` :\n\n.. code:: py\n\n model = fasttext.load_model(\"model_filename.bin\")\n\nFor more information about word representation usage of fasttext, you\ncan refer to our `word representations\ntutorial `__.\n\nText classification model\n-------------------------\n\nIn order to train a text classifier using the method `described\nhere `__,\nwe can use ``fasttext.train_supervised`` function like this:\n\n.. code:: py\n\n import fasttext\n\n model = fasttext.train_supervised('data.train.txt')\n\nwhere ``data.train.txt`` is a text file containing a training sentence\nper line along with the labels. By default, we assume that labels are\nwords that are prefixed by the string ``__label__``\n\nOnce the model is trained, we can retrieve the list of words and labels:\n\n.. code:: py\n\n print(model.words)\n print(model.labels)\n\nTo evaluate our model by computing the precision at 1 (P@1) and the\nrecall on a test set, we use the ``test`` function:\n\n.. code:: py\n\n def print_results(N, p, r):\n print(\"N\\t\" + str(N))\n print(\"P@{}\\t{:.3f}\".format(1, p))\n print(\"R@{}\\t{:.3f}\".format(1, r))\n\n print_results(*model.test('test.txt'))\n\nWe can also predict labels for a specific text :\n\n.. code:: py\n\n model.predict(\"Which baking dish is best to bake a banana bread ?\")\n\nBy default, ``predict`` returns only one label : the one with the\nhighest probability. You can also predict more than one label by\nspecifying the parameter ``k``:\n\n.. code:: py\n\n model.predict(\"Which baking dish is best to bake a banana bread ?\", k=3)\n\nIf you want to predict more than one sentence you can pass an array of\nstrings :\n\n.. code:: py\n\n model.predict([\"Which baking dish is best to bake a banana bread ?\", \"Why not put knives in the dishwasher?\"], k=3)\n\nOf course, you can also save and load a model to/from a file as `in the\nword representation usage <#saving-and-loading-a-model-object>`__.\n\nFor more information about text classification usage of fasttext, you\ncan refer to our `text classification\ntutorial `__.\n\nCompress model files with quantization\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nWhen you want to save a supervised model file, fastText can compress it\nin order to have a much smaller model file by sacrificing only a little\nbit performance.\n\n.. code:: py\n\n # with the previously trained `model` object, call :\n model.quantize(input='data.train.txt', retrain=True)\n\n # then display results and save the new model :\n print_results(*model.test(valid_data))\n model.save_model(\"model_filename.ftz\")\n\n``model_filename.ftz`` will have a much smaller size than\n``model_filename.bin``.\n\nFor further reading on quantization, you can refer to `this paragraph\nfrom our blog\npost `__.\n\nIMPORTANT: Preprocessing data / encoding conventions\n----------------------------------------------------\n\nIn general it is important to properly preprocess your data. In\nparticular our example scripts in the `root\nfolder `__ do this.\n\nfastText assumes UTF-8 encoded text. All text must be `unicode for\nPython2 `__\nand `str for\nPython3 `__.\nThe passed text will be `encoded as UTF-8 by\npybind11 `__\nbefore passed to the fastText C++ library. This means it is important to\nuse UTF-8 encoded text when building a model. On Unix-like systems you\ncan convert text using `iconv `__.\n\nfastText will tokenize (split text into pieces) based on the following\nASCII characters (bytes). In particular, it is not aware of UTF-8\nwhitespace. We advice the user to convert UTF-8 whitespace / word\nboundaries into one of the following symbols as appropiate.\n\n- space\n- tab\n- vertical tab\n- carriage return\n- formfeed\n- the null character\n\nThe newline character is used to delimit lines of text. In particular,\nthe EOS token is appended to a line of text if a newline character is\nencountered. The only exception is if the number of tokens exceeds the\nMAX\\_LINE\\_SIZE constant as defined in the `Dictionary\nheader `__.\nThis means if you have text that is not separate by newlines, such as\nthe `fil9 dataset `__, it will be\nbroken into chunks with MAX\\_LINE\\_SIZE of tokens and the EOS token is\nnot appended.\n\nThe length of a token is the number of UTF-8 characters by considering\nthe `leading two bits of a\nbyte `__ to identify\n`subsequent bytes of a multi-byte\nsequence `__.\nKnowing this is especially important when choosing the minimum and\nmaximum length of subwords. Further, the EOS token (as specified in the\n`Dictionary\nheader `__)\nis considered a character and will not be broken into subwords.\n\nMore examples\n-------------\n\nIn order to have a better knowledge of fastText models, please consider\nthe main\n`README `__\nand in particular `the tutorials on our\nwebsite `__.\n\nYou can find further python examples in `the doc\nfolder `__.\n\nAs with any package you can get help on any Python function using the\nhelp function.\n\nFor example\n\n::\n\n +>>> import fasttext\n +>>> help(fasttext.FastText)\n\n Help on module fasttext.FastText in fasttext:\n\n NAME\n fasttext.FastText\n\n DESCRIPTION\n # Copyright (c) 2017-present, Facebook, Inc.\n # All rights reserved.\n #\n # This source code is licensed under the MIT license found in the\n # LICENSE file in the root directory of this source tree.\n\n FUNCTIONS\n load_model(path)\n Load a model given a filepath and return a model object.\n\n tokenize(text)\n Given a string of text, tokenize it and return a list of tokens\n [...]\n\nAPI\n===\n\n``train_unsupervised`` parameters\n---------------------------------\n\n.. code:: python\n\n input # training file path (required)\n model # unsupervised fasttext model {cbow, skipgram} [skipgram]\n lr # learning rate [0.05]\n dim # size of word vectors [100]\n ws # size of the context window [5]\n epoch # number of epochs [5]\n minCount # minimal number of word occurences [5]\n minn # min length of char ngram [3]\n maxn # max length of char ngram [6]\n neg # number of negatives sampled [5]\n wordNgrams # max length of word ngram [1]\n loss # loss function {ns, hs, softmax, ova} [ns]\n bucket # number of buckets [2000000]\n thread # number of threads [number of cpus]\n lrUpdateRate # change the rate of updates for the learning rate [100]\n t # sampling threshold [0.0001]\n verbose # verbose [2]\n\n``train_supervised`` parameters\n-------------------------------\n\n.. code:: python\n\n input # training file path (required)\n lr # learning rate [0.1]\n dim # size of word vectors [100]\n ws # size of the context window [5]\n epoch # number of epochs [5]\n minCount # minimal number of word occurences [1]\n minCountLabel # minimal number of label occurences [1]\n minn # min length of char ngram [0]\n maxn # max length of char ngram [0]\n neg # number of negatives sampled [5]\n wordNgrams # max length of word ngram [1]\n loss # loss function {ns, hs, softmax, ova} [softmax]\n bucket # number of buckets [2000000]\n thread # number of threads [number of cpus]\n lrUpdateRate # change the rate of updates for the learning rate [100]\n t # sampling threshold [0.0001]\n label # label prefix ['__label__']\n verbose # verbose [2]\n pretrainedVectors # pretrained word vectors (.vec file) for supervised learning []\n\n``model`` object\n----------------\n\n``train_supervised``, ``train_unsupervised`` and ``load_model``\nfunctions return an instance of ``_FastText`` class, that we generaly\nname ``model`` object.\n\nThis object exposes those training arguments as properties : ``lr``,\n``dim``, ``ws``, ``epoch``, ``minCount``, ``minCountLabel``, ``minn``,\n``maxn``, ``neg``, ``wordNgrams``, ``loss``, ``bucket``, ``thread``,\n``lrUpdateRate``, ``t``, ``label``, ``verbose``, ``pretrainedVectors``.\nSo ``model.wordNgrams`` will give you the max length of word ngram used\nfor training this model.\n\nIn addition, the object exposes several functions :\n\n.. code:: python\n\n get_dimension # Get the dimension (size) of a lookup vector (hidden layer).\n # This is equivalent to `dim` property.\n get_input_vector # Given an index, get the corresponding vector of the Input Matrix.\n get_input_matrix # Get a copy of the full input matrix of a Model.\n get_labels # Get the entire list of labels of the dictionary\n # This is equivalent to `labels` property.\n get_line # Split a line of text into words and labels.\n get_output_matrix # Get a copy of the full output matrix of a Model.\n get_sentence_vector # Given a string, get a single vector represenation. This function\n # assumes to be given a single line of text. We split words on\n # whitespace (space, newline, tab, vertical tab) and the control\n # characters carriage return, formfeed and the null character.\n get_subword_id # Given a subword, return the index (within input matrix) it hashes to.\n get_subwords # Given a word, get the subwords and their indicies.\n get_word_id # Given a word, get the word id within the dictionary.\n get_word_vector # Get the vector representation of word.\n get_words # Get the entire list of words of the dictionary\n # This is equivalent to `words` property.\n is_quantized # whether the model has been quantized\n predict # Given a string, get a list of labels and a list of corresponding probabilities.\n quantize # Quantize the model reducing the size of the model and it's memory footprint.\n save_model # Save the model to the given path\n test # Evaluate supervised model using file given by path\n test_label # Return the precision and recall score for each label.\n\nThe properties ``words``, ``labels`` return the words and labels from\nthe dictionary :\n\n.. code:: py\n\n model.words # equivalent to model.get_words()\n model.labels # equivalent to model.get_labels()\n\nThe object overrides ``__getitem__`` and ``__contains__`` functions in\norder to return the representation of a word and to check if a word is\nin the vocabulary.\n\n.. code:: py\n\n model['king'] # equivalent to model.get_word_vector('king')\n 'king' in model # equivalent to `'king' in model.get_words()`\n\nJoin the fastText community\n---------------------------\n\n- `Facebook page `__\n- `Stack\n overflow `__\n- `Google\n group `__\n- `GitHub `__\n\n.. |CircleCI| image:: https://circleci.com/gh/facebookresearch/fastText/tree/master.svg?style=svg\n :target: https://circleci.com/gh/facebookresearch/fastText/tree/master\n","description_content_type":null,"docs_url":null,"download_url":null,"downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/facebookresearch/fastText","keywords":null,"license":"MIT","license_expression":null,"license_files":null,"maintainer":null,"maintainer_email":null,"name":"fasttext","package_url":"https://pypi.org/project/fasttext/","platform":null,"project_url":"https://pypi.org/project/fasttext/","project_urls":{"Homepage":"https://github.com/facebookresearch/fastText"},"provides_extra":null,"release_url":"https://pypi.org/project/fasttext/0.9.3/","requires_dist":["pybind11>=2.2","setuptools>=0.7.0","numpy"],"requires_python":null,"summary":"fasttext Python bindings","version":"0.9.3","yanked":false,"yanked_reason":null}

Dependencies of fasttext (3).
Dependency	Optional
numpy	false
pybind11	false
setuptools	false

Package search

fasttext

project_urls

Dependencies

All package metadata

Releases