{"author":"Matthew Honnibal","author_email":"matt@explosion.ai","bugtrack_url":null,"classifiers":["Environment :: Console","Intended Audience :: Developers","Intended Audience :: Science/Research","License :: OSI Approved :: MIT License","Operating System :: MacOS :: MacOS X","Operating System :: Microsoft :: Windows","Operating System :: POSIX :: Linux","Programming Language :: Cython","Programming Language :: Python :: 2.6","Programming Language :: Python :: 2.7","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.12","Programming Language :: Python :: 3.3","Programming Language :: Python :: 3.4","Programming Language :: Python :: 3.5","Programming Language :: Python :: 3.6","Programming Language :: Python :: 3.7","Programming Language :: Python :: 3.8","Programming Language :: Python :: 3.9","Topic :: Scientific/Engineering"],"description":"

\n\n# cymem: A Cython Memory Helper\n\ncymem provides two small memory-management helpers for Cython. They make it easy\nto tie memory to a Python object's life-cycle, so that the memory is freed when\nthe object is garbage collected.\n\n[![tests](https://github.com/explosion/cymem/actions/workflows/tests.yml/badge.svg)](https://github.com/explosion/cymem/actions/workflows/tests.yml)\n[![pypi Version](https://img.shields.io/pypi/v/cymem.svg?style=flat-square&logo=pypi&logoColor=white)](https://pypi.python.org/pypi/cymem)\n[![conda Version](https://img.shields.io/conda/vn/conda-forge/cymem.svg?style=flat-square&logo=conda-forge&logoColor=white)](https://anaconda.org/conda-forge/cymem)\n[![Python wheels](https://img.shields.io/badge/wheels-%E2%9C%93-4c1.svg?longCache=true&style=flat-square&logo=python&logoColor=white)](https://github.com/explosion/wheelwright/releases)\n\n## Overview\n\nThe most useful is `cymem.Pool`, which acts as a thin wrapper around the calloc\nfunction:\n\n```python\nfrom cymem.cymem cimport Pool\ncdef Pool mem = Pool()\ndata1 = mem.alloc(10, sizeof(int))\ndata2 = mem.alloc(12, sizeof(float))\n```\n\nThe `Pool` object saves the memory addresses internally, and frees them when the\nobject is garbage collected. Typically you'll attach the `Pool` to some cdef'd\nclass. This is particularly handy for deeply nested structs, which have\ncomplicated initialization functions. Just pass the `Pool` object into the\ninitializer, and you don't have to worry about freeing your struct at all — all\nof the calls to `Pool.alloc` will be automatically freed when the `Pool`\nexpires.\n\n## Installation\n\nInstallation is via [pip](https://pypi.python.org/pypi/pip), and requires\n[Cython](http://cython.org). Before installing, make sure that your `pip`,\n`setuptools` and `wheel` are up to date.\n\n```bash\npip install -U pip setuptools wheel\npip install cymem\n```\n\n## Example Use Case: An array of structs\n\nLet's say we want a sequence of sparse matrices. We need fast access, and a\nPython list isn't performing well enough. So, we want a C-array or C++ vector,\nwhich means we need the sparse matrix to be a C-level struct — it can't be a\nPython class. We can write this easily enough in Cython:\n\n```python\n\"\"\"Example without Cymem\n\nTo use an array of structs, we must carefully walk the data structure when\nwe deallocate it.\n\"\"\"\n\nfrom libc.stdlib cimport calloc, free\n\ncdef struct SparseRow:\n size_t length\n size_t* indices\n double* values\n\ncdef struct SparseMatrix:\n size_t length\n SparseRow* rows\n\ncdef class MatrixArray:\n cdef size_t length\n cdef SparseMatrix** matrices\n\n def __cinit__(self, list py_matrices):\n self.length = 0\n self.matrices = NULL\n\n def __init__(self, list py_matrices):\n self.length = len(py_matrices)\n self.matrices = calloc(len(py_matrices), sizeof(SparseMatrix*))\n\n for i, py_matrix in enumerate(py_matrices):\n self.matrices[i] = sparse_matrix_init(py_matrix)\n\n def __dealloc__(self):\n for i in range(self.length):\n sparse_matrix_free(self.matrices[i])\n free(self.matrices)\n\n\ncdef SparseMatrix* sparse_matrix_init(list py_matrix) except NULL:\n sm = calloc(1, sizeof(SparseMatrix))\n sm.length = len(py_matrix)\n sm.rows = calloc(sm.length, sizeof(SparseRow))\n cdef size_t i, j\n cdef dict py_row\n cdef size_t idx\n cdef double value\n for i, py_row in enumerate(py_matrix):\n sm.rows[i].length = len(py_row)\n sm.rows[i].indices = calloc(sm.rows[i].length, sizeof(size_t))\n sm.rows[i].values = calloc(sm.rows[i].length, sizeof(double))\n for j, (idx, value) in enumerate(py_row.items()):\n sm.rows[i].indices[j] = idx\n sm.rows[i].values[j] = value\n return sm\n\n\ncdef void* sparse_matrix_free(SparseMatrix* sm) except *:\n cdef size_t i\n for i in range(sm.length):\n free(sm.rows[i].indices)\n free(sm.rows[i].values)\n free(sm.rows)\n free(sm)\n```\n\nWe wrap the data structure in a Python ref-counted class at as low a level as we\ncan, given our performance constraints. This allows us to allocate and free the\nmemory in the `__cinit__` and `__dealloc__` Cython special methods.\n\nHowever, it's very easy to make mistakes when writing the `__dealloc__` and\n`sparse_matrix_free` functions, leading to memory leaks. cymem prevents you from\nwriting these deallocators at all. Instead, you write as follows:\n\n```python\n\"\"\"Example with Cymem.\n\nMemory allocation is hidden behind the Pool class, which remembers the\naddresses it gives out. When the Pool object is garbage collected, all of\nits addresses are freed.\n\nWe don't need to write MatrixArray.__dealloc__ or sparse_matrix_free,\neliminating a common class of bugs.\n\"\"\"\nfrom cymem.cymem cimport Pool\n\ncdef struct SparseRow:\n size_t length\n size_t* indices\n double* values\n\ncdef struct SparseMatrix:\n size_t length\n SparseRow* rows\n\n\ncdef class MatrixArray:\n cdef size_t length\n cdef SparseMatrix** matrices\n cdef Pool mem\n\n def __cinit__(self, list py_matrices):\n self.mem = None\n self.length = 0\n self.matrices = NULL\n\n def __init__(self, list py_matrices):\n self.mem = Pool()\n self.length = len(py_matrices)\n self.matrices = self.mem.alloc(self.length, sizeof(SparseMatrix*))\n for i, py_matrix in enumerate(py_matrices):\n self.matrices[i] = sparse_matrix_init(self.mem, py_matrix)\n\ncdef SparseMatrix* sparse_matrix_init_cymem(Pool mem, list py_matrix) except NULL:\n sm = mem.alloc(1, sizeof(SparseMatrix))\n sm.length = len(py_matrix)\n sm.rows = mem.alloc(sm.length, sizeof(SparseRow))\n cdef size_t i, j\n cdef dict py_row\n cdef size_t idx\n cdef double value\n for i, py_row in enumerate(py_matrix):\n sm.rows[i].length = len(py_row)\n sm.rows[i].indices = mem.alloc(sm.rows[i].length, sizeof(size_t))\n sm.rows[i].values = mem.alloc(sm.rows[i].length, sizeof(double))\n for j, (idx, value) in enumerate(py_row.items()):\n sm.rows[i].indices[j] = idx\n sm.rows[i].values[j] = value\n return sm\n```\n\nAll that the `Pool` class does is remember the addresses it gives out. When the\n`MatrixArray` object is garbage-collected, the `Pool` object will also be\ngarbage collected, which triggers a call to `Pool.__dealloc__`. The `Pool` then\nfrees all of its addresses. This saves you from walking back over your nested\ndata structures to free them, eliminating a common class of errors.\n\n## Custom Allocators\n\nSometimes external C libraries use private functions to allocate and free\nobjects, but we'd still like the laziness of the `Pool`.\n\n```python\nfrom cymem.cymem cimport Pool, WrapMalloc, WrapFree\ncdef Pool mem = Pool(WrapMalloc(priv_malloc), WrapFree(priv_free))\n```\n","description_content_type":"text/markdown","docs_url":null,"download_url":null,"downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":["Author","Author-Email","Classifier","Description","Description-Content-Type","Home-Page","License","Summary"],"home_page":"https://github.com/explosion/cymem","keywords":null,"license":"MIT","license_expression":null,"license_files":null,"maintainer":null,"maintainer_email":null,"name":"cymem","package_url":"https://pypi.org/project/cymem/","platform":null,"project_url":"https://pypi.org/project/cymem/","project_urls":{"Homepage":"https://github.com/explosion/cymem"},"provides_extra":null,"release_url":"https://pypi.org/project/cymem/2.0.11/","requires_dist":null,"requires_python":null,"summary":"Manage calls to calloc/free through Cython","version":"2.0.11","yanked":false,"yanked_reason":null}

Package search

cymem

project_urls

Dependencies

All package metadata

Releases