{"author":"Neuralmagic, Inc.","author_email":"support@neuralmagic.com","bugtrack_url":null,"classifiers":["Development Status :: 5 - Production/Stable","Intended Audience :: Developers","Intended Audience :: Education","Intended Audience :: Information Technology","Intended Audience :: Science/Research","License :: OSI Approved :: Apache Software License","Operating System :: POSIX :: Linux","Programming Language :: Python :: 3","Programming Language :: Python :: 3 :: Only","Programming Language :: Python :: 3.8","Programming Language :: Python :: 3.9","Topic :: Scientific/Engineering","Topic :: Scientific/Engineering :: Artificial Intelligence","Topic :: Scientific/Engineering :: Mathematics","Topic :: Software Development","Topic :: Software Development :: Libraries :: Python Modules"],"description":"\n\n\n\n\n

$\"Sparsify$ Sparsify [Alpha]

\n\n

ML model optimization product to accelerate inference

\n\n

\n \n $\"Documentation\"$ \n \n \n \n \n \n \n \n \n $\"Main\"$ \n \n \n $\"GitHub$ \n \n \n $\"Stability\"$ \n \n \n $\"GitHub\"$ \n \n \n $\"Contributor$ \n \n \n \n \n \n \n \n \n \n \n

\n\n**🚨 October 2023: Important Sparsify Announcement**\n\nGiven our new focus on enabling sparse large language models (LLMs) to run competitively on CPUs, Sparsify Alpha is undergoing upgrades to focus on fine-tuning and optimizing LLMs. This means that we will no longer be providing bug fixes, prioritizing support, or building new features and integrations for non-LLM flows including the CV and NLP Sparsify Pathways.\n\nNeural Magic is super excited about these new efforts in building Sparsify into the best LLM fine-tuning and optimization tool on the market over the coming months and we cannot wait to share more soon. Thanks for your continued support! \n\n**🚨 July 2023: Sparsify's next generation is now in alpha as of version 1.6.0!**\n\nSparsify enables you to accelerate inference without sacrificing accuracy by applying state-of-the-art pruning, quantization, and distillation algorithms to neural networks with a simple web application and one-command API calls.\n\nSparsify empowers you to compress models through two components:\n- **[Sparsify Cloud](https://apps.neuralmagic.com/sparsify/)** - a web application that allows you to create and manage Sparsify Experiments, explore hyperparameters, predict performance, and compare results across both Experiments and deployment scenarios.\n- **Sparsify CLI/API** - a Python package and GitHub repository that allows you to run Sparsify Experiments locally, sync with the Sparsify Cloud, and integrate them into your workflows.\n\n## Table of Contents\n- [Quickstart Guide](#quickstart-guide)\n - [1. Install and Setup](#1-install-and-setup) \n - [2. Run an Experiment](#2-run-an-experiment) \n - [3. Compare Results](#3-compare-results) \n - [4. Deploy a Model](#4-deploy-a-model) \n- [Companion Guides](#companion-guides) \n- [Resources](#resources) \n\n## Quickstart Guide\n\n\nInterested in test-driving our alpha?\nGet a sneak peek and influence the product's development process.\nThank you in advance for your feedback and interest!\n\n\nThis quickstart details several pathways you can work through. \nWe encourage you to explore one for Sparsify's full benefits. \nWhen you finish the quickstart, sparsifying your models is as easy as:\n\n```bash\nsparsify.run sparse-transfer --use-case image-classification --data imagenette --optim-level 0.5\n```\n\n### 1. Install and Setup\n\n#### 1.1 Verify Prerequisites\n\nFirst, verify that you have the correct software and hardware to run the Sparsify Alpha.\n\n

Software

\n\nSparsify is tested on Python 3.8 and 3.10, ONNX 1.5.0-1.12.0, ONNX opset version 11+, and manylinux compliant systems. \nSparsify is not supported natively on Windows and MAC OS.\n\nAdditionally, for installation from PyPi, pip 20.3+ is required.\n

\n\n

Hardware

\n\nSparsify requires a GPU with CUDA + CuDNN in order to sparsify neural networks.\nWe recommend you use a Linux system with a GPU that has a minimum of 16GB of GPU Memory, 128GB of RAM, 4 CPU cores, and is CUDA-enabled. \nIf you are sparsifying a very large model, you may need more RAM than the recommended 128GB. \nIf you encounter issues setting up your training environment, [file a GitHub issue](https://github.com/neuralmagic/sparsify/issues).\n

\n\n#### 1.2 Create an Account\n\nCreating a new one-time account is simple and free. \nAn account is required to manage your Experiments and API keys. \nVisit the [Neural Magic's Web App Platform](https://account.neuralmagic.com/signup) and create an account by entering your email, name, and unique password. \nIf you already have a Neural Magic Account, [sign in](https://account.neuralmagic.com/signin) with your email.\n\n $\"Sparsify$ \n\n#### 1.3 Install Sparsify\n\n`pip` is the preferred method for installing Sparsify. \nIt is advised to create a [fresh virtual environment](https://docs.python.org/3/library/venv.html) to avoid dependency issues.\n\nInstall with pip using:\n```bash\npip install sparsify-nightly\n```\n\n#### 1.4 Log in via CLI\n\nNext, with Sparsify installed on your training hardware:\n1. Authorize the local CLI to access your account by running the sparsify.login command and providing your API key.\n2. Locate your API key on the homepage of the [Sparsify Cloud](https://apps.neuralmagic.com/sparsify) under the 'Get set up' modal, and copy the command or the API key itself.\n3. Run the following command:\n\n```bash\nsparsify.login API_KEY\n```\n\n### 2. Run an Experiment\n\nExperiments are the core of sparsifying a model. \nThey allow you to apply sparsification algorithms to a dataset and model through the three Experiment types detailed below: \n- [One-Shot](#21-one-shot)\n- [Training-Aware](#22-training-aware)\n- [Sparse-Transfer](#23-sparse-transfer)\n\nAll Experiments are run locally on your training hardware and can be synced with the cloud for further analysis and comparison, using Sparsify's two components:\n- Sparsify Cloud - explore hyperparameters, predict performance, and generate the desired CLI/API command.\n- Sparsify CLI/API - run an experiment.\n\n#### 2.1 One-Shot\n\n| Sparsity | Sparsification Speed | Accuracy |\n|----------|----------------------|----------|\n| **++** | **+++++** | **+++** |\n\nOne-Shot Experiments quickly sparsify your model post-training, providing a 3-5x speedup with minimal accuracy loss, ideal for quick model optimization without retraining your model.\n\nTo run a One-Shot Experiment for your model, dataset, and use case, use the following command:\n```bash\nsparsify.run one-shot --use-case USE_CASE --model MODEL --data DATASET --optim-level OPTIM_LEVEL\n```\n\nFor example, to sparsify a ResNet-50 model on the ImageNet dataset for image classification, run the following commands:\n```bash\nwget https://public.neuralmagic.com/datasets/cv/classification/imagenet_calibration.tar.gz\ntar -xzf imagenet_calibration.tar.gz -C ./imagenet_calibration\nsparsify.run one-shot --use-case image_classification --model \"zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/base-none\" --data ./imagenet_calibration --optim-level 0.5\n```\n\nOr, to sparsify a BERT model on the SST2 dataset for sentiment analysis, run the following commands:\n```bash\nwget https://public.neuralmagic.com/datasets/nlp/text_classification/sst2_bert_calibration.tar.gz\ntar -xzf sst2_bert_calibration.tar.gz\nsparsify.run one-shot --use-case text_classification --model \"zoo:nlp/sentiment_analysis/bert-base/pytorch/huggingface/sst2/base-none\" --data --data ./sst2_bert_calibration --optim-level 0.5\n```\n\nTo dive deeper into One-Shot Experiments, read through the [One-Shot Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/one-shot-experiment-guide.md).\n\n\nNote, One-Shot Experiments currently require the model to be in an ONNX format and the dataset to be in a NumPy format.\nMore details are provided in the One-Shot Experiment Guide.\n\n\n#### 2.2 Sparse-Transfer\n\n| Sparsity | Sparsification Speed | Accuracy |\n|----------|----------------------|-----------|\n| **++++** | **++++** | **+++++** |\n\nSparse-Transfer Experiments quickly create a smaller and faster model for your dataset by transferring from a [SparseZoo](https://sparsezoo.neuralmagic.com/) pre-sparsified foundational model, providing a 5-10x speedup with minimal accuracy loss, ideal for quick model optimization without retraining your model.\n\nTo run a Sparse-Transfer Experiment for your model (optional), dataset, and use case, run the following command:\n```bash\nsparsify.run sparse-transfer --use-case USE_CASE --model OPTIONAL_MODEL --data DATASET --optim-level OPTIM_LEVEL\n```\n\nFor example, to sparse transfer a SparseZoo model to the Imagenette dataset for image classification, run the following command:\n```bash\nsparsify.run sparse-transfer --use-case image_classification --data imagenette --optim-level 0.5\n```\n\nOr, to sparse transfer a SparseZoo model to the SST2 dataset for sentiment analysis, run the following command:\n```bash\nsparsify.run sparse-transfer --use-case text_classification --data sst2 --optim-level 0.5\n```\n\nTo dive deeper into Sparse-Transfer Experiments, read through the [Sparse-Transfer Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/sparse-transfer-experiment-guide.md).\n\n\nNote, Sparse-Transfer Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or Hugging Face Transformers.\nDatasets must additionally match the expected format of the underlying integration.\nMore details and exact formats are provided in the Sparse-Transfer Experiment Guide.\n\n\n#### 2.3 Training-Aware\n\n| Sparsity | Sparsification Speed | Accuracy |\n|-----------|-----------------------|-----------|\n| **+++++** | **++** | **+++++** |\n\nTraining-aware Experiments sparsify your model during training, providing a 6-12x speedup with minimal accuracy loss, ideal for thorough model optimization when the best performance and accuracy are required.\n\nTo run a Training-Aware Experiment for your model, dataset, and use case, run the following command:\n```bash\nsparsify.run training-aware --use-case USE_CASE --model OPTIONAL_MODEL --data DATASET --optim-level OPTIM_LEVEL\n```\n\nFor example, to sparsify a ResNet-50 model on the Imagenette dataset for image classification, run the following command:\n```bash\nsparsify.run training-aware --use-case image_classification --model \"zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenette/base-none\" --data imagenette --optim-level 0.5\n```\n\nOr, to sparsify a BERT model on the SST2 dataset for sentiment analysis, run the following command:\n```bash\nsparsify.run training-aware --use-case text_classification --model \"zoo:nlp/sentiment_analysis/bert-base/pytorch/huggingface/sst2/base-none\" --data sst2 --optim-level 0.5\n```\n\nTo dive deeper into Training-Aware Experiments, read through the [Training-Aware Experiment Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/training-aware-experiment-guide.md).\n\n\nNote that Training-Aware Experiments require the model to be saved in a PyTorch format corresponding to the underlying integration such as Ultralytics YOLOv5 or Hugging Face Transformers.\nDatasets must additionally match the expected format of the underlying integration.\nMore details and exact formats are provided in the Training-Aware Experiment Guide.\n\n\n### 3. Compare Results\n\nOnce you have run your Experiment, the results, logs, and deployment files will be saved under the current working directory in the following format:\n```text\n[EXPERIMENT_TYPE]_[USE_CASE]_{DATE_TIME}\n├── deployment\n│ ├── model.onnx\n│ └── *supporting files*\n├── logs\n│ ├── *logs*\n├── training_artifacts\n│ ├── *training artifacts*\n ├── *metrics and results*\n```\n\nYou can compare the accuracy by looking through the metrics printed out to the console and the metrics saved in the experiment directory.\nAdditionally, you can use [DeepSparse](https://github.com/neuralmagic/deepsparse) to compare the inference performance on your CPU deployment hardware.\n\n\nNote: In the near future, you will be able to visualize the results in Sparsify Cloud, simulate other scenarios and hyperparameters, compare the results to other Experiments, and package for your deployment scenario.\n\n\nTo run a benchmark on your deployment hardware, use the `deepsparse.benchmark` command with your original model and the new optimized model.\nThis will run a number of inferences to simulate a real-world scenario and print out the results.\n\nIt's as simple as running the following command:\n```bash\ndeepsparse.benchmark --model_path MODEL --scenario SCENARIO\n```\n\nFor example, to benchmark a dense ResNet-50 model, run the following command:\n```bash\ndeepsparse.benchmark --model_path \"zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenette/base-none\" --scenario sync\n```\n\nThis can then be compared to the sparsified ResNet-50 model with the following command:\n```bash\ndeepsparse.benchmark --model_path \"zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none\" --scenario sync\n```\n\nThe output will look similar to the following:\n```text\nDeepSparse, Copyright 2021-present / Neuralmagic, Inc. version: 1.6.0.20230629 COMMUNITY | (fc8b788a) (release) (optimized) (system=avx512, binary=avx512)\ndeepsparse.benchmark.benchmark_model INFO deepsparse.engine.Engine:\n\tonnx_file_path: ./model.onnx\n\tbatch_size: 1\n\tnum_cores: 1\n\tnum_streams: 1\n\tscheduler: Scheduler.default\n\tfraction_of_supported_ops: 0.9981\n\tcpu_avx_type: avx512\n\tcpu_vnni: False\n=Original Model Path: ./model.onnx\nBatch Size: 1\nScenario: sync\nThroughput (items/sec): 134.5611\nLatency Mean (ms/batch): 7.4217\nLatency Median (ms/batch): 7.4245\nLatency Std (ms/batch): 0.0264\nIterations: 1346\n```\n\nSee the [DeepSparse Benchmarking User Guide](https://github.com/neuralmagic/deepsparse/blob/main/docs/user-guide/deepsparse-benchmarking.md) for more information on benchmarking.\n\n### 4. Deploy a Model\n\nAs an optional step to this quickstart, now that you have your optimized model, you are ready for inferencing. \nTo get the most inference performance out of your optimized model, we recommend you deploy on Neural Magic's [DeepSparse](https://docs.neuralmagic.com/deepsparse). \nDeepSparse is built to get the best performance out of optimized models on CPUs. \n\nDeepSparse Server takes in a task and a model path and will enable you to serve models and `Pipelines` for deployment in HTTP.\n\nYou can deploy any ONNX model using DeepSparse Server with the following command:\n```bash\ndeepsparse.server --task USE_CASE --model_path MODEL_PATH\n```\n\nWhere `USE_CASE` is the use case of your Experiment and `MODEL_PATH` is the path to the deployment folder from the Experiment.\n\nFor example, to deploy a sparsified ResNet-50 model, run the following command:\n```bash\ndeepsparse.server --task image_classification --model_path \"zoo:cv/classification/resnet_v1-50/pytorch/sparseml/imagenet/pruned95_quant-none\"\n```\n\nIf you're not ready for deploying, congratulations on completing the quickstart!\n\n## Companion Guides\n\n- [Sparsify Cloud User Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/cloud-user-guide.md)\n- [Sparsify Datasets Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/datasets-guide.md)\n- [Sparsify Models Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/models-guide.md)\n- [One-Shot Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/one-shot-experiment-guide.md)\n- [Sparse-Transfer Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/sparse-transfer-experiment-guide.md)\n- [Training-Aware Experiments Guide](https://github.com/neuralmagic/sparsify/blob/main/docs/training-aware-experiment-guide.md)\n\n## Resources\n\nNow that you have explored Sparsify [Alpha], here are other related resources.\n\n### Feedback and Support\n\nReport UI issues and CLI errors, submit bug reports, and provide general feedback about the product to the Sparsify team via the [nm-sparsify Slack Channel](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-1xkdlzwv9-2rvS6yQcCs7VDNUcWxctnw), or via [GitHub Issues](https://github.com/neuralmagic/sparsify/issues). Alpha support is provided through those channels.\n\n### Terms and Conditions\n\nSparsify Alpha is a pre-release version of Sparsify that is still in active development. \nThe product is not yet ready for production use; APIs and UIs are subject to change. \nThere may be bugs in the Alpha version, which we hope to have fixed before Beta and then a general Q3 2023 release. \nThe feedback you provide on quality and usability helps us identify issues, fix them, and make Sparsify even better. \nThis information is used internally by Neural Magic solely for that purpose. \nIt is not shared or used in any other way.\n\nThat being said, we are excited to share this release and hear what you think. \nThank you in advance for your feedback and interest!\n\n### Learning More\n\n- Documentation: [SparseML](https://docs.neuralmagic.com/sparseml/), [SparseZoo](https://docs.neuralmagic.com/sparsezoo/), [Sparsify](https://docs.neuralmagic.com/archive/sparsify/), [DeepSparse](https://docs.neuralmagic.com/deepsparse/)\n- Neural Magic: [Blog,](https://www.neuralmagic.com/blog/) [Resources](https://www.neuralmagic.com/resources/)\n\n### Release History\n\nOfficial builds are hosted on PyPI\n\n- stable: [sparsify](https://pypi.org/project/sparsify/)\n- nightly (dev): [sparsify-nightly](https://pypi.org/project/sparsify-nightly/)\n\nAdditionally, more information can be found via [GitHub Releases.](https://github.com/neuralmagic/sparsify/releases)\n\n### License\n\nThe project is licensed under the [Apache License Version 2.0](https://github.com/neuralmagic/sparsify/blob/main/LICENSE).\n\n## Community\n\n### Contribute\n\nWe appreciate contributions to the code, examples, integrations, and documentation as well as bug reports and feature requests! [Learn how here.](https://github.com/neuralmagic/sparsify/blob/main/CONTRIBUTING.md)\n\n### Join\n\nFor user help or questions about Sparsify, sign up or log in to our [**Neural Magic Community Slack**](https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ). We are growing the community member by member and happy to see you there. Bugs, feature requests, or additional questions can also be posted to our [GitHub Issue Queue.](https://github.com/neuralmagic/sparsify/issues)\n\nYou can get the latest news, webinar and event invites, research papers, and other ML Performance tidbits by [subscribing](https://neuralmagic.com/subscribe/) to the Neural Magic community.\n\nFor more general questions about Neural Magic, please fill out this [form.](http://neuralmagic.com/contact/)\n\n### Cite\n\nFind this project useful in your research or other communications? \nPlease consider citing:\n\n```bibtex\n@InProceedings{\n pmlr-v119-kurtz20a, \n title = {Inducing and Exploiting Activation Sparsity for Fast Inference on Deep Neural Networks}, \n author = {Kurtz, Mark and Kopinsky, Justin and Gelashvili, Rati and Matveev, Alexander and Carr, John and Goin, Michael and Leiserson, William and Moore, Sage and Nell, Bill and Shavit, Nir and Alistarh, Dan}, \n booktitle = {Proceedings of the 37th International Conference on Machine Learning}, \n pages = {5533--5543}, \n year = {2020}, \n editor = {Hal Daumé III and Aarti Singh}, \n volume = {119}, \n series = {Proceedings of Machine Learning Research}, \n address = {Virtual}, \n month = {13--18 Jul}, \n publisher = {PMLR}, \n pdf = {http://proceedings.mlr.press/v119/kurtz20a/kurtz20a.pdf},\n url = {http://proceedings.mlr.press/v119/kurtz20a.html}, \n abstract = {Optimizing convolutional neural networks for fast inference has recently become an extremely active area of research. One of the go-to solutions in this context is weight pruning, which aims to reduce computational and memory footprint by removing large subsets of the connections in a neural network. Surprisingly, much less attention has been given to exploiting sparsity in the activation maps, which tend to be naturally sparse in many settings thanks to the structure of rectified linear (ReLU) activation functions. In this paper, we present an in-depth analysis of methods for maximizing the sparsity of the activations in a trained neural network, and show that, when coupled with an efficient sparse-input convolution algorithm, we can leverage this sparsity for significant performance gains. To induce highly sparse activation maps without accuracy loss, we introduce a new regularization technique, coupled with a new threshold-based sparsification method based on a parameterized activation function called Forced-Activation-Threshold Rectified Linear Unit (FATReLU). We examine the impact of our methods on popular image classification models, showing that most architectures can adapt to significantly sparser activation maps without any accuracy loss. Our second contribution is showing that these these compression gains can be translated into inference speedups: we provide a new algorithm to enable fast convolution operations over networks with sparse activations, and show that it can enable significant speedups for end-to-end inference on a range of popular models on the large-scale ImageNet image classification task on modern Intel CPUs, with little or no retraining cost.} \n}\n```\n\n```bibtex\n@misc{\n singh2020woodfisher,\n title={WoodFisher: Efficient Second-Order Approximation for Neural Network Compression}, \n author={Sidak Pal Singh and Dan Alistarh},\n year={2020},\n eprint={2004.14340},\n archivePrefix={arXiv},\n primaryClass={cs.LG}\n}\n```\n\n\n","description_content_type":"text/markdown","docs_url":null,"download_url":"","downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/neuralmagic/sparsify","keywords":"inference,machine learning,neural network,computer vision,nlp,cv,deep learning,torch,pytorch,tensorflow,keras,automl,sparsity,pruning,deep learning libraries,onnx,quantization","license":"Apache","license_expression":null,"license_files":null,"maintainer":"","maintainer_email":"","name":"sparsify","package_url":"https://pypi.org/project/sparsify/","platform":null,"project_url":"https://pypi.org/project/sparsify/","project_urls":{"Homepage":"https://github.com/neuralmagic/sparsify"},"provides_extra":null,"release_url":"https://pypi.org/project/sparsify/1.6.1/","requires_dist":null,"requires_python":">=3.8.0","summary":"Easy-to-use UI for automatically sparsifying neural networks and creating sparsification recipes for better inference performance and a smaller footprint","version":"1.6.1","yanked":false,"yanked_reason":null}

Package search

sparsify

project_urls

Dependencies

All package metadata

Releases