{"author":"Elias Bassani","author_email":"elias.bssn@gmail.com","bugtrack_url":null,"classifiers":["Intended Audience :: Science/Research","License :: OSI Approved :: MIT License","Operating System :: OS Independent","Programming Language :: Python :: 3","Topic :: Text Processing :: General"],"description":"

\n\n

\n \n \n \n $\"PyPI$ \n \n $\"Download$ \n \n .svg\" alt=\"Documentation Status\">\n \n \n \n $\"License:$ \n \n $\"Open$ \n

\n\n\n\n## ⚡️ Introduction\n\n[ranx](https://github.com/AmenRa/ranx) ([raŋks]) is a library of fast ranking evaluation metrics implemented in [Python](https://en.wikipedia.org/wiki/Python_(programming_language)), leveraging [Numba](https://github.com/numba/numba) for high-speed [vector operations](https://en.wikipedia.org/wiki/Automatic_vectorization) and [automatic parallelization](https://en.wikipedia.org/wiki/Automatic_parallelization).\nIt offers a user-friendly interface to evaluate and compare [Information Retrieval](https://en.wikipedia.org/wiki/Information_retrieval) and [Recommender Systems](https://en.wikipedia.org/wiki/Recommender_system).\n[ranx](https://github.com/AmenRa/ranx) allows you to perform statistical tests and export [LaTeX](https://en.wikipedia.org/wiki/LaTeX) tables for your scientific publications.\nMoreover, [ranx](https://github.com/AmenRa/ranx) provides several [fusion algorithms](https://amenra.github.io/ranx/fusion) and [normalization strategies](https://amenra.github.io/ranx/normalization), and an automatic [fusion optimization](https://amenra.github.io/ranx/fusion/#optimize-fusion) functionality.\n[ranx](https://github.com/AmenRa/ranx) also have a companion repository of pre-computed runs to facilitated model comparisons called [ranxhub](https://amenra.github.io/ranxhub).\nOn [ranxhub](https://amenra.github.io/ranxhub), you can download and share pre-computed runs for Information Retrieval datasets, such as [MSMARCO Passage Ranking](https://arxiv.org/abs/1611.09268).\n[ranx](https://github.com/AmenRa/ranx) was featured in [ECIR 2022](https://ecir2022.org), [CIKM 2022](https://www.cikm2022.org), and [SIGIR 2023](https://sigir.org/sigir2023). \n \nIf you use [ranx](https://github.com/AmenRa/ranx) to evaluate results or conducting experiments involving fusion for your scientific publication, please consider citing it: [evaluation bibtex](https://dblp.org/rec/conf/ecir/Bassani22.html?view=bibtex), [fusion bibtex](https://dblp.org/rec/conf/cikm/BassaniR22.html?view=bibtex), [ranxhub bibtex](https://dblp.org/rec/conf/sigir/Bassani23.html?view=bibtex).\n\nNB: [ranx](https://github.com/AmenRa/ranx) is not suited for evaluating classifiers. Please, refer to the [FAQ](https://amenra.github.io/ranx/faq) for further details.\n\nFor a quick overview, follow the [Usage](#-usage) section.\n\nFor a in-depth overview, follow the [Examples](#-examples) section.\n\n\n## ✨ Features\n\n### Metrics\n* [Hits](https://amenra.github.io/ranx/metrics/#hits)\n* [Hit Rate](https://amenra.github.io/ranx/metrics/#hit-rate-success)\n* [Precision](https://amenra.github.io/ranx/metrics/#precision)\n* [Recall](https://amenra.github.io/ranx/metrics/#recall)\n* [F1](https://amenra.github.io/ranx/metrics/#f1)\n* [r-Precision](https://amenra.github.io/ranx/metrics/#r-precision)\n* [Bpref](https://amenra.github.io/ranx/metrics/#bpref)\n* [Rank-biased Precision (RBP)](https://amenra.github.io/ranx/metrics/#rank-biased-precision)\n* [Mean Reciprocal Rank (MRR)](https://amenra.github.io/ranx/metrics/#mean-reciprocal-rank)\n* [Mean Average Precision (MAP)](https://amenra.github.io/ranx/metrics/#mean-average-precision)\n* [Discounted Cumulative Gain (DCG)](https://amenra.github.io/ranx/metrics/#dcg)\n* [Normalized Discounted Cumulative Gain (NDCG)](https://amenra.github.io/ranx/metrics/#ndcg)\n\nThe metrics have been tested against [TREC Eval](https://github.com/usnistgov/trec_eval) for correctness.\n\n### Statistical Tests\n* [Paired Student's t-Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/t_test.htm) (default)\n* [Fisher's Randomization Test](https://www.itl.nist.gov/div898/software/dataplot/refman1/auxillar/fishrand.htm)\n* [Tukey's HSD Test](https://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm)\n\nPlease, refer to [Smucker et al.](https://dl.acm.org/doi/10.1145/1321440.1321528), [Carterette](https://dl.acm.org/doi/10.1145/2094072.2094076), and [Fuhr](http://www.sigir.org/wp-content/uploads/2018/01/p032.pdf) for additional information on statistical tests for Information Retrieval.\n\n### Off-the-shelf Qrels\nYou can load qrels from [ir-datasets](https://ir-datasets.com) as simply as:\n```python\nqrels = Qrels.from_ir_datasets(\"msmarco-document/dev\")\n```\nA full list of the available qrels is provided [here](https://ir-datasets.com).\n\n### Off-the-shelf Runs\nYou can load runs from [ranxhub](https://amenra.github.io/ranxhub/) as simply as:\n```python\nrun = Run.from_ranxhub(\"run-id\")\n```\nA full list of the available runs is provided [here](https://amenra.github.io/ranxhub//browse).\n\n### Fusion Algorithms\n\n| **Name** | **Name** | **Name** | **Name** | **Name** |\n| -------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------------------------ |\n| [CombMIN](https://amenra.github.io/ranx/fusion/#combmin) | [CombMNZ](https://amenra.github.io/ranx/fusion/#combmnz) | [RRF](https://amenra.github.io/ranx/fusion/#reciprocal-rank-fusion-rrf) | [MAPFuse](https://amenra.github.io/ranx/fusion/#mapfuse) | [BordaFuse](https://amenra.github.io/ranx/fusion/#bordafuse) |\n| [CombMED](https://amenra.github.io/ranx/fusion/#combmed) | [CombGMNZ](https://amenra.github.io/ranx/fusion/#combgmnz) | [RBC](https://amenra.github.io/ranx/fusion/#rank-biased-centroids-rbc) | [PosFuse](https://amenra.github.io/ranx/fusion/#posfuse) | [Weighted BordaFuse](https://amenra.github.io/ranx/fusion/#weighted-bordafuse) |\n| [CombANZ](https://amenra.github.io/ranx/fusion/#combanz) | [ISR](https://amenra.github.io/ranx/fusion/#isr) | [WMNZ](https://amenra.github.io/ranx/fusion/#wmnz) | [ProbFuse](https://amenra.github.io/ranx/fusion/#probfuse) | [Condorcet](https://amenra.github.io/ranx/fusion/#condorcet) |\n| [CombMAX](https://amenra.github.io/ranx/fusion/#combmax) | [Log_ISR](https://amenra.github.io/ranx/fusion/#log_isr) | [Mixed](https://amenra.github.io/ranx/fusion/#mixed) | [SegFuse](https://amenra.github.io/ranx/fusion/#segfuse) | [Weighted Condorcet](https://amenra.github.io/ranx/fusion/#weighted-condorcet) |\n| [CombSUM](https://amenra.github.io/ranx/fusion/#combsum) | [LogN_ISR](https://amenra.github.io/ranx/fusion/#logn_isr) | [BayesFuse](https://amenra.github.io/ranx/fusion/#bayesfuse) | [SlideFuse](https://amenra.github.io/ranx/fusion/#slidefuse) | [Weighted Sum](https://amenra.github.io/ranx/fusion/#wighted-sum) |\n\nPlease, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.\n\n### Normalization Strategies\n\n* [Min-Max Norm](https://amenra.github.io/ranx/normalization/#min-max-norm) \n* [Min-Max Inverted Norm](https://amenra.github.io/ranx/normalization/#min-max-inverted-norm) \n* [Max Norm](https://amenra.github.io/ranx/normalization/#sum-norm) \n* [Sum Norm](https://amenra.github.io/ranx/normalization/#rank-norm) \n* [ZMUV Norm](https://amenra.github.io/ranx/normalization/#max-norm) \n* [Rank Norm](https://amenra.github.io/ranx/normalization/#zmuv-norm) \n* [Borda Norm](https://amenra.github.io/ranx/normalization/#borda-norm)\n\nPlease, refer to the [documentation](https://amenra.github.io/ranx/fusion) for further details.\n\n\n## 🔌 Requirements\n```bash\npython>=3.8\n```\nAs of `v.0.3.5`, [ranx](https://github.com/AmenRa/ranx) requires `python>=3.8`.\n\n## 💾 Installation \n\n```bash\npip install ranx\n```\n\n## 💡 Usage\n\n### Create Qrels and Run\n```python\nfrom ranx import Qrels, Run\n\nqrels_dict = { \"q_1\": { \"d_12\": 5, \"d_25\": 3 },\n \"q_2\": { \"d_11\": 6, \"d_22\": 1 } }\n\nrun_dict = { \"q_1\": { \"d_12\": 0.9, \"d_23\": 0.8, \"d_25\": 0.7,\n \"d_36\": 0.6, \"d_32\": 0.5, \"d_35\": 0.4 },\n \"q_2\": { \"d_12\": 0.9, \"d_11\": 0.8, \"d_25\": 0.7,\n \"d_36\": 0.6, \"d_22\": 0.5, \"d_35\": 0.4 } }\n\nqrels = Qrels(qrels_dict)\nrun = Run(run_dict)\n```\n\n### Evaluate\n```python\nfrom ranx import evaluate\n\n# Compute score for a single metric\nevaluate(qrels, run, \"ndcg@5\")\n>>> 0.7861\n\n# Compute scores for multiple metrics at once\nevaluate(qrels, run, [\"map@5\", \"mrr\"])\n>>> {\"map@5\": 0.6416, \"mrr\": 0.75}\n```\n\n### Compare\n```python\nfrom ranx import compare\n\n# Compare different runs and perform Two-sided Paired Student's t-Test\nreport = compare(\n qrels=qrels,\n runs=[run_1, run_2, run_3, run_4, run_5],\n metrics=[\"map@100\", \"mrr@100\", \"ndcg@10\"],\n max_p=0.01 # P-value threshold\n)\n```\nOutput:\n```python\nprint(report)\n```\n```\n# Model MAP@100 MRR@100 NDCG@10\n--- ------- -------- -------- ---------\na model_1 0.320ᵇ 0.320ᵇ 0.368ᵇᶜ\nb model_2 0.233 0.234 0.239\nc model_3 0.308ᵇ 0.309ᵇ 0.330ᵇ\nd model_4 0.366ᵃᵇᶜ 0.367ᵃᵇᶜ 0.408ᵃᵇᶜ\ne model_5 0.405ᵃᵇᶜᵈ 0.406ᵃᵇᶜᵈ 0.451ᵃᵇᶜᵈ\n```\n\n### Fusion\n```python\nfrom ranx import fuse, optimize_fusion\n\nbest_params = optimize_fusion(\n qrels=train_qrels,\n runs=[train_run_1, train_run_2, train_run_3],\n norm=\"min-max\", # The norm. to apply before fusion\n method=\"wsum\", # The fusion algorithm to use (Weighted Sum)\n metric=\"ndcg@100\", # The metric to maximize\n)\n\ncombined_test_run = fuse(\n runs=[test_run_1, test_run_2, test_run_3], \n norm=\"min-max\", \n method=\"wsum\", \n params=best_params,\n)\n```\n\n## 📖 Examples\n\n| Name | Link |\n| ---------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| Overview | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/1_overview.ipynb) |\n| Qrels and Run | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/2_qrels_and_run.ipynb) |\n| Evaluation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/3_evaluation.ipynb) |\n| Comparison and Report | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/4_comparison_and_report.ipynb) |\n| Fusion | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/5_fusion.ipynb) |\n| Plot | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/7_plot.ipynb) |\n| Share your runs with [ranxhub](https://amenra.github.io/ranxhub) | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AmenRa/ranx/blob/master/notebooks/6_ranxhub.ipynb) |\n\n\n## 📚 Documentation\nBrowse the [documentation](https://amenra.github.io/ranx) for more details and examples.\n\n\n## 🎓 Citation\nIf you use [ranx](https://github.com/AmenRa/ranx) to evaluate results for your scientific publication, please consider citing our [ECIR 2022](https://ecir2022.org) paper:\n

BibTeX

\n \n ```bibtex\n @inproceedings{ranx,\n author = {Elias Bassani},\n title = {ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison},\n booktitle = {{ECIR} {(2)}},\n series = {Lecture Notes in Computer Science},\n volume = {13186},\n pages = {259--264},\n publisher = {Springer},\n year = {2022},\n doi = {10.1007/978-3-030-99739-7\\_30}\n }\n ```\n

\n\nIf you use the fusion functionalities provided by [ranx](https://github.com/AmenRa/ranx) for conducting the experiments of your scientific publication, please consider citing our [CIKM 2022](https://www.cikm2022.org) paper:\n

BibTeX

\n \n ```bibtex\n @inproceedings{ranx.fuse,\n author = {Elias Bassani and\n Luca Romelli},\n title = {ranx.fuse: {A} Python Library for Metasearch},\n booktitle = {{CIKM}},\n pages = {4808--4812},\n publisher = {{ACM}},\n year = {2022},\n doi = {10.1145/3511808.3557207}\n }\n ```\n

\n\nIf you use pre-computed runs from [ranxhub](https://amenra.github.io/ranxhub) to make comparison for your scientific publication, please consider citing our [SIGIR 2023](https://sigir.org/sigir2023) paper:\n

BibTeX

\n \n ```bibtex\n @inproceedings{ranxhub,\n author = {Elias Bassani},\n title = {ranxhub: An Online Repository for Information Retrieval Runs},\n booktitle = {{SIGIR}},\n pages = {3210--3214},\n publisher = {{ACM}},\n year = {2023},\n doi = {10.1145/3539618.3591823}\n }\n ```\n

\n\n## 🎁 Feature Requests\nWould you like to see other features implemented? Please, open a [feature request](https://github.com/AmenRa/ranx/issues/new?assignees=&labels=enhancement&template=feature_request.md&title=%5BFeature+Request%5D+title).\n\n\n## 🤘 Want to contribute?\nWould you like to contribute? Please, drop me an [e-mail](mailto:elias.bssn@gmail.com?subject=[GitHub]%20ranx).\n\n\n## 📄 License\n[ranx](https://github.com/AmenRa/ranx) is an open-sourced software licensed under the [MIT license](LICENSE).\n","description_content_type":"text/markdown","docs_url":null,"download_url":null,"downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/AmenRa/ranx","keywords":"trec_eval, information retrieval, recommender systems, evaluation, ranking, fusion, metasearch, numba","license":null,"license_expression":null,"license_files":null,"maintainer":null,"maintainer_email":null,"name":"ranx","package_url":"https://pypi.org/project/ranx/","platform":null,"project_url":"https://pypi.org/project/ranx/","project_urls":{"Homepage":"https://github.com/AmenRa/ranx"},"provides_extra":null,"release_url":"https://pypi.org/project/ranx/0.3.20/","requires_dist":["numpy","numba>=0.54.1","pandas","tabulate","tqdm","scipy>=1.8.0","ir-datasets","rich","orjson","lz4","cbor2","seaborn","fastparquet"],"requires_python":">=3.8","summary":"ranx: A Blazing-Fast Python Library for Ranking Evaluation, Comparison, and Fusion","version":"0.3.20","yanked":false,"yanked_reason":null}

Dependency	Optional
cbor2	false
fastparquet	false
ir-datasets	false
lz4	false
numba	false
numpy	false
orjson	false
pandas	false
rich	false
scipy	false
seaborn	false
tabulate	false
tqdm	false

Package search

ranx

project_urls

Dependencies

All package metadata

Releases