{"author":null,"author_email":"DrivenData ","bugtrack_url":null,"classifiers":["Intended Audience :: Developers","License :: OSI Approved :: MIT License","Operating System :: OS Independent","Programming Language :: Python :: 3","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.12","Programming Language :: Python :: 3.13","Programming Language :: Python :: 3.9"],"description":"![](https://raw.githubusercontent.com/drivendataorg/cloudpathlib/master/docs/docs/logo.svg)\n\n

\n\n[![Docs Status](https://img.shields.io/badge/docs-stable-informational)](https://cloudpathlib.drivendata.org/)\n[![PyPI](https://img.shields.io/pypi/v/cloudpathlib.svg)](https://pypi.org/project/cloudpathlib/)\n[![conda-forge](https://img.shields.io/conda/vn/conda-forge/cloudpathlib.svg)](https://anaconda.org/conda-forge/cloudpathlib)\n[![conda-forge feedstock](https://img.shields.io/badge/conda--forge-feedstock-yellowgreen)](https://github.com/conda-forge/cloudpathlib-feedstock)\n[![tests](https://github.com/drivendataorg/cloudpathlib/actions/workflows/tests.yml/badge.svg?branch=master)](https://github.com/drivendataorg/cloudpathlib/actions/workflows/tests.yml?query=branch%3Amaster)\n[![codecov](https://codecov.io/gh/drivendataorg/cloudpathlib/branch/master/graph/badge.svg)](https://codecov.io/gh/drivendataorg/cloudpathlib)\n\n> Our goal is to be the meringue of file management libraries: the subtle sweetness of `pathlib` working in harmony with the ethereal lightness of the cloud.\n\nA Python library with classes that mimic `pathlib.Path`'s interface for URIs from different cloud storage services.\n\n```python\nwith CloudPath(\"s3://bucket/filename.txt\").open(\"w+\") as f:\n f.write(\"Send my changes to the cloud!\")\n```\n\n## Why use cloudpathlib?\n\n - **Familiar**: If you know how to interact with `Path`, you know how to interact with `CloudPath`. All of the cloud-relevant `Path` methods are implemented.\n - **Supported clouds**: AWS S3, Google Cloud Storage, and Azure Blob Storage are implemented. FTP is on the way.\n - **Extensible**: The base classes do most of the work generically, so implementing two small classes `MyPath` and `MyClient` is all you need to add support for a new cloud storage service.\n - **Read/write support**: Reading just works. Using the `write_text`, `write_bytes` or `.open('w')` methods will all upload your changes to cloud storage without any additional file management as a developer.\n - **Seamless caching**: Files are downloaded locally only when necessary. You can also easily pass a persistent cache folder so that across processes and sessions you only re-download what is necessary.\n - **Tested**: Comprehensive test suite and code coverage.\n - **Testability**: Local filesystem implementations that can be used to easily mock cloud storage in your unit tests.\n\n\n## Installation\n\n`cloudpathlib` depends on the cloud services' SDKs (e.g., `boto3`, `google-cloud-storage`, `azure-storage-blob`) to communicate with their respective storage service. If you try to use cloud paths for a cloud service for which you don't have dependencies installed, `cloudpathlib` will error and let you know what you need to install.\n\nTo install a cloud service's SDK dependency when installing `cloudpathlib`, you need to specify it using pip's [\"extras\"](https://packaging.python.org/tutorials/installing-packages/#installing-setuptools-extras) specification. For example:\n\n```bash\npip install cloudpathlib[s3,gs,azure]\n```\n\nWith some shells, you may need to use quotes:\n\n```bash\npip install \"cloudpathlib[s3,gs,azure]\"\n```\n\nCurrently supported cloud storage services are: `azure`, `gs`, `s3`. You can also use `all` to install all available services' dependencies.\n\nIf you do not specify any extras or separately install any cloud SDKs, you will only be able to develop with the base classes for rolling your own cloud path class.\n\n### conda\n\n`cloudpathlib` is also available using `conda` from conda-forge. Note that to install the necessary cloud service SDK dependency, you should include the appropriate suffix in the package name. For example:\n\n```bash\nconda install cloudpathlib-s3 -c conda-forge\n```\n\nIf no suffix is used, only the base classes will be usable. See the [conda-forge/cloudpathlib-feedstock](https://github.com/conda-forge/cloudpathlib-feedstock) for all installation options.\n\n### Development version\n\nYou can get latest development version from GitHub:\n\n```bash\npip install https://github.com/drivendataorg/cloudpathlib.git#egg=cloudpathlib[all]\n```\n\nNote that you similarly need to specify cloud service dependencies, such as `all` in the above example command.\n\n## Quick usage\n\nHere's an example to get the gist of using the package. By default, `cloudpathlib` authenticates with the environment variables supported by each respective cloud service SDK. For more details and advanced authentication options, see the [\"Authentication\"](https://cloudpathlib.drivendata.org/stable/authentication/) documentation.\n\n```python\nfrom cloudpathlib import CloudPath\n\n# dispatches to S3Path based on prefix\nroot_dir = CloudPath(\"s3://drivendata-public-assets/\")\nroot_dir\n#> S3Path('s3://drivendata-public-assets/')\n\n# there's only one file, but globbing works in nested folder\nfor f in root_dir.glob('**/*.txt'):\n text_data = f.read_text()\n print(f)\n print(text_data)\n#> s3://drivendata-public-assets/odsc-west-2019/DATA_DICTIONARY.txt\n#> Eviction Lab Data Dictionary\n#>\n#> Additional information in our FAQ evictionlab.org/help-faq/\n#> Full methodology evictionlab.org/methods/\n#>\n#> ... (additional text output truncated)\n\n# use / to join paths (and, in this case, create a new file)\nnew_file_copy = root_dir / \"nested_dir/copy_file.txt\"\nnew_file_copy\n#> S3Path('s3://drivendata-public-assets/nested_dir/copy_file.txt')\n\n# show things work and the file does not exist yet\nnew_file_copy.exists()\n#> False\n\n# writing text data to the new file in the cloud\nnew_file_copy.write_text(text_data)\n#> 6933\n\n# file now listed\nlist(root_dir.glob('**/*.txt'))\n#> [S3Path('s3://drivendata-public-assets/nested_dir/copy_file.txt'),\n#> S3Path('s3://drivendata-public-assets/odsc-west-2019/DATA_DICTIONARY.txt')]\n\n# but, we can remove it\nnew_file_copy.unlink()\n\n# no longer there\nlist(root_dir.glob('**/*.txt'))\n#> [S3Path('s3://drivendata-public-assets/odsc-west-2019/DATA_DICTIONARY.txt')]\n```\n\n## Supported methods and properties\n\nMost methods and properties from `pathlib.Path` are supported except for the ones that don't make sense in a cloud context. There are a few additional methods or properties that relate to specific cloud services or specifically for cloud paths.\n\n| Methods + properties | `AzureBlobPath` | `S3Path` | `GSPath` |\n|:-----------------------|:------------------|:-----------|:-----------|\n| `absolute` | ✅ | ✅ | ✅ |\n| `anchor` | ✅ | ✅ | ✅ |\n| `as_uri` | ✅ | ✅ | ✅ |\n| `drive` | ✅ | ✅ | ✅ |\n| `exists` | ✅ | ✅ | ✅ |\n| `glob` | ✅ | ✅ | ✅ |\n| `is_absolute` | ✅ | ✅ | ✅ |\n| `is_dir` | ✅ | ✅ | ✅ |\n| `is_file` | ✅ | ✅ | ✅ |\n| `is_relative_to` | ✅ | ✅ | ✅ |\n| `iterdir` | ✅ | ✅ | ✅ |\n| `joinpath` | ✅ | ✅ | ✅ |\n| `match` | ✅ | ✅ | ✅ |\n| `mkdir` | ✅ | ✅ | ✅ |\n| `name` | ✅ | ✅ | ✅ |\n| `open` | ✅ | ✅ | ✅ |\n| `parent` | ✅ | ✅ | ✅ |\n| `parents` | ✅ | ✅ | ✅ |\n| `parts` | ✅ | ✅ | ✅ |\n| `read_bytes` | ✅ | ✅ | ✅ |\n| `read_text` | ✅ | ✅ | ✅ |\n| `relative_to` | ✅ | ✅ | ✅ |\n| `rename` | ✅ | ✅ | ✅ |\n| `replace` | ✅ | ✅ | ✅ |\n| `resolve` | ✅ | ✅ | ✅ |\n| `rglob` | ✅ | ✅ | ✅ |\n| `rmdir` | ✅ | ✅ | ✅ |\n| `samefile` | ✅ | ✅ | ✅ |\n| `stat` | ✅ | ✅ | ✅ |\n| `stem` | ✅ | ✅ | ✅ |\n| `suffix` | ✅ | ✅ | ✅ |\n| `suffixes` | ✅ | ✅ | ✅ |\n| `touch` | ✅ | ✅ | ✅ |\n| `unlink` | ✅ | ✅ | ✅ |\n| `with_name` | ✅ | ✅ | ✅ |\n| `with_stem` | ✅ | ✅ | ✅ |\n| `with_suffix` | ✅ | ✅ | ✅ |\n| `write_bytes` | ✅ | ✅ | ✅ |\n| `write_text` | ✅ | ✅ | ✅ |\n| `as_posix` | ❌ | ❌ | ❌ |\n| `chmod` | ❌ | ❌ | ❌ |\n| `cwd` | ❌ | ❌ | ❌ |\n| `expanduser` | ❌ | ❌ | ❌ |\n| `group` | ❌ | ❌ | ❌ |\n| `hardlink_to` | ❌ | ❌ | ❌ |\n| `home` | ❌ | ❌ | ❌ |\n| `is_block_device` | ❌ | ❌ | ❌ |\n| `is_char_device` | ❌ | ❌ | ❌ |\n| `is_fifo` | ❌ | ❌ | ❌ |\n| `is_mount` | ❌ | ❌ | ❌ |\n| `is_reserved` | ❌ | ❌ | ❌ |\n| `is_socket` | ❌ | ❌ | ❌ |\n| `is_symlink` | ❌ | ❌ | ❌ |\n| `lchmod` | ❌ | ❌ | ❌ |\n| `link_to` | ❌ | ❌ | ❌ |\n| `lstat` | ❌ | ❌ | ❌ |\n| `owner` | ❌ | ❌ | ❌ |\n| `readlink` | ❌ | ❌ | ❌ |\n| `root` | ❌ | ❌ | ❌ |\n| `symlink_to` | ❌ | ❌ | ❌ |\n| `as_url` | ✅ | ✅ | ✅ |\n| `clear_cache` | ✅ | ✅ | ✅ |\n| `cloud_prefix` | ✅ | ✅ | ✅ |\n| `copy` | ✅ | ✅ | ✅ |\n| `copytree` | ✅ | ✅ | ✅ |\n| `download_to` | ✅ | ✅ | ✅ |\n| `etag` | ✅ | ✅ | ✅ |\n| `fspath` | ✅ | ✅ | ✅ |\n| `is_junction` | ✅ | ✅ | ✅ |\n| `is_valid_cloudpath` | ✅ | ✅ | ✅ |\n| `rmtree` | ✅ | ✅ | ✅ |\n| `upload_from` | ✅ | ✅ | ✅ |\n| `validate` | ✅ | ✅ | ✅ |\n| `walk` | ✅ | ✅ | ✅ |\n| `with_segments` | ✅ | ✅ | ✅ |\n| `blob` | ✅ | ❌ | ✅ |\n| `bucket` | ❌ | ✅ | ✅ |\n| `container` | ✅ | ❌ | ❌ |\n| `key` | ❌ | ✅ | ❌ |\n| `md5` | ✅ | ❌ | ✅ |\n\n----\n\n^{Icon made by srip from www.flaticon.com.}\n
^{Sample code block generated using the [reprexpy package](https://github.com/crew102/reprexpy).}\n\n","description_content_type":"text/markdown","docs_url":null,"download_url":null,"downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":null,"keywords":"pathlib, cloud storage, s3, azure blob storage, google cloud storage","license":null,"license_expression":null,"license_files":["LICENSE"],"maintainer":null,"maintainer_email":null,"name":"cloudpathlib","package_url":"https://pypi.org/project/cloudpathlib/","platform":null,"project_url":"https://pypi.org/project/cloudpathlib/","project_urls":{"Bug Tracker":"https://github.com/drivendataorg/cloudpathlib/issues","Changelog":"https://cloudpathlib.drivendata.org/stable/changelog/","Documentation":"https://cloudpathlib.drivendata.org/","Repository":"https://github.com/drivendataorg/cloudpathlib"},"provides_extra":["all","azure","gs","s3"],"release_url":"https://pypi.org/project/cloudpathlib/0.21.1/","requires_dist":["typing-extensions>4; python_version < \"3.11\"","cloudpathlib[azure]; extra == \"all\"","cloudpathlib[gs]; extra == \"all\"","cloudpathlib[s3]; extra == \"all\"","azure-storage-blob>=12; extra == \"azure\"","azure-storage-file-datalake>=12; extra == \"azure\"","google-cloud-storage; extra == \"gs\"","boto3>=1.34.0; extra == \"s3\""],"requires_python":">=3.9","summary":"pathlib-style classes for cloud storage services.","version":"0.21.1","yanked":false,"yanked_reason":null}

Dependency	Optional
typing-extensions	false
azure-storage-blob	true
azure-storage-file-datalake	true
boto3	true
cloudpathlib	true
google-cloud-storage	true

Package search

cloudpathlib

project_urls

Dependencies

All package metadata

Releases