{"author":"Microsoft Corporation","author_email":"Akshat.Harit@microsoft.com","bugtrack_url":null,"classifiers":["Development Status :: 4 - Beta","License :: OSI Approved :: MIT License","Programming Language :: Python","Programming Language :: Python :: 2","Programming Language :: Python :: 2.7","Programming Language :: Python :: 3","Programming Language :: Python :: 3.10","Programming Language :: Python :: 3.11","Programming Language :: Python :: 3.3","Programming Language :: Python :: 3.4","Programming Language :: Python :: 3.5","Programming Language :: Python :: 3.6","Programming Language :: Python :: 3.7","Programming Language :: Python :: 3.8","Programming Language :: Python :: 3.9"],"description":"Microsoft Azure Data Lake Store Filesystem Library for Python\r\n=============================================================\r\n\r\n.. image:: https://travis-ci.org/Azure/azure-data-lake-store-python.svg?branch=dev\r\n :target: https://travis-ci.org/Azure/azure-data-lake-store-python\r\n.. image:: https://coveralls.io/repos/github/Azure/azure-data-lake-store-python/badge.svg?branch=master\r\n :target: https://coveralls.io/github/Azure/azure-data-lake-store-python?branch=master\r\n\r\nThis project is the Python filesystem library for Azure Data Lake Store.\r\n\r\nINSTALLATION\r\n============\r\n\r\nTo install from source instead of pip (for local testing and development):\r\n\r\n.. code-block:: bash\r\n\r\n > pip install -r dev_requirements.txt\r\n > python setup.py develop\r\n\r\nUsage: Sample Code\r\n==================\r\n\r\nTo play with the code, here is a starting point:\r\n\r\n.. code-block:: python\r\n\r\n from azure.datalake.store import core, lib, multithread\r\n token = lib.auth(tenant_id, username, password)\r\n adl = core.AzureDLFileSystem(token, store_name=store_name)\r\n\r\n # typical operations\r\n adl.ls('')\r\n adl.ls('tmp/', detail=True)\r\n adl.ls('tmp/', detail=True, invalidate_cache=True)\r\n adl.cat('littlefile')\r\n adl.head('gdelt20150827.csv')\r\n\r\n # file-like object\r\n with adl.open('gdelt20150827.csv', blocksize=2**20) as f:\r\n print(f.readline())\r\n print(f.readline())\r\n print(f.readline())\r\n # could have passed f to any function requiring a file object:\r\n # pandas.read_csv(f)\r\n\r\n with adl.open('anewfile', 'wb') as f:\r\n # data is written on flush/close, or when buffer is bigger than\r\n # blocksize\r\n f.write(b'important data')\r\n\r\n adl.du('anewfile')\r\n\r\n # recursively download the whole directory tree with 5 threads and\r\n # 16MB chunks\r\n multithread.ADLDownloader(adl, \"\", 'my_temp_dir', 5, 2**24)\r\n\r\nProgress can be tracked using a callback function in the form `track(current, total)`\r\nWhen passed, this will keep track of transferred bytes and be called on each complete chunk.\r\n\r\nHere's an example using the Azure CLI progress controller as the `progress_callback`:\r\n\r\n.. code-block:: python\r\n\r\n from cli.core.application import APPLICATION\r\n\r\n def _update_progress(current, total):\r\n hook = APPLICATION.get_progress_controller(det=True)\r\n hook.add(message='Alive', value=current, total_val=total)\r\n if total == current:\r\n hook.end()\r\n\r\n ...\r\n ADLUploader(client, destination_path, source_path, thread_count, overwrite=overwrite,\r\n chunksize=chunk_size,\r\n buffersize=buffer_size,\r\n blocksize=block_size,\r\n progress_callback=_update_progress)\r\n\r\nThis will output a progress bar to the stdout:\r\n\r\n.. code-block:: bash\r\n\r\n Alive[######################### ] 40.0881%\r\n \r\n Finished[#############################################################] 100.0000%\r\n\r\nUsage: Command Line Sample\r\n==========================\r\n\r\nTo interact with the API at a higher-level, you can use the provided\r\ncommand-line interface in \"samples/cli.py\". You will need to set\r\nthe appropriate environment variables \r\n\r\n* :code:`azure_username`\r\n\r\n* :code:`azure_password`\r\n\r\n* :code:`azure_data_lake_store_name`\r\n\r\n* :code:`azure_subscription_id`\r\n\r\n* :code:`azure_resource_group_name`\r\n\r\n* :code:`azure_service_principal`\r\n\r\n* :code:`azure_service_principal_secret`\r\n\r\nto connect to the Azure Data Lake Store. Optionally, you may need to define :code:`azure_tenant_id` or :code:`azure_data_lake_store_url_suffix`.\r\n\r\nBelow is a simple sample, with more details beyond.\r\n\r\n\r\n.. code-block:: bash\r\n\r\n python samples\\cli.py ls -l\r\n\r\nExecute the program without arguments to access documentation.\r\n\r\nTo start the CLI in interactive mode, run \"python samples/cli.py\"\r\nand then type \"help\" to see all available commands (similiar to Unix utilities):\r\n\r\n.. code-block:: bash\r\n\r\n > python samples/cli.py\r\n azure> help\r\n\r\n Documented commands (type help ):\r\n ========================================\r\n cat chmod close du get help ls mv quit rmdir touch\r\n chgrp chown df exists head info mkdir put rm tail\r\n\r\n azure>\r\n\r\n\r\nWhile still in interactive mode, you can run \"ls -l\" to list the entries in the\r\nhome directory (\"help ls\" will show the command's usage details). If you're not\r\nfamiliar with the Unix/Linux \"ls\" command, the columns represent 1) permissions,\r\n2) file owner, 3) file group, 4) file size, 5-7) file's modification time, and\r\n8) file name.\r\n\r\n.. code-block:: bash\r\n\r\n > python samples/cli.py\r\n azure> ls -l\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 02 12:44 azure1\r\n -rwxrwx--- 0123abcd 0123abcd 1048576 Jul 25 18:33 abc.csv\r\n -r-xr-xr-x 0123abcd 0123abcd 36 Jul 22 18:32 xyz.csv\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 03 13:46 tmp\r\n azure> ls -l --human-readable\r\n drwxrwx--- 0123abcd 0123abcd 0B Aug 02 12:44 azure1\r\n -rwxrwx--- 0123abcd 0123abcd 1M Jul 25 18:33 abc.csv\r\n -r-xr-xr-x 0123abcd 0123abcd 36B Jul 22 18:32 xyz.csv\r\n drwxrwx--- 0123abcd 0123abcd 0B Aug 03 13:46 tmp\r\n azure>\r\n\r\n\r\nTo download a remote file, run \"get remote-file [local-file]\". The second\r\nargument, \"local-file\", is optional. If not provided, the local file will be\r\nnamed after the remote file minus the directory path.\r\n\r\n.. code-block:: bash\r\n\r\n > python samples/cli.py\r\n azure> ls -l\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 02 12:44 azure1\r\n -rwxrwx--- 0123abcd 0123abcd 1048576 Jul 25 18:33 abc.csv\r\n -r-xr-xr-x 0123abcd 0123abcd 36 Jul 22 18:32 xyz.csv\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 03 13:46 tmp\r\n azure> get xyz.csv\r\n 2016-08-04 18:57:48,603 - ADLFS - DEBUG - Creating empty file xyz.csv\r\n 2016-08-04 18:57:48,604 - ADLFS - DEBUG - Fetch: xyz.csv, 0-36\r\n 2016-08-04 18:57:49,726 - ADLFS - DEBUG - Downloaded to xyz.csv, byte offset 0\r\n 2016-08-04 18:57:49,734 - ADLFS - DEBUG - File downloaded (xyz.csv -> xyz.csv)\r\n azure>\r\n\r\n\r\nIt is also possible to run in command-line mode, allowing any available command\r\nto be executed separately without remaining in the interpreter.\r\n\r\nFor example, listing the entries in the home directory:\r\n\r\n.. code-block:: bash\r\n\r\n > python samples/cli.py ls -l\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 02 12:44 azure1\r\n -rwxrwx--- 0123abcd 0123abcd 1048576 Jul 25 18:33 abc.csv\r\n -r-xr-xr-x 0123abcd 0123abcd 36 Jul 22 18:32 xyz.csv\r\n drwxrwx--- 0123abcd 0123abcd 0 Aug 03 13:46 tmp\r\n >\r\n\r\n\r\nAlso, downloading a remote file:\r\n\r\n.. code-block:: bash\r\n\r\n > python samples/cli.py get xyz.csv\r\n 2016-08-04 18:57:48,603 - ADLFS - DEBUG - Creating empty file xyz.csv\r\n 2016-08-04 18:57:48,604 - ADLFS - DEBUG - Fetch: xyz.csv, 0-36\r\n 2016-08-04 18:57:49,726 - ADLFS - DEBUG - Downloaded to xyz.csv, byte offset 0\r\n 2016-08-04 18:57:49,734 - ADLFS - DEBUG - File downloaded (xyz.csv -> xyz.csv)\r\n >\r\n\r\nTests\r\n=====\r\n\r\nFor detailed documentation about our test framework, please visit the \r\n`tests folder `__.\r\n\r\nNeed Help?\r\n==========\r\n\r\nBe sure to check out the Microsoft Azure `Developer Forums on Stack Overflow `__\r\nif you have trouble with the provided code. Most questions are tagged `azure and python `__.\r\n\r\n\r\nContribute Code or Provide Feedback\r\n===================================\r\n\r\nIf you would like to become an active contributor to this project please\r\nfollow the instructions provided in `Microsoft Azure Projects Contribution Guidelines `__. \r\nFurthermore, check out `GUIDANCE.md `__ \r\nfor specific information related to this project.\r\n\r\nIf you encounter any bugs with the library please file an issue in the\r\n`Issues `__\r\nsection of the project.\r\n\r\n\r\nCode of Conduct\r\n===============\r\nThis project has adopted the `Microsoft Open Source Code of Conduct `__. \r\nFor more information see the `Code of Conduct FAQ `__ or contact \r\n`opencode@microsoft.com `__ with any additional questions or comments.\r\n\r\n\r\n.. :changelog:\r\n\r\nRelease History\r\n===============\r\n\r\n0.0.53 (2023-04-11)\r\n+++++++++++++++++++\r\n* Add MSAL support. Remove ADAL support\r\n* Suppress deprecation warning when detecting pyopenssl existence.\r\n\r\n0.0.52 (2020-11-25)\r\n+++++++++++++++++++\r\n* Changed logging verbosity when closing a stream\r\n* Filter out default acl for files when using recursive acl operations\r\n\r\n0.0.51 (2020-10-15)\r\n+++++++++++++++++++\r\n* Add more logging for zero byte reads to investigate root cause.\r\n\r\n0.0.50 (2020-09-10)\r\n+++++++++++++++++++\r\n* Fix bug with retrying for ADAL exception parsing.\r\n\r\n0.0.49 (2020-08-05)\r\n+++++++++++++++++++\r\n* Fix bug with NoRetryPolicy\r\n* Remove Python 3.4,5 in travis configuration.\r\n* Fix logging for unicode\r\n\r\n0.0.48 (2019-10-18)\r\n+++++++++++++++++++\r\n* Buffer writes to prevent out of memory problems\r\n* Add Python 3.7 in travis configuration\r\n\r\n0.0.47 (2019-08-14)\r\n+++++++++++++++++++\r\n* Remove logging of bearer token\r\n* Documentation related changes(Add readme.md and correct some formatting)\r\n\r\n0.0.46 (2019-06-25)\r\n+++++++++++++++++++\r\n* Expose per request timeout. Default to 60.\r\n* Concat will not retry by default.\r\n* Bug fixes.\r\n\r\n0.0.45 (2019-05-10)\r\n+++++++++++++++++++\r\n* Update open and close ADLFile semantics\r\n* Refactor code and improve performance for opening a file\r\n\r\n0.0.44 (2019-03-05)\r\n+++++++++++++++++++\r\n* Add continuation token to LISTSTATUS api call\r\n* Update api-version to 2018-09-01\r\n\r\n0.0.43 (2019-03-01)\r\n+++++++++++++++++++\r\n* Fix bug in downloader when glob returns a single file\r\n\r\n0.0.42 (2019-02-26)\r\n+++++++++++++++++++\r\n* Update docstrings\r\n* Remove logging setlevel to DEBUG for recursive acl operations\r\n\r\n0.0.41 (2019-01-31)\r\n+++++++++++++++++++\r\n* Remove GetContentSummary api call\r\n* Move check_token() under retry block\r\n* Expose timeout parameter for AdlDownloader and AdlUploader\r\n* Raise an exception instead of silently break for zero length reads\r\n\r\n0.0.40 (2019-01-08)\r\n+++++++++++++++++++\r\n* Fix zero length read\r\n* Remove dependence on custom wheel and conform to PEP 420\r\n\r\n0.0.39 (2018-11-14)\r\n+++++++++++++++++++\r\n* Fix for Chunked Decoding exception thrown while reading response.content\r\n\r\n0.0.38 (2018-11-12)\r\n+++++++++++++++++++\r\n* Added support for recursive acl functions\r\n* Fixed bug due to missing filesessionid in get_chunk\r\n\r\n0.0.37 (2018-11-02)\r\n+++++++++++++++++++\r\n* Reverted some changes introduced in 0.0.35 that didn't work with other tokens\r\n\r\n0.0.36 (2018-10-31)\r\n+++++++++++++++++++\r\n* Fixed typo in refresh_token call\r\n\r\n0.0.35 (2018-10-29)\r\n+++++++++++++++++++\r\n* Added retry for getting tokens\r\n* Added requests>=2.20 because of CVE 2018-18074\r\n* Fixed test parameters and updated test recordings\r\n\r\n0.0.34 (2018-10-15)\r\n+++++++++++++++++++\r\n* Fixed concat issue with plus(or other symbols) in filename\r\n* Added readinto method\r\n* Changed api-version to 2018-05-01 for all.\r\n\r\n0.0.32 (2018-10-04)\r\n+++++++++++++++++++\r\n* Fixed test bug\r\n* Fixed empty folder upload bug\r\n* Fixed ADL Downloader block size bug\r\n\r\n0.0.31 (2018-09-10)\r\n+++++++++++++++++++\r\n* Added support for batched ls\r\n\r\n0.0.30 (2018-08-28)\r\n+++++++++++++++++++\r\n* Fixed .travis.yml order to add azure-nspg dependency\r\n\r\n0.0.29 (2018-08-22)\r\n+++++++++++++++++++\r\n* Fixed HISTORY.rst and Pypi documentation\r\n\r\n0.0.28 (2018-08-20)\r\n+++++++++++++++++++\r\n* Added recovery from DatalakeBadOffsetException\r\n\r\n0.0.27 (2018-08-08)\r\n+++++++++++++++++++\r\n* Fixed bug in single file check\r\n* Added Python2 exception compatibility\r\n\r\n0.0.26 (2018-08-03)\r\n+++++++++++++++++++\r\n* Fixed bug due to not importing errno\r\n* Fixed bug in os.makedirs race condition\r\n* Updated Readme with correct environment variables and fixed some links\r\n\r\n0.0.25 (2018-07-26)\r\n+++++++++++++++++++\r\n* Fixed downloading of empty directories and download of directory structure with only a single file\r\n\r\n0.0.24 (2018-07-16)\r\n+++++++++++++++++++\r\n* Retry policy implemented for all operations, default being Exponential Retry Policy\r\n\r\n0.0.23 (2018-07-11)\r\n+++++++++++++++++++\r\n* Fixed the incorrect download location in case of UNC local paths\r\n\r\n0.0.22 (2018-06-02)\r\n+++++++++++++++++++\r\n* Encoding filepaths in URI\r\n\r\n0.0.21 (2018-06-01)\r\n+++++++++++++++++++\r\n* Remove unused msrest dependency\r\n\r\n0.0.20 (2018-05-25)\r\n+++++++++++++++++++\r\n* Compatibility of the sdist with wheel 0.31.0\r\n\r\n0.0.19 (2018-03-14)\r\n-------------------\r\n* Fixed upload issue where destination filename was wrong while upload of directory with single file #208\r\n\r\n0.0.18 (2018-02-05)\r\n-------------------\r\n* Fixed read issue where whole file was cached while doing positional reads #198\r\n* Fixed readline as well for the same\r\n\r\n0.0.17 (2017-09-21)\r\n-------------------\r\n* Fixed README.rst indentation error\r\n* Changed management endpoint\r\n\r\n0.0.16 (2017-09-11)\r\n-------------------\r\n* Fixed Multi chunk transfer hangs as merging chunks fails #187\r\n* Added syncflag and leaseid in create, append calls.\r\n* Added filesessionid in create, append and open calls.\r\n\r\n0.0.15 (2017-07-26)\r\n-------------------\r\n* Enable Data Lake Store progress controller callback #174\r\n* Fix File state incorrectly marked as \"errored\" if contains chunks is \"pending\" state #182\r\n* Fix Race condition due to `transfer` future `done_callback` #177\r\n\r\n0.0.14 (2017-07-10)\r\n-------------------\r\n* Fix an issue where common prefixes in paths for upload and download were collapsed into only unique paths.\r\n\r\n0.0.13 (2017-06-28)\r\n-------------------\r\n* Add support for automatic refreshing of service principal credentials\r\n\r\n0.0.12 (2017-06-20)\r\n-------------------\r\n* Fix a regression with ls returning the top level folder if it has no contents. It now properly returns an empty array if a folder has no children.\r\n\r\n0.0.11 (2017-06-02)\r\n-------------------\r\n* Update to name incomplete file downloads with a `.inprogress` suffix. This suffix is removed when the download completes successfully.\r\n\r\n0.0.10 (2017-05-24)\r\n-------------------\r\n* Allow users to explicitly use or invalidate the internal, local cache of the filesystem that is built up from previous `ls` calls. It is now set to always call the service instead of the cache by default.\r\n* Update to properly create the wheel package during build to ensure all pip packages are available.\r\n* Update folder upload/download to properly throw early in the event that the destination files exist and overwrite was not specified. NOTE: target folder existence (or sub folder existence) does not automatically cause failure. Only leaf node existence will result in failure.\r\n* Fix a bug that caused file not found errors when attempting to get information about the root folder.\r\n\r\n0.0.9 (2017-05-09)\r\n------------------\r\n* Enforce basic SSL utilization to ensure performance due to `GitHub issue 625 `\r\n\r\n0.0.8 (2017-04-26)\r\n------------------\r\n* Fix server-side throttling retry support. This is not a guarantee that if the server is throttling the upload (or download) it will eventually succeed, but there is now a back-off retry in place to make it more likely.\r\n\r\n0.0.7 (2017-04-19)\r\n------------------\r\n* Update the build process to more efficiently handle multi-part namespaces for pip.\r\n\r\n0.0.6 (2017-03-15)\r\n------------------\r\n* Fix an issue with path caching that should drastically improve performance for download\r\n\r\n0.0.5 (2017-03-01)\r\n------------------\r\n* Fix for downloader to ensure there is access to the source path before creating destination files\r\n* Fix for credential objects to inherit from msrest.authentication for more universal authentication support\r\n* Add support for the following:\r\n\r\n * set_expiry: allows for setting expiration on files\r\n * ACL management:\r\n\r\n * set_acl: allows for the full replacement of an ACL on a file or folder\r\n * set_acl_entries: allows for \"patching\" an existing ACL on a file or folder\r\n * get_acl_status: retrieves the ACL information for a file or folder\r\n * remove_acl_entries: removes the specified entries from an ACL on a file or folder\r\n * remove_acl: removes all non-default ACL entries from a file or folder\r\n * remove_default_acl: removes all default ACL entries from a folder\r\n\r\n* Remove unsupported and unused \"TRUNCATE\" operation.\r\n* Added API-Version support with a default of the latest api version (2016-11-01)\r\n\r\n0.0.4 (2017-02-07)\r\n------------------\r\n* Fix for folder upload to properly delete folders with contents when overwrite specified.\r\n* Fix to set verbose output to False/Off by default. This removes progress tracking output by default but drastically improves performance.\r\n\r\n0.0.3 (2017-02-02)\r\n------------------\r\n* Fix to setup.py to include the HISTORY.rst file. No other changes\r\n\r\n0.0.2 (2017-01-30)\r\n------------------\r\n* Addresses an issue with lib.auth() not properly defaulting to 2FA\r\n* Fixes an issue with Overwrite for ADLUploader sometimes not being honored.\r\n* Fixes an issue with empty files not properly being uploaded and resulting in a hang in progress tracking.\r\n* Addition of a samples directory showcasing examples of how to use the client and upload and download logic.\r\n* General cleanup of documentation and comments.\r\n* This is still based on API version 2016-11-01\r\n\r\n0.0.1 (2016-11-21)\r\n------------------\r\n* Initial preview release. Based on API version 2016-11-01.\r\n* Includes initial ADLS filesystem functionality and extended upload and download support.\r\n","description_content_type":"","docs_url":null,"download_url":"","downloads":{"last_day":-1,"last_month":-1,"last_week":-1},"dynamic":null,"home_page":"https://github.com/Azure/azure-data-lake-store-python","keywords":"azure","license":"MIT License","license_expression":null,"license_files":null,"maintainer":"","maintainer_email":"","name":"azure-datalake-store","package_url":"https://pypi.org/project/azure-datalake-store/","platform":null,"project_url":"https://pypi.org/project/azure-datalake-store/","project_urls":{"Homepage":"https://github.com/Azure/azure-data-lake-store-python"},"provides_extra":null,"release_url":"https://pypi.org/project/azure-datalake-store/0.0.53/","requires_dist":["cffi","msal (<2,>=1.16.0)","requests (>=2.20.0)","azure-nspkg ; python_version<'3.0'","pathlib2 ; python_version<'3.4'","futures ; python_version<='2.7'"],"requires_python":"","summary":"Azure Data Lake Store Filesystem Client Library for Python","version":"0.0.53","yanked":false,"yanked_reason":null}

Dependency	Optional
azure-nspkg	false
cffi	false
futures	false
msal	false
pathlib2	false
requests	false

Package search

azure-datalake-store

project_urls

Dependencies

All package metadata

Releases