Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ABSStore ImportError #618

Closed
jhamman opened this issue Sep 24, 2020 · 6 comments · Fixed by #759
Closed

ABSStore ImportError #618

jhamman opened this issue Sep 24, 2020 · 6 comments · Fixed by #759

Comments

@jhamman
Copy link
Member

jhamman commented Sep 24, 2020

I'm finding that current versions of azure-storage-blob (12.5.0) are not importable when using the ABSStore. This was actually warned in this comment: #528 (comment). Is anyone using this store anymore? Presumably, fsspec's mapper interface should work

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr

store = zarr.storage.ABSStore('carbonplan-scratch',
                              prefix=f'cmip-raw-conus/',
                              account_name="carbonplan",
                              account_key=os.environ["BLOB_ACCOUNT_KEY"])

Problem description

Simply creating the ABSStore raises an ImportError:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-7-ad89fee562aa> in <module>
      4                               prefix=f'cmip-raw-conus/',
      5                               account_name="carbonplan",
----> 6                               account_key=os.environ["BLOB_ACCOUNT_KEY"])
      7 store

/srv/conda/envs/notebook/lib/python3.7/site-packages/zarr/storage.py in __init__(self, container, prefix, account_name, account_key, blob_service_kwargs)
   1980     def __init__(self, container, prefix='', account_name=None, account_key=None,
   1981                  blob_service_kwargs=None):
-> 1982         from azure.storage.blob import BlockBlobService
   1983         self.container = container
   1984         self.prefix = normalize_storage_path(prefix)

ImportError: cannot import name 'BlockBlobService' from 'azure.storage.blob' (/srv/conda/envs/notebook/lib/python3.7/site-packages/azure/storage/blob/__init__.py)

ABSStore is unusable in its current state.

Version and installation information

  • Value of zarr.__version__: '2.4.0'
  • Value of numcodecs.__version__: '0.7.2'
  • Version of Python interpreter: 3.7.8
  • Operating system (Linux/Windows/Mac): Linux
  • How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): conda-forge

conda env export :

name: notebook
channels:
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=1_gnu
  - abseil-cpp=20200225.2=he1b5a44_2
  - adal=1.2.4=pyh9f0ad1d_0
  - affine=2.3.0=py_0
  - aiofiles=0.5.0=py_0
  - aiohttp=3.6.2=py37h516909a_0
  - alembic=1.4.3=pyh9f0ad1d_0
  - altair=4.1.0=py_1
  - altair_data_server=0.4.1=py_0
  - appdirs=1.4.3=py_1
  - argon2-cffi=20.1.0=py37h8f50634_1
  - arrow-cpp=1.0.1=py37h8c03bf9_3_cpu
  - arviz=0.9.0=py_0
  - asciitree=0.3.3=py_2
  - asn1crypto=1.4.0=pyh9f0ad1d_0
  - async-timeout=3.0.1=py_1000
  - async_generator=1.10=py_0
  - attrs=20.2.0=pyh9f0ad1d_0
  - aws-sdk-cpp=1.7.164=hc831370_1
  - awscli=1.18.141=py37hc8dfbb8_0
  - azure-core=1.8.1=pyh9f0ad1d_0
  - azure-datalake-store=0.0.50=pyh9f0ad1d_0
  - azure-identity=1.4.0=pyh9f0ad1d_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=py_2
  - backports.functools_lru_cache=1.6.1=py_0
  - beautifulsoup4=4.9.1=py_1
  - binutils_impl_linux-64=2.35=h18a2f87_9
  - binutils_linux-64=2.35=heab0d09_28
  - bleach=3.2.0=pyh9f0ad1d_0
  - blinker=1.4=py_1
  - bokeh=2.2.1=py37hc8dfbb8_0
  - boost-cpp=1.72.0=h8e57a91_0
  - boto3=1.15.0=pyh9f0ad1d_0
  - botocore=1.18.0=pyh9f0ad1d_0
  - bottleneck=1.3.2=py37h03ebfcd_1
  - branca=0.3.1=py_0
  - brotli=1.0.9=he1b5a44_0
  - brotlipy=0.7.0=py37h8f50634_1000
  - bzip2=1.0.8=h516909a_3
  - c-ares=1.16.1=h516909a_3
  - ca-certificates=2020.6.20=hecda079_0
  - cachetools=4.1.1=py_0
  - cachey=0.2.1=pyh9f0ad1d_0
  - cairo=1.16.0=hcf35c78_1003
  - cartopy=0.17.0=py37he1be148_1005
  - certifi=2020.6.20=py37hc8dfbb8_0
  - certipy=0.1.3=py_0
  - cffi=1.14.3=py37h2b28604_0
  - cfgv=3.2.0=py_0
  - cfitsio=3.470=hb60a0a2_2
  - cftime=1.2.1=py37h03ebfcd_0
  - chardet=3.0.4=py37hc8dfbb8_1007
  - click=7.1.2=pyh9f0ad1d_0
  - click-plugins=1.1.1=py_0
  - cligj=0.5.0=py_0
  - cloudpickle=1.6.0=py_0
  - colorama=0.4.3=py_0
  - colorcet=2.0.1=py_0
  - configobj=5.0.6=py_0
  - croniter=0.3.30=py_0
  - cryptography=3.1=py37hb09aad4_0
  - curl=7.68.0=hf8cf82a_0
  - cycler=0.10.0=py_2
  - cython=0.28.5=py37hf484d3e_1000
  - cytoolz=0.10.1=py37h516909a_0
  - dask=2.25.0=py_0
  - dask-core=2.25.0=py_0
  - dask-gateway=0.8.0=py37hc8dfbb8_0
  - dask-kubernetes=0.10.1=py_0
  - dask-labextension=3.0.0=py_0
  - datashader=0.11.1=pyh9f0ad1d_0
  - datashape=0.5.4=py_1
  - dbus=1.13.6=he372182_0
  - decorator=4.4.2=py_0
  - defusedxml=0.6.0=py_0
  - distlib=0.3.1=pyh9f0ad1d_0
  - distributed=2.25.0=py37hc8dfbb8_0
  - docker-py=4.3.1=py37hc8dfbb8_0
  - docker-pycreds=0.4.0=py_0
  - docopt=0.6.2=py_1
  - docutils=0.15.2=py37_0
  - earthengine-api=0.1.235=pyh9f0ad1d_0
  - editdistance=0.5.3=py37h3340039_1
  - entrypoints=0.3=py37hc8dfbb8_1001
  - esmf=8.0.0=hcda5390_0
  - esmpy=8.0.0=py37h5ca1d4c_0
  - expat=2.2.9=he1b5a44_2
  - fastapi=0.61.1=py_0
  - fasteners=0.14.1=py_3
  - fastparquet=0.4.1=py37h03ebfcd_0
  - fastprogress=1.0.0=py_0
  - filelock=3.0.12=pyh9f0ad1d_0
  - fiona=1.8.9.post2=py37hdff7cfa_0
  - fontconfig=2.13.1=h86ecdb6_1001
  - freetype=2.10.2=he06d7ca_0
  - freexl=1.0.5=h516909a_1002
  - fribidi=1.0.10=h516909a_0
  - future=0.18.2=py37hc8dfbb8_1
  - gcc_impl_linux-64=7.5.0=hda68d29_13
  - gcc_linux-64=7.5.0=hf34d7eb_28
  - gcsfs=0.7.1=py_0
  - gdal=2.4.2=py37h5f563d9_4
  - geopandas=0.8.1=py_0
  - geos=3.7.2=he1b5a44_2
  - geotiff=1.5.1=h560c3f3_2
  - geoviews=1.8.1=py_0
  - geoviews-core=1.8.1=py_0
  - gettext=0.19.8.1=hc5be6a0_1002
  - gflags=2.2.2=he1b5a44_1004
  - giflib=5.1.7=h516909a_1
  - glib=2.66.0=h0dae87d_0
  - glog=0.4.0=h49b9bf7_3
  - google-api-core=1.22.2=py37hc8dfbb8_0
  - google-api-python-client=1.12.1=pyh9f0ad1d_0
  - google-auth=1.21.2=py_0
  - google-auth-httplib2=0.0.3=py_3
  - google-auth-oauthlib=0.4.1=py_2
  - google-cloud-core=1.4.1=pyh9f0ad1d_0
  - google-cloud-storage=1.31.0=pyh9f0ad1d_0
  - google-crc32c=1.0.0=py37h193935f_0
  - google-resumable-media=1.0.0=pyh9f0ad1d_0
  - googleapis-common-protos=1.51.0=py37hc8dfbb8_2
  - graphene=1.4=py_0
  - graphite2=1.3.13=he1b5a44_1001
  - graphql-core=3.1.2=py_0
  - graphql-relay=3.0.0=py_0
  - graphviz=2.42.3=h0511662_0
  - grpc-cpp=1.30.2=heedbac9_0
  - grpcio=1.31.0=py37hb0870dc_0
  - gunicorn=20.0.4=py37hc8dfbb8_1
  - gxx_impl_linux-64=7.5.0=h64c220c_13
  - gxx_linux-64=7.5.0=ha781d05_28
  - h11=0.9.0=py_0
  - h5netcdf=0.8.1=py_0
  - h5py=2.10.0=nompi_py37h513d04c_102
  - harfbuzz=2.4.0=h9f30f68_3
  - hdf4=4.2.13=hf30be14_1003
  - hdf5=1.10.5=nompi_h3c11f04_1104
  - heapdict=1.0.1=py_0
  - holoviews=1.13.3=pyh9f0ad1d_0
  - httplib2=0.18.1=pyh9f0ad1d_0
  - httptools=0.1.1=py37h516909a_0
  - hvplot=0.6.0=pyh9f0ad1d_0
  - icu=64.2=he1b5a44_1
  - identify=1.5.3=pyh9f0ad1d_0
  - idna=2.10=pyh9f0ad1d_0
  - imagecodecs-lite=2019.12.3=py37h03ebfcd_1
  - imageio=2.9.0=py_0
  - importlib-metadata=1.7.0=py37hc8dfbb8_0
  - importlib_metadata=1.7.0=0
  - iniconfig=1.0.1=pyh9f0ad1d_0
  - intake=0.6.0=py_0
  - intake-esm=2020.8.15=py_0
  - intake-geopandas=0.2.3=py_0
  - intake-parquet=0.2.3=py_0
  - intake-stac=0.2.3=py_0
  - intake-xarray=0.3.1=py_0
  - ipykernel=5.3.4=py37h43977f1_0
  - ipyleaflet=0.13.3=pyh9f0ad1d_0
  - ipython=7.18.1=py37hc6149b9_0
  - ipython_genutils=0.2.0=py_1
  - ipytree=0.1.8=py_0
  - ipywidgets=7.5.1=pyh9f0ad1d_1
  - isodate=0.6.0=py_1
  - itsdangerous=1.1.0=py_0
  - jedi=0.17.2=py37hc8dfbb8_0
  - jeepney=0.4.3=py_0
  - jinja2=2.11.2=pyh9f0ad1d_0
  - jmespath=0.10.0=pyh9f0ad1d_0
  - joblib=0.16.0=py_0
  - jpeg=9d=h516909a_0
  - json-c=0.13.1=hbfbb72e_1002
  - json5=0.9.4=pyh9f0ad1d_0
  - jsonschema=3.2.0=py37hc8dfbb8_1
  - jupyter-server-proxy=1.5.0=py_0
  - jupyter_client=6.1.7=py_0
  - jupyter_core=4.6.3=py37hc8dfbb8_1
  - jupyter_telemetry=0.0.5=py_0
  - jupyterhub-base=1.1.0=py37hc8dfbb8_4
  - jupyterhub-singleuser=1.1.0=py37hc8dfbb8_4
  - jupyterlab=2.2.8=py_0
  - jupyterlab_pygments=0.1.1=pyh9f0ad1d_0
  - jupyterlab_server=1.2.0=py_0
  - kealib=1.4.13=hec59c27_0
  - kernel-headers_linux-64=2.6.32=h77966d4_13
  - keyring=21.4.0=py37hc8dfbb8_0
  - kiwisolver=1.2.0=py37h99015e2_0
  - krb5=1.16.4=h2fd8d38_0
  - kubernetes_asyncio=11.3.0=pyh9f0ad1d_0
  - lcms2=2.11=hbd6801e_0
  - ld_impl_linux-64=2.35=h769bd43_9
  - libblas=3.8.0=17_openblas
  - libcblas=3.8.0=17_openblas
  - libcrc32c=1.1.1=he1b5a44_2
  - libcurl=7.68.0=hda55be3_0
  - libdap4=3.20.4=hd3bb157_0
  - libedit=3.1.20191231=he28a2e2_2
  - libevent=2.1.10=hcdb4288_2
  - libffi=3.2.1=he1b5a44_1007
  - libgcc-ng=9.3.0=h24d8f2e_16
  - libgdal=2.4.2=hbef8c27_4
  - libgfortran-ng=7.5.0=hdf63c60_16
  - libgomp=9.3.0=h24d8f2e_16
  - libiconv=1.16=h516909a_0
  - libkml=1.3.0=hd79254b_1012
  - liblapack=3.8.0=17_openblas
  - libllvm10=10.0.1=he513fc3_3
  - libnetcdf=4.6.2=h303dfb8_1003
  - libopenblas=0.3.10=pthreads_hb3c22a3_4
  - libpng=1.6.37=hed695b0_2
  - libpq=11.5=hd9ab2ff_2
  - libprotobuf=3.12.4=h8b12597_0
  - libsodium=1.0.18=h516909a_0
  - libspatialindex=1.9.3=he1b5a44_3
  - libspatialite=4.3.0a=he1bb1e1_1029
  - libssh2=1.9.0=hab1572f_5
  - libstdcxx-ng=9.3.0=hdf63c60_16
  - libtiff=4.1.0=hc7e4089_6
  - libtool=2.4.6=h516909a_1005
  - libutf8proc=2.5.0=h516909a_2
  - libuuid=2.32.1=h14c3975_1000
  - libuv=1.34.0=h516909a_0
  - libwebp-base=1.1.0=h516909a_3
  - libxcb=1.13=h14c3975_1002
  - libxml2=2.9.10=hee79883_0
  - libxslt=1.1.33=h31b3aaa_0
  - llvmlite=0.34.0=py37h5202443_1
  - locket=0.2.0=py_2
  - lxml=4.5.2=py37he3881c9_0
  - lz4-c=1.9.2=he1b5a44_3
  - mako=1.1.3=pyh9f0ad1d_0
  - markdown=3.2.2=py_0
  - markupsafe=1.1.1=py37h8f50634_1
  - marshmallow=3.8.0=py_0
  - marshmallow-oneofschema=2.0.1=py_0
  - matplotlib=3.3.2=0
  - matplotlib-base=3.3.2=py37hd478181_0
  - mechanicalsoup=0.12.0=py_0
  - mercantile=1.1.6=pyh9f0ad1d_0
  - mistune=0.8.4=py37h8f50634_1001
  - monotonic=1.5=py_0
  - more-itertools=8.5.0=py_0
  - mpi=1.0=mpich
  - mpich=3.3.2=hc856adb_0
  - mpl-probscale=0.2.4=pyh9f0ad1d_0
  - msal=1.5.0=pyh9f0ad1d_0
  - msal_extensions=0.2.2=py37hc8dfbb8_0
  - msgpack-python=1.0.0=py37h99015e2_1
  - msrest=0.6.19=pyh9f0ad1d_0
  - msrestazure=0.6.2=py_0
  - multidict=4.7.5=py37h8f50634_1
  - multipledispatch=0.6.0=py_0
  - munch=2.5.0=py_0
  - mypy_extensions=0.4.3=py37hc8dfbb8_1
  - nb_conda_kernels=2.2.4=py37hc8dfbb8_0
  - nbclient=0.5.0=py_0
  - nbconvert=6.0.3=py37hc8dfbb8_0
  - nbformat=5.0.7=py_0
  - nbgitpuller=0.9.0=py_0
  - nc-time-axis=1.2.0=py_1
  - ncurses=6.2=he1b5a44_1
  - nest-asyncio=1.4.0=py_0
  - netcdf-fortran=4.4.5=h0789656_1004
  - netcdf4=1.5.1.2=py37h73a1b54_1
  - networkx=2.5=py_0
  - nodeenv=1.5.0=pyh9f0ad1d_0
  - nodejs=13.13.0=hf5d1a2b_0
  - notebook=6.1.4=py37hc8dfbb8_0
  - numba=0.51.2=py37h9fdb41a_0
  - numcodecs=0.7.2=py37h3340039_0
  - numpy=1.19.1=py37h7ea13bd_2
  - oauthlib=3.0.1=py_0
  - olefile=0.46=py_0
  - openjpeg=2.3.1=h981e76c_3
  - openssl=1.1.1g=h516909a_1
  - owslib=0.20.0=py_0
  - packaging=20.4=pyh9f0ad1d_0
  - pamela=1.0.0=py_0
  - pandas=1.1.2=py37h3340039_0
  - pandoc=2.10.1=h516909a_0
  - pandocfilters=1.4.2=py_1
  - panel=0.9.7=py_0
  - pangeo-dask=2020.08.30=1
  - pangeo-notebook=2020.08.31=0
  - pango=1.42.4=h7062337_4
  - param=1.9.3=py_0
  - parquet-cpp=1.5.1=2
  - parso=0.7.1=pyh9f0ad1d_0
  - partd=1.1.0=py_0
  - patsy=0.5.1=py_0
  - pcre=8.44=he1b5a44_0
  - pendulum=2.1.2=py37hc8dfbb8_0
  - pexpect=4.8.0=py37hc8dfbb8_1
  - pickleshare=0.7.5=py37hc8dfbb8_1001
  - pillow=7.2.0=py37h718be6c_1
  - pip=20.2.3=py_0
  - pixman=0.38.0=h516909a_1003
  - pluggy=0.13.1=py37hc8dfbb8_2
  - poppler=0.67.0=h14e79db_8
  - poppler-data=0.4.9=1
  - portalocker=1.7.0=py37hc8dfbb8_0
  - portpicker=1.3.1=py37_0
  - postgresql=11.5=hc63931a_2
  - pre-commit=2.7.1=py37hc8dfbb8_0
  - pre_commit=2.7.1=0
  - prefect=0.13.7=py_0
  - proj4=6.1.0=he751ad9_2
  - prometheus_client=0.8.0=pyh9f0ad1d_0
  - promise=2.3=py37hc8dfbb8_1
  - prompt-toolkit=3.0.7=py_0
  - protobuf=3.12.4=py37h3340039_0
  - psutil=5.7.2=py37h8f50634_0
  - pthread-stubs=0.4=h14c3975_1001
  - ptyprocess=0.6.0=py_1001
  - py=1.9.0=pyh9f0ad1d_0
  - pyarrow=1.0.1=py37hcae2be3_3_cpu
  - pyasn1=0.4.8=py_0
  - pyasn1-modules=0.2.7=py_0
  - pycparser=2.20=pyh9f0ad1d_2
  - pyct=0.4.6=py_0
  - pyct-core=0.4.6=py_0
  - pycurl=7.43.0.5=py37hce7685b_2
  - pydantic=1.6.1=py37h8f50634_0
  - pydap=3.2.2=py37_1000
  - pyepsg=0.4.0=py_0
  - pygments=2.7.1=py_0
  - pyjwt=1.7.1=py_0
  - pykdtree=1.3.1=py37h03ebfcd_1003
  - pyopenssl=19.1.0=py_1
  - pyparsing=2.4.7=pyh9f0ad1d_0
  - pyproj=2.2.1=py37hc44880f_0
  - pyresample=1.16.0=py37h0da4684_0
  - pyrsistent=0.17.3=py37h8f50634_0
  - pyshp=2.1.2=pyh9f0ad1d_0
  - pysocks=1.7.1=py37hc8dfbb8_1
  - pystac=0.5.2=pyh9f0ad1d_0
  - pystan=2.19.1.1=py37h9fdb41a_2
  - pytest=6.0.2=py37hc8dfbb8_0
  - python=3.7.8=h6f2ec95_1_cpython
  - python-box=4.2.3=py_0
  - python-dateutil=2.7.5=py_0
  - python-editor=1.0.4=py_0
  - python-graphviz=0.14.1=pyh9f0ad1d_0
  - python-json-logger=0.1.11=py_0
  - python-kubernetes=11.0.0=py37hc8dfbb8_0
  - python-multipart=0.0.5=py_0
  - python-slugify=4.0.1=pyh9f0ad1d_0
  - python-wget=3.2=py_0
  - python_abi=3.7=1_cp37m
  - pytz=2020.1=pyh9f0ad1d_0
  - pytzdata=2020.1=pyh9f0ad1d_0
  - pyviz_comms=0.7.6=pyh9f0ad1d_0
  - pywavelets=1.1.1=py37h161383b_2
  - pyyaml=5.3.1=py37h8f50634_0
  - pyzmq=19.0.2=py37hac76be4_0
  - rasterio=1.0.25=py37hb3f55d8_1
  - re2=2020.08.01=he1b5a44_1
  - readline=8.0=he28a2e2_2
  - requests=2.24.0=pyh9f0ad1d_0
  - requests-oauthlib=1.3.0=pyh9f0ad1d_0
  - richdem=0.3.4=py37hb3f55d8_0
  - rio-cogeo=1.1.6=py_0
  - rioxarray=0.0.31=py_0
  - rsa=4.4.1=pyh9f0ad1d_0
  - rtree=0.9.4=py37h8526d28_1
  - ruamel.yaml=0.16.12=py37h8f50634_0
  - ruamel.yaml.clib=0.2.2=py37h8f50634_0
  - s3fs=0.4.2=py_0
  - s3transfer=0.3.3=py37hc8dfbb8_1
  - sat-stac=0.4.0=pyh9f0ad1d_0
  - scikit-image=0.17.2=py37h9fdb41a_2
  - scikit-learn=0.23.2=py37h6785257_0
  - scipy=1.5.2=py37hb14ef9d_0
  - seaborn=0.11.0=0
  - seaborn-base=0.11.0=py_0
  - secretstorage=3.1.2=py37hc8dfbb8_1
  - send2trash=1.5.0=py_0
  - setuptools=49.6.0=py37hc8dfbb8_1
  - shapely=1.6.4=py37hec07ddf_1006
  - simpervisor=0.3=py_1
  - simplejson=3.17.2=py37h8f50634_0
  - six=1.15.0=pyh9f0ad1d_0
  - snappy=1.1.8=he1b5a44_3
  - snuggs=1.4.7=py_0
  - sortedcontainers=2.2.2=pyh9f0ad1d_0
  - soupsieve=2.0.1=py_1
  - sqlalchemy=1.3.19=py37h8f50634_0
  - sqlite=3.33.0=h4cf870e_0
  - starlette=0.13.6=py_0
  - starlette-base=0.13.6=py_0
  - statsmodels=0.12.0=py37h8f50634_0
  - supermercado=0.1.1=pyh9f0ad1d_0
  - sysroot_linux-64=2.12=h77966d4_13
  - tabulate=0.8.7=pyh9f0ad1d_0
  - tblib=1.6.0=py_0
  - terminado=0.8.3=py37hc8dfbb8_1
  - testpath=0.4.4=py_0
  - text-unidecode=1.3=py_0
  - threadpoolctl=2.1.0=pyh5ca1d4c_0
  - thrift=0.11.0=py37he1b5a44_1001
  - thrift-cpp=0.13.0=hbe8ec66_4
  - tifffile=2020.6.3=py_0
  - tippecanoe=1.35.0=h792bddf_1
  - tk=8.6.10=hed695b0_0
  - toml=0.10.1=pyh9f0ad1d_0
  - toolz=0.10.0=py_0
  - tornado=6.0.4=py37h8f50634_1
  - tqdm=4.49.0=pyh9f0ad1d_0
  - traitlets=4.3.3=py37hc8dfbb8_1
  - traittypes=0.2.1=py_1
  - typing_extensions=3.7.4.2=py_0
  - tzcode=2020a=h516909a_0
  - ujson=3.2.0=py37h3340039_0
  - unidecode=1.1.1=py_0
  - uritemplate=3.0.1=py_0
  - urllib3=1.25.10=py_0
  - uvicorn=0.11.3=py37hc8dfbb8_1
  - uvloop=0.14.0=py37h8f50634_1
  - virtualenv=20.0.20=py37hc8dfbb8_1
  - wcwidth=0.2.5=pyh9f0ad1d_1
  - webencodings=0.5.1=py_1
  - webob=1.8.6=py_0
  - websocket-client=0.57.0=py37hc8dfbb8_2
  - websockets=8.1=py37h8f50634_1
  - wheel=0.35.1=pyh9f0ad1d_0
  - widgetsnbextension=3.5.1=py37hc8dfbb8_1
  - wsproto=0.13.0=py_0
  - xarray=0.16.0=py_0
  - xerces-c=3.2.2=h8412b87_1004
  - xesmf=0.3.0=py_0
  - xlrd=1.2.0=pyh9f0ad1d_1
  - xorg-kbproto=1.0.7=h14c3975_1002
  - xorg-libice=1.0.10=h516909a_0
  - xorg-libsm=1.2.3=h84519dc_1000
  - xorg-libx11=1.6.12=h516909a_0
  - xorg-libxau=1.0.9=h14c3975_0
  - xorg-libxdmcp=1.1.3=h516909a_0
  - xorg-libxext=1.3.4=h516909a_0
  - xorg-libxpm=3.5.13=h516909a_0
  - xorg-libxrender=0.9.10=h516909a_1002
  - xorg-libxt=1.1.5=h516909a_1003
  - xorg-renderproto=0.11.1=h14c3975_1002
  - xorg-xextproto=7.3.0=h14c3975_1002
  - xorg-xproto=7.0.31=h14c3975_1007
  - xpublish=0.1.0=py_0
  - xz=5.2.5=h516909a_1
  - yaml=0.2.5=h516909a_0
  - yarl=1.3.0=py37h516909a_1000
  - yarn=1.22.5=0
  - zarr=2.4.0=py_0
  - zeromq=4.3.2=he1b5a44_3
  - zict=2.0.0=py_0
  - zipp=3.1.0=py_0
  - zlib=1.2.11=h516909a_1009
  - zstd=1.4.5=h6597ccf_2
  - pip:
    - adlfs==0.5.3+12.g0cbfe53
    - azure-common==1.1.25
    - azure-storage-blob==12.5.0
    - azure-storage-common==2.1.0
    - fsspec==0.8.3+1.g8bec5de
    - numbagg==0.1
    - rechunker==0.2.0
    - rio-tiler-mvt==0.0.1.dev1
    - showit==1.1.4
    - urlpath==1.1.7
    - vtzero==0.0.1b1
prefix: /srv/conda/envs/notebook

cc @tjcrone

@rabernat
Copy link
Contributor

Should we consider deprecating ABSStore in favor of https://github.com/dask/adlfs, which has matured a lot recently?

That would bring our treatment of ABS more in line with the other cloud object stores.

@tjcrone
Copy link
Member

tjcrone commented Sep 24, 2020

@jhamman, the BlockBlobService is no longer available in recent versions of the Azure storage module. BlobServiceClient replaces this in azure.storage.blob, and transitioning to this new approach in ABSStore should be relatively straightforward but it's not clear how easy it would be to maintain backward compatibility. The fsspec mapper interface does work and it may be the best thing to rely on going forward.

@tjcrone
Copy link
Member

tjcrone commented Sep 24, 2020

@rabernat, if adlfs, despite the name, is able to access Azure Blob storage, then this might also be a viable way forward. Data Lake is not Blob, but there is some indication in the adlfs documentation that it can access Blob.

@rabernat
Copy link
Contributor

fsspec-compatible Azure Datake and Azure Blob Storage access

@jhamman
Copy link
Member Author

jhamman commented Sep 24, 2020

FWIW, I haven't been able to get adlfs to work yet (I'll open an issue there in a bit).

from adlfs import AzureBlobFileSystem


path = 'carbonplan-data/raw/terraclimate/4000m/raster.zarr'
fs = AzureBlobFileSystem(
    account_name="carbonplan",
    account_key=os.environ["BLOB_ACCOUNT_KEY"]
)
mapper = fs.get_mapper(path)
xr.open_zarr(mapper, consolidated=True)  # hangs!!!

While I can get a mapper object, accessing keys from the mapper ends up hanging indefinitely.

Also, since I know you're wondering, read-only access over http is working just fine for these stores.

@tjcrone
Copy link
Member

tjcrone commented Sep 24, 2020

@jhamman as a stop-gap, if you want to use ABSStore you can install a pre-12 (2.1) version of azure-storage-blob:

pip install azure-storage-blob==2.1.0

Updating ABSStore to work with v12 versions of the azure-storage SDK would likely be just a few lines of code. Maintaining backward compatibility would be more difficult but almost certainly doable. I would be happy to continue supporting ABSStore if other Zarr devs support this approach, but I am also happy for us to move in a different direction if that's what we would like to do. This would be a great time and maybe even a great place to have this discussion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants