Reading '.levels' files from Azure blob storage

[edd3x]

I have xcube server running via a web-app container on MS Azure, and my datasets are saved as '.levels' files and linked to their corresponding 'zarr' files in Azure blob storage. The server loads and displays the '.zarr' files with no issues, but I always get "_The specified resource does not exis_t" error when I set config path files to read level files. I thought the issue was with the "0.link" file so I set the link path to the full URL to the zarrs on the blob storage, but I still get the errors. What can I do to fix this?

Thanks

Edward

[forman]

Hi Edward, can you please share the relevant parts of your server configuration?

[edd3x]

Hi Forman,
Thanks, I solved the earlier issue by listing both .zarr and .levels in the server config and then setting the .zarr file to hidden,
Like this:
Datasets:

• Identifier: S2_T30VVJ
  Title: "S2 NDVI Cairngorms All Vars"
  FileSystem: s3
  Endpoint: "https:/blob.core.windows.net"
  Path: "/zarr-levels/Cairngorms_T30VVJ_all_stack.zarr"
  Anonymous: true
  Hidden: true
  TimeSeriesDataset: S2_T30VVJ_Timeseries
  Style: default

• Identifier: S2_T30VVJ
  Title: "S2 NDVI Cairngorms All Vars Level"
  FileSystem: s3
  Endpoint: "https://blob.core.windows.net"
  Path: "/zarr-levels/Cairngorms_T30VVJ_all_stack.levels"
  Anonymous: true
  TimeSeriesDataset: S2_T30VVJ_Timeseries
  Style: default

[edd3x]

I, however, have another issue where I cannot list or read files using the DataStore approach for the server config. see the config below:

DataStores:

• Identifier: 'ERA_Data'
  StoreId: s3
  StoreParams:
  root: 'zarr-levels'
  max_depth: 0
  storage_options:
  anon: true
  client_kwargs:
  endpoint_url: https://blob.core.windows.net
  Datasets:
  • Identifier: S2_NDVI
    Path: '*.levels'
    Title: S2 NDVI Tiles
    Style: default0

Styles:

• Identifier: default0
  ColorMappings:
  NDVI:
  ColorBar: "BrBG"
  ValueRange: [-1, 1]
  Soil Moisture:
  ColorBar: "BrBG"
  ValueRange: [-1, 1]

[AliceBalfanz]

Hi Ed,

it seems, that xcube currently does not support azure blob. I am surprised, that it worked for you using Datasets - or was this still local? because I tried the configuration you mentioned earlier, and I was not able to access the data.

Hi Forman, Thanks, I solved the earlier issue by listing both .zarr and .levels in the server config and then setting the .zarr file to hidden, Like this: Datasets:

• Identifier: S2_T30VVJ
  Title: "S2 NDVI Cairngorms All Vars"
  FileSystem: s3
  Endpoint: "https:/blob.core.windows.net"
  Path: "/zarr-levels/Cairngorms_T30VVJ_all_stack.zarr"
  Anonymous: true
  Hidden: true
  TimeSeriesDataset: S2_T30VVJ_Timeseries
  Style: default
• Identifier: S2_T30VVJ
  Title: "S2 NDVI Cairngorms All Vars Level"
  FileSystem: s3
  Endpoint: "https://blob.core.windows.net"
  Path: "/zarr-levels/Cairngorms_T30VVJ_all_stack.levels"
  Anonymous: true
  TimeSeriesDataset: S2_T30VVJ_Timeseries
  Style: default

[forman]

It seems we just need to add a new filesystem-based data store, adlfs, should be straight forward, see https://github.com/fsspec/adlfs.

@edd3x would you volunteer for testing?

[edd3x]

@forman Yes, I would like to volunteer to test. Thanks

[edd3x]

@forman, I if the feature has been added and when I can start testing it,
Thanks

[forman]

Closed in 1.0.0.dev3