Skip to content

0.13.0
Compare
Choose a tag to compare
@forman forman released this 08 Feb 13:55
v0.13.0
6a319a3

Changes in 0.13.0

Enhancements

  • xcube Server has been rewritten almost from scratch.

    • Introduced a new endpoint ${server_url}/s3 that emulates
      and AWS S3 object storage for the published datasets. (#717)
      The bucket name can be either:

      • s3://datasets - publishes all datasets in Zarr format.
      • s3://pyramids - publishes all datasets in a multi-level levels
        format (multi-resolution N-D images)
        that comprises level datasets in Zarr format.

      Datasets published through the S3 API are slightly
      renamed for clarity. For bucket s3://pyramids:

      • if a dataset identifier has suffix .levels, the identifier remains;
      • if a dataset identifier has suffix .zarr, it will be replaced by
        .levels only if such a dataset doesn't exist;
      • otherwise, the suffix .levels is appended to the identifier.
        For bucket s3://datasets the opposite is true:
      • if a dataset identifier has suffix .zarr, the identifier remains;
      • if a dataset identifier has suffix .levels, it will be replaced by
        .zarr only if such a dataset doesn't exist;
      • otherwise, the suffix .zarr is appended to the identifier.

      With the new S3 endpoints in place, xcube Server instances can be used
      as xcube data stores as follows:

      store = new_data_store(
    "s3", 
    root="datasets",   # bucket "datasets", use also "pyramids"
    max_depth=2,       # optional, but we may have nested datasets
    storage_options=dict(
        anon=True,
        client_kwargs=dict(
            endpoint_url='http://localhost:8080/s3' 
        )
    )
)

    • The limited s3bucket endpoints are no longer available and are
      replaced by s3 endpoints.

    • Added new endpoint /viewer that serves a self-contained,
      packaged build of
      xcube Viewer.
      The packaged viewer can be overridden by environment variable
      XCUBE_VIEWER_PATH that must point to a directory with a
      build of a compatible viewer.

    • The --show option of xcube serve
      has been renamed to --open-viewer.
      It now uses the self-contained, packaged build of
      xcube Viewer. (#750)

    • The --show option of xcube serve
      now outputs various aspects of the server configuration.

    • Added experimental endpoint /volumes.
      It is used by xcube Viewer to render 3-D volumes.

  • xcube Server is now more tolerant with respect to datasets it can not
    open without errors. Implementation detail: It no longer fails if
    opening datasets raises any exception other than DatasetIsNotACubeError.
    (#789)

  • xcube Server's colormap management has been improved in several ways:

    • Colormaps are no longer managed globally. E.g., on server configuration
      change, new custom colormaps are reloaded from files.
    • Colormaps are loaded dynamically from underlying
      matplotlib and cmocean registries, and custom SNAP color palette files.
      That means, latest matplotlib colormaps are now always available. (#687)
    • Colormaps can now be reversed (name suffix "_r"),
      can have alpha blending (name suffix "_alpha"),
      or both (name suffix "_r_alpha").
    • Loading of custom colormaps from SNAP *.cpd has been rewritten.
      Now also the isLogScaled property of the colormap is recognized. (#661)
    • The module xcube.util.cmaps has been redesigned and now offers
      three new classes for colormap management:
      • Colormap - a colormap
      • ColormapCategory - represents a colormap category
      • ColormapRegistry - manages colormaps and their categories

  • The xcube filesystem data stores such as "file", "s3", "memory"
    can now filter the data identifiers reported by get_data_ids(). (#585)
    For this purpose, the data stores now accept two new optional keywords
    which both can take the form of a wildcard pattern or a sequence
    of wildcard patterns:

    1. excludes: if given and if any pattern matches the identifier,
      the identifier is not reported.
    2. includes: if not given or if any pattern matches the identifier,
      the identifier is reported.

  • Added convenience method DataStore.list_data_ids() that works
    like get_data_ids(), but returns a list instead of an iterator. (#776)

  • Added Notebook
    xcube-viewer-in-jl.ipynb
    that explains how xcube Viewer can now be utilised in JupyterLab
    using the new (still experimental) xcube JupyterLab extension
    xcube-jl-ext.

  • Replaced usages of deprecated numpy dtype numpy.bool
    by Python type bool.

Fixes

  • xcube CLI tools no longer emit warnings when trying to import
    installed packages named xcube_* as xcube plugins.

  • The xcube.util.timeindex module can now handle 0-dimensional
    ndarrays as indexers. This effectively avoids the warning
    Can't determine indexer timezone; leaving it unmodified.
    which was emitted in such cases.

  • xcube serve will now also accept datasets with coordinate names
    longitude and latitude, even if the attribute long_name isn't set.
    (#763)

  • Function xcube.core.resampling.affine.affine_transform_dataset()
    now assumes that geographic coordinate systems are equal by default and
    hence a resampling based on an affine transformation can be performed.

  • Fixed a problem with xcube server's WMTS implementation.
    For multi-level resolution datasets with very coarse low resolution levels,
    the tile matrix sets WorldCRS84Quad and WorldWebMercatorQuad have
    reported a negative minimum z-level.

  • Implementation of function xcube.core.geom.rasterize_features()
    has been changed to account for consistent use of a target variable's
    fill_value and dtype for a given feature.
    In-memory (decoded) variables now always use dtype float64 and use
    np.nan to represent missing values. Persisted (encoded) variable data
    will make use of the target fill_value and dtype. (#778)

  • Relative local filesystem paths to datasets are now correctly resolved
    against the base directory of the xcube Server's configuration, i.e.
    configuration parameter base_dir. (#758)

  • Fixed problem with xcube gen raising FileNotFoundError
    with Zarr >= 2.13.

  • Provided backward compatibility with Python 3.8. (#760)

Other

  • The CLI tool xcube edit has been deprecated in favour of the
    xcube patch. (#748)

  • Deprecated CLI xcube tile has been removed.

  • Deprecated modules, classes, methods, and functions
    have finally been removed:

    • xcube.core.geom.get_geometry_mask()
    • xcube.core.mldataset.FileStorageMultiLevelDataset
    • xcube.core.mldataset.open_ml_dataset()
    • xcube.core.mldataset.open_ml_dataset_from_local_fs()
    • xcube.core.mldataset.open_ml_dataset_from_object_storage()
    • xcube.core.subsampling.get_dataset_subsampling_slices()
    • xcube.core.tiledimage
    • xcube.core.tilegrid

  • The following classes, methods, and functions have been deprecated:

    • xcube.core.xarray.DatasetAccessor.levels()
    • xcube.util.cmaps.get_cmap()
    • xcube.util.cmaps.get_cmaps()

  • A new function compute_tiles() has been
    refactored out from function xcube.core.tile.compute_rgba_tile().

  • Added method get_level_for_resolution(xy_res) to
    abstract base class xcube.core.mldataset.MultiLevelDataset.

  • Removed outdated example resources from examples/serve/demo.

  • Account for different spatial resolutions in x and y in
    xcube.core.geom.get_dataset_bounds().

  • Make code robust against 0-size coordinates in
    xcube.core.update._update_dataset_attrs().

  • xcube Server has been enhanced to load multi-module Python code
    for dynamic cubes both from both directories and zip archives.
    For example, the following dataset definition computes a dynamic
    cube from dataset "local" using function "compute_dataset" in
    Python module "resample_in_time.py":

      Path: resample_in_time.py
  Function: compute_dataset
  InputDatasets: ["local"]

    Users can now pack "resample_in_time.py" among any other modules and
    packages into a zip archive. Note that the original module name
    is now a prefix to the function name:

      Path: modules.zip
  Function: resample_in_time:compute_dataset
  InputDatasets: ["local"]

    Implementation note: this has been achieved by using
    xcube.core.byoa.CodeConfig in
    xcube.core.mldataset.ComputedMultiLevelDataset.

  • Instead of the Function keyword it is now
    possible to use the Class keyword.
    While Function references a function that receives one or
    more datasets (type xarray.Dataset) and returns a new one,
    Class references a callable that receives one or
    more multi-level datasets and returns a new one.
    The callable is either a class derived from
    or a function that returns an instance of
    xcube.core.mldataset.MultiLevelDataset.

  • Module xcube.core.mldataset has been refactored into
    a sub-package for clarity and maintainability.

  • Removed deprecated example examples/tile.

New Contributors

Full Changelog: v0.12.1...v0.13.0

Contributors

thomasstorm and ymoisan
Assets 2
Loading