HDF5 1.10 leads to backwards incompatible netCDF4 files

[lesserwhirls]

As outlined in Unidata/netcdf-c#250 (and this email message), the latest version of HDF5 (1.10) does not write backwards compatible hdf5 files, which means any users who builds netcdf-c against the latest version will produce netCDF4 files that are backwards incompatible. This will be a huge issue for data producers or anyone wishing to share their netCDF files with others.

Not sure how netCDF4-Python would like to handle this, but we will certainly be telling users of netCDF-Java and the THREDDS Data Server (which call out to the netCDF-C library) to not use the HDF5 v1.10 until the situation becomes more clear. Note that HDF5 v.10 can read files produced by HDF5 v1.8.x, just not the other way. As far as I know, nothing is gained in the netCDF-C layer by using HDF v1.10, so sticking to HDF v1.8.x is probably best from a users prospective.

[jswhit]

Yikes! Perhaps the python module should issue a warning if it detects hdf5 1.10. At the very least, we should change the build instructions.

[Dave-Allured]

It appears there is a compatibility mode in HDF5 1.10 for writing downward compatible netcdf-4 files. Would someone please confirm my interpretation of the release documentation? Please read under "Obtaining, Testing, and Experimenting with This Software" on this page:

https://www.hdfgroup.org/HDF5/docNewFeatures/

Files will be written in the older format if these two conditions are met:

1. Set write compatibility:

• H5Pset_libver_bounds (file_id, low=H5F_LIBVER_EARLIEST_F, high=H5F_LIBVER_LATEST_F)

1. Avoid using the so-called "new features " in HDF5 1.10. In other words, do not change current HDF5 function sequences for writing Netcdf-4 files.

--Dave

[WardF]

It looks like there's a straight-forward fix available as pointed out by @Dave-Allured. I'm doing formal testing right now but will strive to have a release candidate out ASAP with the fix, with a full release to follow as soon as I can. I'm thinking Monday sometime, but the Unidata usercomm meeting is that day so we'll see. If possible I'll do it this weekend.

The end result will ultimately be that if people want to generate backwards-compatible data files (and why wouldn't they) using netcdf4+hdf5 1.10, they'll need to use netcdf-c 4.4.1 or later.

Details for the C library fix are being maintained at Unidata/netcdf-c#250.

[ryanjdillon]

Just pinging here, as I'm guessing this is a stale issue. HDF5 is now up to 1.12.