Slow when writing to Zarr

[aolt]

I am trying to convert a multiple files grib files Xarray dataset into Zarr. Reading these files is relatively quick, but writing to Zarr is going very slow. What I am doing:

import xarray as xr
chunks = {"time": 24, "latitude": 103, "longitude": 180}
ds = xr.open_mfdataset("{}/*/*/*_{}_*.grb".format(source, var), chunks=chunks, engine='cfgrib', backend_kwargs = {'indexpath': '/mnt/test/era5/grib_index/idx'}, concat_dim='time')
ds.to_zarr("mypath")

It writes about 1MB per 10 seconds and it is using 100% CPU. There is a lot performance capacity on the disk, so I assume it should be about how it reads grib files.

>>> xarray.__version__
'0.11.0'

python -V
Python 3.7.0

pip list |grep cfgrib
cfgrib           0.9.4.1  

Each file is about 1.5GB and I have about 216 files to write as Zarr

[alexamici]

@aolt first guess the 'indexpath': '/mnt/test/era5/grib_index/idx' attempts to write all indices to the same file so you don't have index persistence and this typically kills performance.

You can use '{path}' in indexpath to differentiate files in the open_mfdataset call. Something like:

>>> ds = xr.open_mfdataset("{}/*/*/*_{}_*.grb".format(source, var), chunks=chunks, engine='cfgrib', backend_kwargs = {'indexpath': '/tmp/{path}.idx'}, concat_dim='time')
>>> ds.to_zarr("mypath")

Also chunking on latitude / longitude is not very useful with GRIB files in general, avoid it if you can.

[aolt]

Thanks for the trick with 'indexpath', It didn't help though :(
Avoiding chunking on lat/lon helps indeed, but still I am on 100% CPU load and about 20MB/s write where I have plenty performance left on the disk.

[alexamici]

@aolt improving performance, especially when using dask, is in our todo list, but not very high as long as we work on stabilising the API, sorry.

[alexamici]

I opened #33 for general dask performance issue, I leave this issue open for now, but my guess is that there is nothing specific to Zarr.

[aolt]

thanks @alexamici, I think this is specific to cfgrib. I get much higher performance with reading NetCDF with xarray/scipy and writing into Zarr.

[alexamici]

I close the issue as it looks like a generic disk performance issue.