Dimension order sorcery

[nicholas512]

Change how GlobSim reads netcdf files so you can easily change the dimension order of the source file.

(lat,lon,time) is significantly faster than (time,lat,lon)--the default.

ideally, it will be possible to flexibly switch between default and "optimized" dimension behaviour.

Script for optimizing netcdf files

the script globsim/interpolate/optimize.sh transforms downloaded netcdf files so that reading all times is much faster

• Dimension order: time as the fastest-varying dimension
• Chunking: store data in chunks of size (5, 5, max, max) (latitude, longitude, level, time). I'm not certain that 5 is the right size for latitude and longitude but it seems to work.
• example usage:

globsim/interpolate/optimize.sh /data/globsim/n60/era5 /data/globsim/n60_optim/era5 --year 2001  # processes files for 2001

Flexible dimension reading

• Add a function reorder_and_slice_array that accepts an array, an ordering scheme for the array, and an ordering scheme
• You can also provide named slice boundaries in order to extract data out of the array
  • e.g. all data at a particular time reorder_and_slice_array(slice_boundaries={'time':slice(1,100)})
  • e.g. a spatial subset reorder_and_slice_array( slice_boundaries={'latitude':slice(1,5), 'longitude':slice(1,5)})
• This could be made into a python utility eventually to more clearly integrate into globsim

keyword toggle

• in the toml config file, set optimized=true to use the optimized files

Benchmarks:

• interpolation of 2 nearby sites for 6 years of era5 took about 5 minutes.
• • interpolation of same sites for 11 years of era5 took about 17 minutes.
• optimization script takes about 1 hour for 12 months of era5 data ( 45 x 205 grid cells)

Closes #127
Closes #154