Kotlin migration and some improvement suggestions

[kubovy]

My recent development, I thought I will share it with you. If you like we can also discuss some strategy how to get the following, or parts of it, merged. I can also split this draft PR into multiple ones to make it easier to review. This should show you the ideas I had to improve this nice library.

I found easier development with Kotlin, since it gets rid of certain boilerplate and handles common issues, e.g. NPE. You can see especially on the end result that the API is more fixed during development and one gets less opportunity to run into runtime errors. Also you can see that by keeping the same functionality and even extending it considerably the code size shrank about 3000 lines of code (about 20%).

During my development I got a deeper understanding of the format and the differences between the two editions. See e.g. the GribMessage vs. GribRecord interfaces.

I thought it may be nice to support also some transformations in the future. See for example the "cutOut" which cuts a sub region from the GRIB and since there is a partial write support the result can also be written out. Having clear and unified APIs (see api package) will enable to write such functionality easier.

I was searching for a Java library supporting both GRIB editions. Those I found were whether not maintained anymore or supported only one of the two editions in a very simple way. The general code structure in this library reflects more closely the specification. For my own project I need certain improvements, e.g. simple write support (no need to support every type of grid or data template), some simple transformations and a easy to orient and development code base. I see this library having a great potential and I hope you find those contribution (or its parts) useful and hope to get them into the code-base.

To the different commits:

Kotlin migration (no functional changes, pure semi-automated migration)

• Optionals part of language - more stable API
• Simpler API (properties = field, getter, setter)
• ...
• needs gradle 6 (using gradle wrapper)

Extend GRIB2 parameters

• Allow index ranges
• Filter RESERVED and MISSING params out to fail on processing
• Extend METEROLOGICAL (Discipline 0) - TEMPERATURE (Category 0)
• Extend METEROLOGICAL (Discipline 0) - MOISTURE (Category 1)
• Extend METEROLOGICAL (Discipline 0) - MOMENTUM (Category 2)
• Extend METEROLOGICAL (Discipline 0) - MASS (Category 3)
• Add METEROLOGICAL (Discipline 0) - CLOUD (Category 6)
• Add METEROLOGICA (Discipline 0) - THERMODYNAMIC_STABILITY_INDICES (Category 7)
• Add METEROLOGICAL (Discipline 0) - TRACE_GASES (Category 14)
• Add HYDROLOGICAL (Discipline 1) - BASIC (Category 0)
• Add HYDROLOGICAL (Discipline 1) - PROBABILITIES (Category 1)
• Add LAND_SURFACE (Discipline 2) - VEGETATION_BIOMASS (Category 0)
• Add LAND_SURFACE (Discipline 2) - AGRICULTURAL_AQUACULTURAL (Category 1)
• Add LAND_SURFACE (Discipline 2) - SOIL (Category 3)
• Add LAND_SURFACE (Discipline 2) - FIRE_WEATHER (Category 4)
• Add LAND_SURFACE (Discipline 2) - LAND_SURFACE (Category 5)
• Add OCEANOGRAPHIC (Discipline 10) - WAVES (Category 0)
• ADD OCEANOGRAPHIC (Discipline 10) - CURRENTS (Category 1)
• ADD OCEANOGRAPHIC (Discipline 10) - ICE (Category 2)

Extend levels

Filter parameters to speedup reading

Data Representation Template 5.0 and 5.2

• Move DRS/DS to type based sub-classes
• Implement Grid Point Data - Simple Packing (DRS/DS)
• Implement Grid Point Data - Complex Packing (DRS/DS)

GRIB2 Write support

• Unify Read/Write API
• Write generic write support for all sections (including tests)
• Write support for Grid Point Data - Simple Packing
• Revise documentation for GRIB2 class - links to www.nco.ncep.noaa.gov
• Common interface for GRIB sections

GRIB2 section repeat support

• Sections 2 to 7, 3 to 7, or 4 to 7 may be repeated
• Split GribRecord to GribMessage and GribRecord to reflect this concept

Improved performance while handling read exceptions

On exceptions while reading a GRIB message the rest of the message will be skipped rather than seeking for next "GRIB" magic sequence, which is rather slow. This relies on the "recordLength" from the Indicator Section.

Fallback will still seek next "GRIB" magic sequence. This is done before reading each GRIB message to allow non-GRIB data as is the case for example from https://www.cpc.ncep.noaa.gov/products/wesley/get_gfs.html

Add OpenSkiron test data

• Fix time zones
• Remove TimeZone.setDefault(TimeZone.getTimeZone("UTC"))
    and set UTC time zone explicitly not to mess with other
    parts where this library is used

GRIB1 Write support

• Unify Read/Write API
• Write generic write support for all sections (including tests)
• Revise documentation for GRIB1 class - links to https://apps.ecmwf.int/codes/grib/format/grib1/overview

GRIB2 Cut our a sub-region including tests

This change needed to:

• move data related API to Data Section
• unify GRIB1 and GRIB2 related API
• improve and unify Data Section API across editions
• abstract and unify Grid Section API across editions
• GRIB1 API prepared but not implemented

Filter parameters to speedup reading

• unify related API across editions
• common level interface
• common parameter interface
• common product definition section interface

Cleanup byte conversion helper

• simplify and move to Grib(Input/Output)Stream

Change custom compare methods to Comparable interface

Conversion Data Representation Section 2/3 to Data Representation Section 0

• Simple conversion from DRS2/3 to DRS0 by re-instantiating
• Update all related properties in a GRIB2 message
• Re-instantiate related Data Section
• This enables writing GRIB2 messages without write support in DRS2/3

Update command line tool to support new features

• parameter filter
• output file for writing
• data representation section conversion
• cut-out region

GRIB2 Bit Map Section

GRIB2 Product Definition Template 4.8

Average, Accumulation and/or Extreme values or
other Statistically-processed values at a horizontal level or
in a horizontal layer in a continuous or non-continuous time interval

Message/record read filter to speedup reading data of interest

[spidru]

I'm quite impressed with the amount of work you've put in here! To be honest it's quite a lot to take in, especially since I'm not really familiar with Kotlin. This might take me a while to process, since I also need to be sure I am also in a position to fully maintain the code myself. I will comment on individual code changes gradually.

I'm glad to hear you see the potential in the library. I was also looking for a maintained GRIB library (ideally in Java) when I was getting started with this project. I initially started with JGrib, but after discovering several bugs and that it is no longer maintained, I decided to fork out my own version, complete with GRIB2 support, while keeping a simple API for both GRIB versions.

[kubovy]

Thanks! I believe you will find Kotlin as a future version of Java. In times of Java8 I called it Java 11 :). If you are familiar with optionals and functional interfaces you are half way there. One think I like in particular is the concept of properties replacing fields, getters and setter (https://kotlinlang.org/docs/properties.html). Properties in particular make the API much more readable and IMHO easier to maintain. I tried to demonstrate it in the commits after the "Kotlin migration".

I would prefer integrating my contributions to your work rather than using own fork. Let me know when I can support with integration (splitting, reorganizing, ci, tests, changing, ...). I will check your comments as they come in.