Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GRIB table updates #601

Merged
merged 10 commits into from
Feb 12, 2021
Merged

Conversation

lesserwhirls
Copy link
Collaborator

@lesserwhirls lesserwhirls commented Feb 10, 2021

Updated the following GRIB tables:

  • MRMS GRIB2 Tables to v12.0.
  • NCEP GRIB 1 Tables.
  • experimental HRRR GRIB2 tables to v4.
  • NDFD GRIB2 tables to reflect degrib v2.25.

Fixed the following new test failures:

  • Two new failing tests due to GRIB name changes due to the GRIB table updates.
  • Unrelated test failure to due an updated last modified time for OSDC object store test.

Finally, bumped the version of the apache xmlbeans dependency.

The full test run against this PR is passing on Jenkins.


This change is Reviewable

The last modified time for one of the GOES 16 objects stored in the Open
Science Data Cloud changed and was causing a test to fail. The actual
file represented in the blob and its associated data didn't change, so
perhaps the OSDC bucket was migrated to a different system?
Copy link
Contributor

@haileyajohnson haileyajohnson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wow, @lesserwhirls this seems like it was really tedious 😬
Looks good to me, as long as the Python checks out with @dopplershift !

Reviewed 32 of 32 files at r1.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on @dopplershift)

@lesserwhirls
Copy link
Collaborator Author

Yeah, the python is a bit of a hot mess, but not too terribly so. @dopplershift - I modified what you had before to work with python 3, and to provide the option to interactively look at the differences between MRMS tables when there are differences between tables for a given discipline-category-number combination.

Copy link
Member

@dopplershift dopplershift left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Python is complicated, but it's by no means an abomination.

I've made some comments to varying degrees from nitpicky to some good improvements. Feel free to disregard anything you don't feel like dealing with--the important thing is that it produces the file you want.

selection -= 1
valid_selection = True

if not valid_selection:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could just be the else clause to the if block right above, since the only way valid_selection is set to True is if you get in one of those. You could also make the whole while block a while True: and use break to leave it and completely eliminate valid_selection.

lesserwhirls added a commit to lesserwhirls/thredds that referenced this pull request Feb 10, 2021
Backport from netCDF-Java 5 (Unidata/netcdf-java#601)
@lesserwhirls
Copy link
Collaborator Author

Thanks for the feedback @dopplershift! Should have those taken care of in the latest changes.

lesserwhirls added a commit to lesserwhirls/thredds that referenced this pull request Feb 10, 2021
Backport from netCDF-Java 5 (Unidata/netcdf-java#601)
lesserwhirls added a commit to lesserwhirls/thredds that referenced this pull request Feb 10, 2021
Backport from netCDF-Java 5 (Unidata/netcdf-java#601)
Copy link
Collaborator Author

@lesserwhirls lesserwhirls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 28 of 32 files reviewed, 11 unresolved discussions (waiting on @dopplershift, @haileyajohnson, and @lesserwhirls)


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 142 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

I think this should do the same:

for version in range(1, len(param_versions)):
    if not parameter_comparison(first, param_versions[version]):
        break
else: # Didn't break
    unique_params.append(param_versions[-1])
    continue

dupes += 1
...

I'm not sure if that's actually clearer, but it eliminates the has_diffs flag.

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 164 at r1 (raw file):

Previously, lesserwhirls (Sean Arms) wrote…

sigh got it now, missed it before. Thanks for the catch!

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 179 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

This could just be the else clause to the if block right above, since the only way valid_selection is set to True is if you get in one of those. You could also make the whole while block a while True: and use break to leave it and completely eliminate valid_selection.

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 206 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

pathlib may make things easier here:

from pathlib import Path
tables = Path('tables/').iterdir()

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 211 at r1 (raw file):

Previously, lesserwhirls (Sean Arms) wrote…

I was trying to support these three cases:

convert_mrms_table.py --tables path/to/table1
convert_mrms_table.py --tables path/to/table1 --tables path/to/table2
convert_mrms_table.py --tables path/to/table1 path/to/table2

which required that .add_argument use action='append', nargs='+'. That resulted in argparse creating a list of lists. However, that's just overcomplicating things really, so I've simplified it by only supporting the case of multiple --table flags (see update).

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 213 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

Usually no spaces around = here.

Done.

Copy link
Collaborator Author

@lesserwhirls lesserwhirls left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 28 of 32 files reviewed, 11 unresolved discussions (waiting on @dopplershift and @haileyajohnson)


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 3 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

Usually from datetime import datetime, but not a huge deal either.

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 7 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

exists isn't being used

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 61 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

Could use f-strings and parentheses to replace with:

return (f'{param.Discipline}, {param.Category}, {param.Parameter}, "{param.Name}", '
        f'"{param.Description}", "{param.Unit}", {param.No_Coverage}, '
        f'{param.Missing}')

That's purely stylistic though.

Done.


grib/src/main/resources/resources/grib2/mrms/convert_mrms_table.py, line 69 at r1 (raw file):

Previously, dopplershift (Ryan May) wrote…

Same thing here.

Done.

@haileyajohnson haileyajohnson merged commit e109626 into Unidata:maint-5.x Feb 12, 2021
@lesserwhirls lesserwhirls deleted the gribTables branch February 27, 2021 23:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants