Skip to content
This repository was archived by the owner on Sep 1, 2022. It is now read-only.

GRIB1 data length overflow #445

Closed
JohnLCaron opened this issue Feb 24, 2016 · 11 comments
Closed

GRIB1 data length overflow #445

JohnLCaron opened this issue Feb 24, 2016 · 11 comments
Assignees
@lesserwhirls
Labels
Area: GRIB Type: Bug

Comments

@JohnLCaron
Copy link
Collaborator

JohnLCaron commented Feb 24, 2016 •
edited by lesserwhirls
Loading

When GRIB1 data length > 2^23-1, apparently this is the way it is encoded:

sizeSection4 = (read the 3 bytes of section 4)
If (sizeSection4 < 120) {
// computed using total length – length of all other sections
sizeTotal = (sizeTotal – 7FFFFF ) * 120 – sizeSection4 + 4
sizeSection4 = sizeTotal – 8 –sizeSection1 – sizeSection2 – sizeSection3 – 4;
}

May be ECMWF specific, not sure (sent note to ask).
@JohnLCaron JohnLCaron self-assigned this Feb 24, 2016
@JohnLCaron
Copy link
Collaborator Author

sample file:

http://jmp.sh/qGRzDNB

reported by

martijn.vanderpauw@meteogroup.com

@JohnLCaron
Copy link
Collaborator Author

ECMWF redefines the GRIB format without (AFAIK) documenting it. Now we all run around and adjust. Can we lodge a protest with the United Nations?

OTOH, maybe they did get official permission, and we dont know it.
OTOH, why reward bad behavior?
OTOH, this is just the way it is in GRIBlandia, so stop your whining, its not a bad solution to data size too big.
OTOH, it messes with my pretty code.
OTOH, grow up and get over it.

@lesserwhirls
Copy link
Collaborator

OTOH, it's non-standard GRIB, so the best course of action is to burn it!

IT"S GRIB - BURN IT

But more seriously, it's non-standard GRIB. GRIB is a pain as it is, no need to further complicate it with non-standard stuff. I'd rather try to get users to put pressure on the organization to get them to do the right thing. That said, it's ECMWF, so I have zero idea how they respond to things like this.

@JohnLCaron
Copy link
Collaborator Author

Ok, im going to transfer ownership to you.
if you do decide to do it, note that Grib1RecordScanner has to get more complex because can no longer count on the message length being correct.

@lesserwhirls lesserwhirls changed the title GRIB1 data length overflow GRIB1 data length overflow Sep 8, 2016
@DCSCHUS
Copy link

DCSCHUS commented Apr 5, 2017

Do you have any updates on the status of this issue? We have users that have been requesting access to this dataset through TDS/OPeNDAP. Thx.

@lesserwhirls
Copy link
Collaborator

Greetings @DCSCHUS - do you have an example file I could play with (I may have asked before, but I've misplaced the file if I have it).

@DCSCHUS
Copy link

DCSCHUS commented May 26, 2017 via email

@lesserwhirls
Copy link
Collaborator

lesserwhirls commented Jun 7, 2017 via email

@DCSCHUS
Copy link

DCSCHUS commented Jun 7, 2017 via email

@sefadolunay
Copy link

Hi, we have emailed with Sean about netCdf java support of ECMWF grib1 files at 27/12/2017.

These files can be read with grib-api, an ECMWF grib library which can be found here

I provide here a sample file and its grib-api dump.
Sample file
grib-api dump

Basically, what is wrong with these files is the "totalLength" section at bytes 5-7 of each message. Netcdf Java tries to validate these files by looking at "totalLength" area and checks whether the end of section 5 message is "7777". Due to wrong length value, it gives "not a valid cdm file" exception.

In my opinion, these files have a fixed length sections for each message. So, grib-api can read them without referring to "totalLength" section. But this information needs to be approved.

I need to be able to open these files with netcdf java. Let me know if I can contribute the open source development process.

Thank you.

@cofinoa
Copy link
Contributor

cofinoa commented Jan 11, 2018

@sefadolunay,
you can use that files setting 2 properties at Grib1RecordScanner class:

  ucar.nc2.grib.grib1.Grib1RecordScanner.setAllowBadDsLength(true);
  ucar.nc2.grib.grib1.Grib1RecordScanner.setAllowBadIsLength(true);
  NetcdfFile nc = NetcdfFile.open(filename);
  System.out.println(nc);

Regards

cofinoa added a commit to cofinoa/thredds that referenced this issue Jan 12, 2018
@cofinoa
Fixes Unidata#445: GRIB1 data length overflow
028a22a
@cofinoa cofinoa mentioned this issue Jan 12, 2018
Merged
lesserwhirls added a commit that referenced this issue Jan 16, 2018
@lesserwhirls
Merge pull request #1005 from cofinoa/#445_GRIB1_data_length_overflow
9fc0776 
Fixes #445: GRIB1 data length overflow
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@lesserwhirls lesserwhirls

Labels
Area: GRIB Type: Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@lesserwhirls @dopplershift @JohnLCaron @cofinoa @DCSCHUS @sefadolunay