Hotfix to add releaseTime to adpupa converter #1328

haydenlj · 2023-08-16T15:20:03Z

Description

Temporary hotfix to add releaseTime to files output by the new adpupa prepbufr to ioda converter so that output files can be used by Skylab. Can be reverted once the planned more robust additions have been made

Issue(s) addressed

Resolves #1322

Dependencies

none

Impact

Output IODA files can be used in Skylab

Checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have run the unit tests before creating the PR

rmclaren

I don't believe this is what you want... The reason is that now you've mixed your data set with fields that are grouped by prepbufrDataLevelCategory (latitude, longitude, stationElevation... and so on) with things that are not grouped (timeOffset, specificHumidity)... The problem here is that you've lost your ability to correlate between these fields (for example you can no longer tell how latitude, longitude lines up with specific humidity). You have to be consistent with grouping to make datasets that make sense.

haydenlj · 2023-08-16T16:02:39Z

I don't believe this is what you want... The reason is that now you've mixed your data set with fields that are grouped by prepbufrDataLevelCategory (latitude, longitude, stationElevation... and so on) with things that are not grouped (timeOffset, specificHumidity)... The problem here is that you've lost your ability to correlate between these fields (for example you can no longer tell how latitude, longitude lines up with specific humidity). You have to be consistent with grouping to make datasets that make sense.

@rmclaren Ah ok, in that case I misunderstand what the bufr get function is doing. So it is sorting by prepbufrDataLevelCategory rather than just flattening along that axis?

rmclaren · 2023-08-16T16:10:37Z

@haydenlj The relative order stays the same (its not due to sorting). When you group a field what you are doing is reshaping the array. If you don't reshape your arrays in a consistent way then data in those arrays won't line up in the way you expect.

PraveenKumar-NOAA

Only the grouping issues, rest looks fine.

test/testinput/bufr_ncep_prepbufr_adpupa.py

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

huishao-r

I can't comment on other parts regarding the grouping. But the release time seems fine with me.

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

test/testinput/bufr_ncep_prepbufr_adpupa.py

Changes addressed

BenjaminRuston · 2023-08-30T14:43:33Z

we do not think there is any grouping issue and we would like to have NOAA EMC test this PR to see if it still reproduces the answer you expect.

@emilyhcliu please advise

PraveenKumar-NOAA · 2023-08-30T15:04:44Z

we do not think there is any grouping issue and we would like to have NOAA EMC test this PR to see if it still reproduces the answer you expect.

@emilyhcliu please advise

@BenjaminRuston I will be happy to test this out.

PraveenKumar-NOAA · 2023-09-05T18:13:27Z

we do not think there is any grouping issue and we would like to have NOAA EMC test this PR to see if it still reproduces the answer you expect.

@emilyhcliu please advise

@BenjaminRuston I tested this PR/ Python script and it is reproducing the expected results. If @emilyhcliu @rmclaren are happy with this new format, I will approve this PR.

PraveenKumar-NOAA · 2023-09-05T18:27:17Z

@haydenlj Could you also try running the ioda-validate.x program on the output file that gets created? That will help ensure we also don't have something like MetaData/height as an integer when it should be a float. Or plenty of other possible examples as well. This just seems like the right time to catch anything we can that could become technical debt later.

@gthompsnJCSDA I ran ioda-validate.x program on the output file and there were a few errors/ warnings in the final results. A few new prepBUFR variables like prepbufrDataLevelCategory (CAT) and temperatureEventCode (TPC) also need to be added to the ObsSpace.yaml. I will create an issue for this.

gthompsnJCSDA · 2023-09-05T19:45:18Z

@haydenlj Could you also try running the ioda-validate.x program on the output file that gets created? That will help ensure we also don't have something like MetaData/height as an integer when it should be a float. Or plenty of other possible examples as well. This just seems like the right time to catch anything we can that could become technical debt later.

@gthompsnJCSDA I ran ioda-validate.x program on the output file and there were a few errors/ warnings in the final results. A few new prepBUFR variables like prepbufrDataLevelCategory (CAT) and temperatureEventCode (TPC) also need to be added to the ObsSpace.yaml. I will create an issue for this.

It seems we must already have something similar to DataLevelCategory - could you try to re-use something that is similar, such as verticalSignificance? And, isn't temperatureEvent already in the ObsSpace.yaml and was already exactly designed for radiosonde data I thought (similar to windEvent and moistureEvent, etc.).

BenjaminRuston · 2023-09-06T17:58:07Z

@emilyhcliu looks like @PraveenKumar-NOAA has tested and happy with result are you ready to move forward

PraveenKumar-NOAA · 2023-09-07T18:28:13Z

@gthompsnJCSDA thanks, there is no need to add any extra variables to the ObsSpace.yaml for the adpupa prepbufr.

@haydenlj please use ObsValue/verticalSignificance for MetaData/prepbufrDataLevelCategory (CAT) and QCFlags/qualityFlags for the MetaData/temperatureEventCode (TPC), then there will be no error in your ioda-validate.x run, you just need to take care of a few warnings. I can do that easily.

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

haydenlj · 2023-09-07T19:13:53Z

QCFlags/qualityFlags

Ok thanks @PraveenKumar-NOAA I updated the names

BenjaminRuston · 2023-09-11T19:15:25Z

@haydenlj looks like the testoutput may need to be updated:

Write data to variables
DIFFER : NUMBER OF GROUPS : 4 ("/") <> 3 ("/")
DIFFER : VARIABLE : prepbufrDataLevelCategory : DOES NOT EXIST IN "testrun/prepbufr_adpupa_api.nc"
DIFFER : VARIABLE : temperatureEventCode : DOES NOT EXIST IN "testrun/prepbufr_adpupa_api.nc"
DIFFER : VARIABLE : verticalSignificance : DOES NOT EXIST IN "testoutput/prepbufr_adpupa_api.nc"
DIFFER : Failed to find group /QCFlags in file "testoutput/prepbufr_adpupa_api.nc"

emilyhcliu · 2023-09-11T20:08:24Z

@haydenlj Could you give me the definition of the releaseTime? Is it the earliest time from dateTime of each profile?

PraveenKumar-NOAA

Looks good to me.

BenjaminRuston · 2023-09-13T23:23:02Z

test/testinput/bufr_ncep_prepbufr_adpupa.py

    print("cycleTimeSinceEpoch")
    cycleTimeSinceEpoch = np.int64(calendar.timegm(time.strptime(date, '%Y%m%d%H%M')))
    hrdr = np.int64(hrdr*3600)
    hrdr += cycleTimeSinceEpoch

+    ulan = np.repeat(hrdr[:,0], hrdr.shape[1])


@emilyhcliu here is the definition of the variable that is used to define the release time which is the first entry of the times array

… processing

BenjaminRuston

@haydenlj had to make one more change to make sure it ran through Skylab

Hotfix to add releaseTime to adpupa converter

aa7b3e2

haydenlj added the OBS OBS processing, UFO label Aug 16, 2023

haydenlj requested review from rmclaren, fcvdb, emilyhcliu, smaticka, PraveenKumar-NOAA and BenjaminRuston August 16, 2023 15:20

haydenlj self-assigned this Aug 16, 2023

rmclaren previously requested changes Aug 16, 2023

View reviewed changes

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

3c5c2d0

PraveenKumar-NOAA reviewed Aug 16, 2023

View reviewed changes

test/testinput/bufr_ncep_prepbufr_adpupa.py Show resolved Hide resolved

test/testinput/bufr_ncep_prepbufr_adpupa.py Show resolved Hide resolved

haydenlj requested a review from gthompsnJCSDA August 16, 2023 16:39

haydenlj added 2 commits August 16, 2023 14:56

data matchup correction

5fad5d2

Merge branch 'bugfix/hotfix_for_releaseTime' of https://github.com/JC…

f1c04e6

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

haydenlj requested review from PraveenKumar-NOAA and rmclaren August 16, 2023 19:57

huishao-r reviewed Aug 22, 2023

View reviewed changes

haydenlj added 3 commits August 22, 2023 11:02

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

7444b3b

correct launchTime axis order

19ad996

Merge branch 'bugfix/hotfix_for_releaseTime' of https://github.com/JC…

c8753a3

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

haydenlj commented Aug 23, 2023

View reviewed changes

test/testinput/bufr_ncep_prepbufr_adpupa.py Show resolved Hide resolved

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

204838a

haydenlj added 4 commits August 25, 2023 13:50

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

9232a97

update test output file

9ac64f3

test for station elevation shape correction

a91c8ea

update testoutput

38f98ab

haydenlj and others added 2 commits August 29, 2023 16:32

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

178bf51

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

c3c49c3

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

ea52916

BenjaminRuston requested a review from huishao-r September 6, 2023 17:58

haydenlj added 3 commits September 7, 2023 12:07

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

32d32e3

change variable names

effd0df

Merge branch 'bugfix/hotfix_for_releaseTime' of https://github.com/JC…

cbe9422

…SDA-internal/ioda-converters into bugfix/hotfix_for_releaseTime

PraveenKumar-NOAA approved these changes Sep 11, 2023

View reviewed changes

BenjaminRuston removed the do not merge Something is wrong, do not merge label Sep 13, 2023

BenjaminRuston and others added 3 commits September 13, 2023 11:07

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

4a2d7ad

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

8ffa0c5

update testoutput for file converted from prepbufr adpupa

defc4ed

BenjaminRuston reviewed Sep 13, 2023

View reviewed changes

BenjaminRuston and others added 4 commits September 14, 2023 18:41

remove prepbufrdatalevelcategory from output it is still used in data…

bd4f3c4

… processing

update file with removal of prepbufrdatalevelcategory in output

08eecbc

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

0394a16

Merge branch 'develop' into bugfix/hotfix_for_releaseTime

e3bcb8d

BenjaminRuston approved these changes Sep 15, 2023

View reviewed changes

PatNichols merged commit c9a0ca6 into develop Sep 19, 2023

PatNichols deleted the bugfix/hotfix_for_releaseTime branch September 19, 2023 18:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hotfix to add releaseTime to adpupa converter #1328

Hotfix to add releaseTime to adpupa converter #1328

haydenlj commented Aug 16, 2023 •

edited

Loading

rmclaren left a comment •

edited

Loading

haydenlj commented Aug 16, 2023

rmclaren commented Aug 16, 2023

PraveenKumar-NOAA left a comment

huishao-r left a comment

BenjaminRuston commented Aug 30, 2023

PraveenKumar-NOAA commented Aug 30, 2023

PraveenKumar-NOAA commented Sep 5, 2023

PraveenKumar-NOAA commented Sep 5, 2023 •

edited

Loading

gthompsnJCSDA commented Sep 5, 2023

BenjaminRuston commented Sep 6, 2023

PraveenKumar-NOAA commented Sep 7, 2023

haydenlj commented Sep 7, 2023

BenjaminRuston commented Sep 11, 2023

emilyhcliu commented Sep 11, 2023

PraveenKumar-NOAA left a comment

BenjaminRuston Sep 13, 2023

BenjaminRuston left a comment

Hotfix to add releaseTime to adpupa converter #1328

Hotfix to add releaseTime to adpupa converter #1328

Conversation

haydenlj commented Aug 16, 2023 • edited Loading

Description

Issue(s) addressed

Dependencies

Impact

Checklist

rmclaren left a comment • edited Loading

Choose a reason for hiding this comment

haydenlj commented Aug 16, 2023

rmclaren commented Aug 16, 2023

PraveenKumar-NOAA left a comment

Choose a reason for hiding this comment

huishao-r left a comment

Choose a reason for hiding this comment

BenjaminRuston commented Aug 30, 2023

PraveenKumar-NOAA commented Aug 30, 2023

PraveenKumar-NOAA commented Sep 5, 2023

PraveenKumar-NOAA commented Sep 5, 2023 • edited Loading

gthompsnJCSDA commented Sep 5, 2023

BenjaminRuston commented Sep 6, 2023

PraveenKumar-NOAA commented Sep 7, 2023

haydenlj commented Sep 7, 2023

BenjaminRuston commented Sep 11, 2023

emilyhcliu commented Sep 11, 2023

PraveenKumar-NOAA left a comment

Choose a reason for hiding this comment

BenjaminRuston Sep 13, 2023

Choose a reason for hiding this comment

BenjaminRuston left a comment

Choose a reason for hiding this comment

haydenlj commented Aug 16, 2023 •

edited

Loading

rmclaren left a comment •

edited

Loading

PraveenKumar-NOAA commented Sep 5, 2023 •

edited

Loading