Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a converter for amusa from bufr to ioda #754

Closed
wants to merge 78 commits into from
Closed
Changes from 4 commits
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
9285f5f
update prototype_3d observation list to process satwind_goes-16, upda…
RussTreadon-NOAA Nov 9, 2023
9189e6f
Merge develop at 957a2fc into feature/gdas-validation
RussTreadon-NOAA Nov 9, 2023
f5320c8
update GDAS/orion.lua to use spack-stack-1.5.1, bufr/12.0.1, and atla…
RussTreadon-NOAA Nov 9, 2023
8c34966
Remove a double-declaired "obs post filters" in satwnd GOES YAMLs.
emilyhcliu Nov 11, 2023
78fc82f
Merge branch 'develop' into feature/gdas-validation
CoryMartin-NOAA Nov 13, 2023
e1a5375
Modify BUFR converter and filter yamls for GOES AMV (#724)
emilyhcliu Nov 14, 2023
dda513f
Make the python bufr2ioda driver more generic (#725)
CoryMartin-NOAA Nov 14, 2023
d368a2a
Merge branch 'develop' into feature/gdas-validation
CoryMartin-NOAA Nov 14, 2023
6e6a480
Modify BUFR converter and filter yamls for MetoOp Scatwind (#731)
emilyhcliu Nov 15, 2023
5c305aa
Have the driver script process both python scripts and YAMLs + bufr2i…
CoryMartin-NOAA Nov 15, 2023
9fc4d73
Use parallel processing to speed up obs processing (#733)
CoryMartin-NOAA Nov 16, 2023
8f6b9c1
temp
Nov 16, 2023
0970ec8
Fix YAML so minimization works for satwind G17 (#743)
CoryMartin-NOAA Nov 17, 2023
ae22466
Consolidate parallel processing (#745)
CoryMartin-NOAA Nov 17, 2023
d331a25
Add AHI/Himawari-8 assimilation to end-to-end GDASApp validation (#746)
BrettHoover-NOAA Nov 17, 2023
59db885
Merge branch 'develop' into feature/gdas-validation
CoryMartin-NOAA Nov 17, 2023
673c086
amsua converter initial set
Nov 17, 2023
0092f53
merge gdas-validation
xincjin-NOAA Nov 19, 2023
820d845
forgot undating run_bufr2ioda.py
xincjin-NOAA Nov 19, 2023
e4bc8e0
a working version
xincjin-NOAA Nov 19, 2023
10f01db
Write the bufr2ioda config files in DATA (#748)
CoryMartin-NOAA Nov 20, 2023
bcfefae
remove a json file
xincjin-NOAA Nov 20, 2023
37409a7
Merge branch 'feature/gdas-validation' into feature/gdas-validation_a…
xincjin-NOAA Nov 20, 2023
8762fbd
correct code norm
xincjin-NOAA Nov 20, 2023
2d32978
a few more
xincjin-NOAA Nov 20, 2023
c6ebf2f
a few more
xincjin-NOAA Nov 20, 2023
1a1d09e
a few more
xincjin-NOAA Nov 20, 2023
d982094
a few more
xincjin-NOAA Nov 20, 2023
a910c82
dele last line
xincjin-NOAA Nov 20, 2023
56810e2
no new line at the end of the file
xincjin-NOAA Nov 20, 2023
7845c79
Update GNSSRO assimilation to end-to-end GDASApp validation (#753)
XuanliLi-NOAA Nov 20, 2023
047e914
Update for ATMS for End-to-End Test (#757)
emilyhcliu Nov 20, 2023
abe9fce
Update amsua obs YAML files
emilyhcliu Nov 21, 2023
f572a2b
For end-to-end configuration AMSU-A YAML files: update n19 and add n1…
emilyhcliu Nov 21, 2023
179d879
Add various platforms for amsua for 3dvar test
emilyhcliu Nov 21, 2023
92e99a4
Bug fixes: define Sensor_ID, modify obsfile names to follow naming co…
emilyhcliu Nov 21, 2023
f34c145
Update diagnostic flags, add thinning and scan edge removal filters
emilyhcliu Nov 21, 2023
d7a0378
Remove extra comments
emilyhcliu Nov 21, 2023
2dd725f
[End-to-End Test Code Sprint] Add SEVIRI METEOSAT-8 satwinds to end-t…
BrettHoover-NOAA Nov 21, 2023
25ab8ce
Do nothing.
emilyhcliu Nov 21, 2023
4fc8cf0
Merge branch 'feature/gdas-validation' into feature/gdas-validation-a…
emilyhcliu Nov 21, 2023
df6c6a1
Merge branch 'feature/gdas-validation' into feature/gdas-validation_a…
emilyhcliu Nov 21, 2023
94abd9a
Added SEVIRI/METEOSAT-11 satwinds to end-to-end validation (#767)
BrettHoover-NOAA Nov 27, 2023
b5ac9d8
Merge branch 'feature/gdas-validation' into feature/gdas-validation-a…
CoryMartin-NOAA Nov 27, 2023
e7fc7cb
A few updates for ATMS end-to-end testing (#768)
emilyhcliu Nov 27, 2023
40a1d36
Merge branch 'feature/gdas-validation' into feature/gdas-validation-a…
emilyhcliu Nov 27, 2023
cd93876
Remove temporary checkout from JCSDA-internal/ufo
emilyhcliu Nov 27, 2023
27ee384
add no python option
xincjin-NOAA Nov 28, 2023
0e1ffd4
Re-arrange and tidy up the YAML files for AMSU-A
emilyhcliu Nov 28, 2023
266e153
change post filters to pre filters for diagnostic flags
emilyhcliu Nov 28, 2023
56bfda2
Revert 3dvar_dripcg.yaml
CoryMartin-NOAA Nov 28, 2023
94cec74
a new version of amsua converter
xincjin-NOAA Nov 29, 2023
da3b45f
Merge branch 'feature/gdas-validation_amsua' of https://github.com/NO…
xincjin-NOAA Nov 29, 2023
812904c
revert CMakeLists.txt
xincjin-NOAA Nov 29, 2023
deddffd
revert parm/atm/variational/3dvar_dripcg.yaml
xincjin-NOAA Nov 29, 2023
c739271
Merge branch 'feature/gdas-validation-amsua' into feature/gdas-valida…
xincjin-NOAA Nov 29, 2023
f29a762
for test
xincjin-NOAA Nov 30, 2023
771c959
Merge branch 'develop' into feature/gdas-validation
CoryMartin-NOAA Nov 30, 2023
ce8fad8
a few small changes
xincjin-NOAA Nov 30, 2023
941cb2b
optimized apply_ant_corr
xincjin-NOAA Dec 1, 2023
654dcdf
Minor observation YAML updates for ATMS, SATWIND and SCATWIND (#789)
emilyhcliu Dec 4, 2023
947b397
Merge branch 'develop' into feature/gdas-validation
CoryMartin-NOAA Dec 4, 2023
de8ff12
add remove function
xincjin-NOAA Dec 5, 2023
30ab4f9
Update IASI YAML files for end-to-end testing (#769)
emilyhcliu Dec 5, 2023
7f0a295
temp
xincjin-NOAA Dec 7, 2023
4c2e8b8
merge
xincjin-NOAA Dec 7, 2023
f63bfd4
a few small changes
xincjin-NOAA Dec 9, 2023
80e04d8
Merge branch 'develop' into feature/gdas-validation
RussTreadon-NOAA Dec 13, 2023
c2d9edb
Merge branch 'develop' into feature/gdas-validation
RussTreadon-NOAA Dec 13, 2023
99c431a
Update observation YAMLs for conventional surface pressure (#803)
emilyhcliu Dec 21, 2023
30543ab
a working version matched with GSI
xincjin-NOAA Jan 22, 2024
bd04d95
code norm check
xincjin-NOAA Jan 22, 2024
086cff3
for ctest
xincjin-NOAA Jan 23, 2024
5e34fc0
gdas_validation merge
xincjin-NOAA Jan 23, 2024
6ff21df
merge develop
xincjin-NOAA Jan 23, 2024
bb2ea3c
merge with develop
xincjin-NOAA Mar 19, 2024
c49d17d
make it working after merging develop
xincjin-NOAA Mar 20, 2024
ba8a1fa
some clean and log
xincjin-NOAA Mar 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 99 additions & 43 deletions parm/atm/obs/config/conv_ps.yaml
Original file line number Diff line number Diff line change
@@ -3,11 +3,11 @@ obs space:
obsdatain:
engine:
type: H5File
obsfile: $(DATA)/obs/$(OPREFIX)sondes.{{ current_cycle | to_YMDH }}.nc4
obsfile: $(DATA)/obs/$(OPREFIX)conventional_ps.prepbufr.nc
obsdataout:
engine:
type: H5File
obsfile: $(DATA)/diags/diag_sondes_{{ current_cycle | to_YMDH }}.nc4
obsfile: $(DATA)/diags/diag_conventional_ps_prepbufr_{{ current_cycle | to_YMDH }}.nc
io pool:
max pool size: 1
simulated variables: [stationPressure]
@@ -21,6 +21,11 @@ obs operator:
geovar_sfc_geomz: surface_altitude
geovar_geomz: geopotential_height

linear obs operator:
name: Identity
variables:
- name: stationPressure

obs prior filters:
# Initial Error Assignments for SFC Observations
- filter: Perform Action
@@ -103,36 +108,107 @@ obs prior filters:
xvals: [80000, 75000, 70000, 65000, 60000, 55000 ]
errors: [ 110, 120, 120, 120, 120, 1.0e+11]

obs post filters:
# Observation range sanity check
- filter: Bounds Check
filter variables:
- name: stationPressure
minvalue: 37499.0
maxvalue: 106999.0
action:
name: reject
# Create PreQC group variable (pqm in GSI read_prepbufr)
- filter: Variable Assignment
assignments:
- name: InputObsError/stationPressure
type: float
source variable: ObsErrorData/stationPressure

# Reject all ObsType 183
- filter: RejectList
# Set observation quality-realted variables
# Create PreQC group variable (pqm in GSI read_prepbufr)
- filter: Variable Assignment
assignments:
- name: PreQC/stationPressure
type: int
source variable: QualityMarker/stationPressure

# Create PreUseFlag group variable (usage in GSI read_prepbufr)
# Initialize
- filter: Variable Assignment
assignments:
- name: PreUseFlag/stationPressure
type: int
source variable: PreQC/stationPressure

- filter: Variable Assignment
where:
- variable:
name: PreUseFlag/stationPressure
is_in: 1-15
assignments:
- name: PreUseFlag/stationPressure
value: 0
# Re-assignment
- filter: Variable Assignment
where:
- variable:
name: ObsType/stationPressure
is_in: 183
assignments:
- name: PreUseFlag/stationPressure
value: 100

# Reject surface pressure below 500 hPa
- filter: Bounds Check
filter variables:
- name: stationPressure
minvalue: 50000.00
action:
name: reject
- filter: Variable Assignment
where:
- variable:
name: ObsValue/stationPressure
is_defined:
- variable:
name: ObsValue/stationPressure
maxvalue: 50000.00
where operator: and
assignments:
- name: PreUseFlag/stationPressure
value: 100

- filter: RejectList
- filter: Variable Assignment
where:
- variable:
name: PreQC/stationPressure
is_in: 4-15
is_in: 9, 12, 15
assignments:
- name: PreUseFlag/stationPressure
value: 100

- filter: Variable Assignment
where:
- variable:
name: PreQC/stationPressure
is_in: 4-15
assignments:
- name: PreUseFlag/stationPressure
value: 101

## Observation range sanity check
#- filter: Bounds Check
# filter variables:
# - name: stationPressure
# minvalue: 37499.0
# maxvalue: 106999.0
# action:
# name: reject

## Reject all ObsType 183
#- filter: RejectList
# where:
# - variable:
# name: ObsType/stationPressure
# is_in: 183

## Reject surface pressure below 500 hPa
#- filter: Bounds Check
# filter variables:
# - name: stationPressure
# minvalue: 50000.00
# action:
# name: reject

#- filter: RejectList
# where:
# - variable:
# name: PreQC/stationPressure
# is_in: 4-15

# Inflate obs error based on obs type
- filter: Perform Action
@@ -145,6 +221,7 @@ obs post filters:
name: inflate error
inflation factor: 1.2

obs post filters:
# Calculate obs error inflation factors for duplicated observations at the same location
- filter: Variable Assignment
assignments:
@@ -172,26 +249,6 @@ obs post filters:
name: inflate error
inflation factor: 0.7

# Reduce original observation error based on obs type and subtype
# In this case: reduce original obs error for buoy
- filter: Variable Assignment
where:
- variable:
name: ObsType/stationPressure
is_in: 180
- variable:
name: ObsSubType/stationPressure
is_in: 0
assignments:
- name: ObsError/stationPressure
type: float
function:
name: ObsFunction/Arithmetic
options:
variables:
- name: ObsError/stationPressure
coefs: [0.7]

# Calculate obs error inflation factors for large discrepancies between model and observations
- filter: Variable Assignment
assignments:
@@ -290,6 +347,5 @@ obs post filters:
is_not_in: 0, 1
action:
name: reject

# End of Filters

1 change: 1 addition & 0 deletions parm/atm/obs/lists/gdas_prototype_3d.yaml
Original file line number Diff line number Diff line change
@@ -22,4 +22,5 @@ observers:
##- !INC ${OBS_YAML_DIR}/iasi_metop-b.yaml
##- !INC ${OBS_YAML_DIR}/sfc.yaml
##- !INC ${OBS_YAML_DIR}/sfcship.yaml
##- !INC ${OBS_YAML_DIR}/conv_ps.yaml
##- !INC ${OBS_YAML_DIR}/gnssro.yaml
62 changes: 21 additions & 41 deletions parm/atm/obs/testing/conv_ps.yaml
Original file line number Diff line number Diff line change
@@ -104,15 +104,14 @@ obs prior filters:
xvals: [80000, 75000, 70000, 65000, 60000, 55000 ]
errors: [ 110, 120, 120, 120, 120, 1.0e+11]

obs post filters:
# Observation range sanity check
- filter: Bounds Check
filter variables:
- name: stationPressure
minvalue: 37499.0
maxvalue: 106999.0
action:
name: reject
## Observation range sanity check
#- filter: Bounds Check
# filter variables:
# - name: stationPressure
# minvalue: 37499.0
# maxvalue: 106999.0
# action:
# name: reject

# Reject all ObsType 183
- filter: RejectList
@@ -121,19 +120,19 @@ obs post filters:
name: ObsType/stationPressure
is_in: 183

# Reject surface pressure below 500 hPa
- filter: Bounds Check
filter variables:
- name: stationPressure
minvalue: 50000.00
action:
name: reject
## Reject surface pressure below 500 hPa
#- filter: Bounds Check
# filter variables:
# - name: stationPressure
# minvalue: 50000.00
# action:
# name: reject

- filter: RejectList
where:
- variable:
name: PreQC/stationPressure
is_in: 4-15
#- filter: RejectList
# where:
# - variable:
# name: PreQC/stationPressure
# is_in: 4-15

# Inflate obs error based on obs type
- filter: Perform Action
@@ -146,6 +145,7 @@ obs post filters:
name: inflate error
inflation factor: 1.2

obs post filters:
# Calculate obs error inflation factors for duplicated observations at the same location
- filter: Variable Assignment
assignments:
@@ -173,26 +173,6 @@ obs post filters:
name: inflate error
inflation factor: 0.7

# Reduce original observation error based on obs type and subtype
# In this case: reduce original obs error for buoy
- filter: Variable Assignment
where:
- variable:
name: ObsType/stationPressure
is_in: 180
- variable:
name: ObsSubType/stationPressure
is_in: 0
assignments:
- name: ObsError/stationPressure
type: float
function:
name: ObsFunction/Arithmetic
options:
variables:
- name: ObsError/stationPressure
coefs: [0.7]

# Calculate obs error inflation factors for large discrepancies between model and observations
- filter: Variable Assignment
assignments:
1 change: 1 addition & 0 deletions parm/atm/variational/3dvar_dripcg.yaml
Original file line number Diff line number Diff line change
@@ -3,6 +3,7 @@ cost function:
time window:
begin: '{{ ATM_WINDOW_BEGIN | to_isotime }}'
length: $(ATM_WINDOW_LENGTH)
bound to include: begin
analysis variables: &3dvars [ua,va,t,ps,sphum,ice_wat,liq_wat,o3mr]
geometry:
fms initialization:
Loading