Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the eomg job #3331

Open
wants to merge 12 commits into
base: develop
Choose a base branch
from
Open

Conversation

DavidGrumm-NOAA
Copy link
Contributor

@DavidGrumm-NOAA DavidGrumm-NOAA commented Feb 14, 2025

Description

The eomg job is no longer supported for the GSI and the GDASApp has its own jobs to handle observational diagnostics (i.e. when lobsdiag=".true."): atmensanlobs and atmensanlsol. Thus, the eomg job will be removed from the global workflow without affecting the new atmensanlobs and atmensanlsol jobs.

This PR resolves issue #2525.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES/NO (If YES, please add a link to any PRs that are pending.)
    • [NO] EMC verif-global
    • [NO] GDAS
    • [NO] GFS-utils
    • [NO] GSI
    • [NO] GSI-monitor
    • [NO] GSI-utils
    • [NO] UFS-utils
    • [NO] UFS-weather-model
    • [NO] wxflow

How has this been tested?

Checklist

  • Any dependent changes have been merged and published
  • [X ] My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

@DavidGrumm-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA @DavidHuber-NOAA @aerorahul : this PR is ready for review. There are some differences between the baseline output and the updated code output, which is unexpected. @DavidHuber-NOAA has suggested we go ahead with this PR and then rerun the test once approvals are made.

Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In addition to the comments, there are some additional places where eomg needs to be removed. Specifically:

parm/config/gfs/config.resources*
parm/config/gfs/config.eobs (this config was used by eomg and there are a couple of eomg-specific variables)
parm/archive/enkf.yaml.j2

Also, there is still a mention of eomg in gfs_tasks.py (under the eupd method).

@DavidHuber-NOAA
Copy link
Contributor

Also, there are some checks for lobsdiag_forenkf in the following files that should be checked. If these checks are for JEDI-based analyses, then they're fine, otherwise the code should be simplified to remove the contents inside of the lobsdiag_forenkf logical blocks. Just a reminder, eomg used to be triggered when lobsdiag_forenkf = .false.

parm/archive/enkf_restarta_grp.yaml.j2
parm/archive/enkf_grp.yaml.j2
parm/config/gfs/config.anal
scripts/exgdas_enkf_update.sh
scripts/exglobal_atmos_analysis.sh

@DavidHuber-NOAA
Copy link
Contributor

@DavidGrumm-NOAA Per our discussion, please just remove eomg and lobsdiag references from GSI-specific files and not those files that are shared by both the GSI and GDASApp. Specifically, please target these files:

parm/config/gfs/config.anal
scripts/exgdas_enkf_update.sh
scripts/exglobal_atmos_analysis.sh
parm/config/gfs/config.resources
parm/config/gfs/config.resources.HERCULES
parm/config/gfs/config.eobs (this config was used by eomg and there are a couple of eomg-specific variables)

@DavidHuber-NOAA DavidHuber-NOAA changed the title Remove the eomg job issue Remove the eomg job Feb 19, 2025
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks pretty good. I just have a couple of suggestions on the namelists since we cannot control the default values of lobsdiag_forenkf and one other place in the setup scripts that needs the flag removed.

fi
fi
mkdir -p "${COMOUT_ATMOS_ANALYSIS_MEM}"
mkdir -p "${COM_ATMOS_ANALYSIS_MEM}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is supposed to be COMOUT_ATMOS_ANALYSIS_MEM.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@DavidGrumm-NOAA It looks like this still needs to be fixed. This caused a failure in the eupd job.

Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one small change remaining. After that, I think CI tests can be started.

DavidHuber-NOAA
DavidHuber-NOAA previously approved these changes Mar 3, 2025
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me.

@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Mar 4, 2025
@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 with error logs:

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_f5b893c3/logs/2021032500/enkfgdas_eupd.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 with error logs:

Follow link here to view the contents of the above file(s): (link)

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Mar 4, 2025
@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C96C48_hybatmDA_f5b893c3

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_hybatmaerosnowDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C96C48_hybatmaerosnowDA_f5b893c3

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C48mx500_hybAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_f5b893c3

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96_atm3DVar FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96_atm3DVar_f5b893c3/logs/2021122100/gdas_verfrad.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96_atm3DVar FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C96_atm3DVar_f5b893c3

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f5b893c3/logs/2021032500/gdas_verfrad.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_f5b893c3

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 with error logs:

/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_f5b893c3/logs/2024022400/enkfgdas_earc_tars_01.log

Follow link here to view the contents of the above file(s): (link)

@emcbot
Copy link

emcbot commented Mar 4, 2025

Experiment C96C48_ufs_hybatmDA FAILED on Hera in Build# 1 in
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/EXPDIR/C96C48_ufs_hybatmDA_f5b893c3

@DavidHuber-NOAA
Copy link
Contributor

@DavidGrumm-NOAA In addition to the eupd job failure, it looks like the verfrad job is failing. Can you take a look at the log an try to determine the issue: /scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f5b893c3/logs/2021032500/gdas_verfrad.log

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Mar 4, 2025
@emcbot
Copy link

emcbot commented Mar 4, 2025

CI Failed on Hera in Build# 1
Built and ran in directory /scratch1/NCEPDEV/global/CI/3331


Experiment C96mx100_S2S_f5b893c3 Completed 1 Cycles: *SUCCESS* at Tue Mar  4 10:29:48 UTC 2025
Experiment C48mx500_hybAOWCDA_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 10:35:57 UTC 2025
Experiment C48mx500_hybAOWCDA_f5b893c3 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_f5b893c3/logs/2021032500/enkfgdas_eupd.log
Experiment C96C48_hybatmDA_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 10:36:00 UTC 2025
Experiment C96C48_hybatmDA_f5b893c3 Terminated: *FAIL*
Experiment C96C48_hybatmaerosnowDA_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 10:36:02 UTC 2025
Experiment C96C48_hybatmaerosnowDA_f5b893c3 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96C48_hybatmDA_f5b893c3/logs/2021122100/enkfgdas_eupd.log
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96C48_hybatmaerosnowDA_f5b893c3/logs/2021122018/enkfgdas_eupd.log
Experiment C48_ATM_f5b893c3 Completed 1 Cycles: *SUCCESS* at Tue Mar  4 11:00:13 UTC 2025
Experiment C96_atm3DVar_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 11:00:15 UTC 2025
Experiment C96_atm3DVar_f5b893c3 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96_atm3DVar_f5b893c3/logs/2021122100/gdas_verfrad.log
Experiment C48mx500_3DVarAOWCDA_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 11:06:35 UTC 2025
Experiment C48mx500_3DVarAOWCDA_f5b893c3 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_f5b893c3/logs/2021032500/gdas_verfrad.log
Experiment C96C48_ufs_hybatmDA_f5b893c3 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Tue Mar  4 11:43:09 UTC 2025
Experiment C96C48_ufs_hybatmDA_f5b893c3 Terminated: *FAIL*
Error logs:
/scratch1/NCEPDEV/global/CI/3331/RUNTESTS/COMROOT/C96C48_ufs_hybatmDA_f5b893c3/logs/2024022400/enkfgdas_earc_tars_01.log
Experiment C48_S2SW_f5b893c3 Completed 1 Cycles: *SUCCESS* at Tue Mar  4 12:44:09 UTC 2025
Experiment C48_S2SWA_gefs_f5b893c3 Completed 1 Cycles: *SUCCESS* at Tue Mar  4 12:56:45 UTC 2025

@RussTreadon-NOAA
Copy link
Contributor

WCOSS2 g-w CI

Install DavidGrumm-NOAA:eomg_1 at f5b893c on Cactus. Run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 04 2025 11:50:45    Mar 04 2025 13:11:08
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 04 2025 11:50:47    Mar 04 2025 12:11:06
202103250000      Active    Mar 04 2025 11:50:47             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Mar 04 2025 11:50:49    Mar 04 2025 12:11:10
202103250000      Active    Mar 04 2025 11:50:49             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 04 2025 11:50:52    Mar 04 2025 14:05:57
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Mar 04 2025 11:50:54    Mar 04 2025 13:21:22
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 04 2025 11:50:56    Mar 04 2025 12:11:24
202112210000      Active    Mar 04 2025 11:50:56             -          
202112210600      Active    Mar 04 2025 11:50:56             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Mar 04 2025 11:50:58    Mar 04 2025 12:11:27
202112201800      Active    Mar 04 2025 11:50:58             -          
202112210000      Active    Mar 04 2025 11:50:58             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Mar 04 2025 11:51:01    Mar 04 2025 12:11:30
202112210000      Active    Mar 04 2025 11:51:01             -          
202112210600      Active    Mar 04 2025 11:51:01             -          
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Mar 04 2025 11:51:03    Mar 04 2025 12:11:34
202402240000        Done    Mar 04 2025 11:51:03    Mar 04 2025 15:11:02
202402240600        Done    Mar 04 2025 11:51:03    Mar 04 2025 15:16:17
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96mx100_S2S_pr3331
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    Mar 04 2025 11:51:05    Mar 04 2025 12:51:29

Several of the DA cases encountered failures

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3331
202103250000           gdas_analdiag                   182318550                DEAD                 -29         2         949.0
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3331
202103250000           gdas_analdiag                   182317558                DEAD                 -29         2         947.0
202103250000           enkfgdas_eupd                   182316062                DEAD                   1         2          18.0
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_pr3331
202112210000           gdas_analdiag                   182318598                DEAD                 -29         2         947.0
202112210600               gdas_anal                   182320226                DEAD                   1         2          31.0
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3331
202112201800           gdas_analdiag                   182317605                DEAD                 -29         2         920.0
202112201800           enkfgdas_eupd                   182316105                DEAD                   1         2          18.0
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3331
202112210000            gdas_verfrad                   182319663                DEAD                   1         2          69.0
202112210000           enkfgdas_eupd                   182316113                DEAD                   1         2          18.0

Comment on lines -726 to +725
lobsdiag_forenkf=${lobsdiag_forenkf},
lobsdiag_forenkf=.true.,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On inspection, this should be reverted:

Suggested change
lobsdiag_forenkf=${lobsdiag_forenkf},
lobsdiag_forenkf=.true.,
lobsdiag_forenkf=${lobsdiag_forenkf}

@@ -56,7 +56,6 @@ cao_check=${cao_check:-".true."}
ta2tb=${ta2tb:-".true."}

# Diagnostic files options
lobsdiag_forenkf=${lobsdiag_forenkf:-".false."}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here:

Suggested change
lobsdiag_forenkf=${lobsdiag_forenkf:-".false."}
lobsdiag_forenkf=${lobsdiag_forenkf:-".false."}

@@ -24,7 +21,4 @@ fi
export USE_RADSTAT="NO" # This can be only used when bias correction is non-zero.
export GENDIAG="YES" # Diagnostic files must be created for EnKF

export lobsdiag_forenkf=".true." # write out jacobians from eobs
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here as well:

Suggested change
export lobsdiag_forenkf=".true." # write out jacobians from eobs
export lobsdiag_forenkf=".true." # write out jacobians from eobs
# need to specify .true. setting since config.anal sets to .false.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@@ -25,10 +25,6 @@ if [[ ${LEVS} = "128" ]]; then
export SETUP="gpstop=55,nsig_ext=45,${SETUP:-}"
fi

# Set namelist option for LETKF
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And lastly here:

Suggested change
# Set namelist option for LETKF
# Set namelist option for LETKF
export lobsdiag_forenkf=".false." # anal does not need to write out jacobians
# set to .true. in config.eobs and config.eupd

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants