Migration to role account for Jenkins on Hercules #3423

TerrenceMcGuinness-NOAA · 2025-03-06T16:57:34Z

Description

This PR has the necessary updates to the CI scripts in Jenkins for running on Hercules with the glopara role account.

PyGithub is not in Spack Stack yet and is needed in support of the GitHub logging and gist creation python codes. This python module will be installed in $HOME/.local in the role accounts until the module is added to Spack Stack.

Same with GitHub CLI (which will be supported in Spack Stack 1.9) it will be installed in ${ROLE_HOME}/utils/bin which should be in role account's $PATH where ROLE_HOME is at the same level as fix and data.

This is setup to run in the role account with the following added directories and utils:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI

Location on disk where the CI test cases actually build and run specified in Jenkinsfile

└── GFS_CI_CD
    ├── global-workflow
    │   └── workflow
    ├── GFS_BASH_CI
    ├── GitLab
    └── Jenkins
        ├── agent
        └── workspace

Top level CI directories (this will file structure be duplicated for each role account)

GFS_CI_CD

CI system directory for Jenkins launching scripts and in support of BASH CI

GFS_CI_CD/global-workflow

Directory where the JAVA agent is launched from

Jenkins/agent

Jenkins work space (system level workspace)

Jenkins/workspace

Resolves #3378

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

This PR can be tested in place

…ation

… on Hercules role account

emcbot · 2025-03-06T19:12:12Z

Experiment C96mx100_S2S FAILED on Hercules in Build# 2 with error logs:

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-06T19:12:13Z

Experiment C48_S2SW FAILED on Hercules in Build# 2 with error logs:

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-06T19:12:14Z

Experiment C48_ATM FAILED on Hercules in Build# 2 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_ATM_bc6da134/logs/2021032312/gfs_stage_ic.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-06T19:12:21Z

Experiment C96mx100_S2S FAILED on Hercules in Build# 2 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C96mx100_S2S_bc6da134

emcbot · 2025-03-06T19:12:23Z

Experiment C48_S2SW FAILED on Hercules in Build# 2 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48_S2SW_bc6da134

emcbot · 2025-03-06T19:12:24Z

Experiment C48_ATM FAILED on Hercules in Build# 2 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48_ATM_bc6da134

emcbot · 2025-03-06T19:12:25Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 2 with error logs:

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-06T19:12:54Z

CI Failed on Hercules in Build# 2
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423


Experiment C48_ATM_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Mar  6 13:11:57 CST 2025
Experiment C48_ATM_bc6da134 Terminated: *FAIL*
Experiment C96mx100_S2S_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Mar  6 13:11:58 CST 2025
Experiment C96mx100_S2S_bc6da134 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_ATM_bc6da134/logs/2021032312/gfs_stage_ic.log
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96mx100_S2S_bc6da134/logs/1994050100/sfs_cleanup.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96mx100_S2S_bc6da134/logs/1994050100/sfs_stage_ic.log
Experiment C48_S2SW_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Mar  6 13:11:59 CST 2025
Experiment C48_S2SW_bc6da134 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SW_bc6da134/logs/2021032312/gfs_stage_ic.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SW_bc6da134/logs/2021032312/gfs_waveinit.log
Experiment C48_S2SWA_gefs_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 3 dead at Thu Mar  6 13:12:02 CST 2025
Experiment C48_S2SWA_gefs_bc6da134 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SWA_gefs_bc6da134/logs/2021032312/gefs_prep_emissions.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SWA_gefs_bc6da134/logs/2021032312/gefs_stage_ic.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SWA_gefs_bc6da134/logs/2021032312/gefs_wave_init.log
Experiment C96_atm3DVar_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Thu Mar  6 13:12:05 CST 2025
Experiment C96_atm3DVar_bc6da134 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96_atm3DVar_bc6da134/logs/2021122018/gdas_stage_ic.log
Experiment C48mx500_hybAOWCDA_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Mar  6 13:12:07 CST 2025
Experiment C48mx500_hybAOWCDA_bc6da134 Terminated: *FAIL*
Experiment C48mx500_3DVarAOWCDA_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Mar  6 13:12:07 CST 2025
Experiment C48mx500_3DVarAOWCDA_bc6da134 Terminated: *FAIL*
Experiment C96C48_hybatmDA_bc6da134 Terminated with 0
FAIL
FAIL tasks failed and 2 dead at Thu Mar  6 13:12:08 CST 2025
Experiment C96C48_hybatmDA_bc6da134 Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_bc6da134/logs/2021032418/enkfgdas_stage_ic.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_bc6da134/logs/2021032418/gdas_stage_ic.log
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_bc6da134/logs/2021032418/gdas_stage_ic.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_bc6da134/logs/2021032418/gdas_waveinit.log
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96C48_hybatmDA_bc6da134/logs/2021122018/enkfgdas_stage_ic.log
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96C48_hybatmDA_bc6da134/logs/2021122018/gdas_stage_ic.log

TerrenceMcGuinness-NOAA · 2025-03-06T20:47:35Z

Jobs failed because the role account on Hercules was not in the stmp group. Kate F. put in a ticket for it to be added.

…ess-NOAA/global-workflow into ci_jenkins_role_hercules

emcbot · 2025-03-07T19:52:13Z

Experiment C48_ATM FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_ATM_a4c413cf/logs/2021032312/gfs_arch_vrfy.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T19:52:20Z

Experiment C48_ATM FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48_ATM_a4c413cf

emcbot · 2025-03-07T20:04:35Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96C48_hybatmDA_a4c413cf/logs/2021122100/gdas_fit2obs.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T20:04:42Z

Experiment C96C48_hybatmDA FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C96C48_hybatmDA_a4c413cf

emcbot · 2025-03-07T20:10:28Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96_atm3DVar_a4c413cf/logs/2021122100/gdas_fit2obs.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T20:10:29Z

Experiment C48mx500_hybAOWCDA FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_a4c413cf/logs/2021032500/gdas_fit2obs.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T20:10:36Z

Experiment C96_atm3DVar FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C96_atm3DVar_a4c413cf

emcbot · 2025-03-07T20:10:37Z

Experiment C48mx500_hybAOWCDA FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48mx500_hybAOWCDA_a4c413cf

emcbot · 2025-03-07T20:10:48Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_a4c413cf/logs/2021032500/gdas_fit2obs.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T20:10:56Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_a4c413cf

emcbot · 2025-03-07T22:11:44Z

Experiment C48_S2SW FAILED on Hercules in Build# 4 with error logs:

/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SW_a4c413cf/logs/2021032312/gfs_arch_vrfy.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2025-03-07T22:11:51Z

Experiment C48_S2SW FAILED on Hercules in Build# 4 in
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/EXPDIR/C48_S2SW_a4c413cf

emcbot · 2025-03-07T22:12:00Z

CI Failed on Hercules in Build# 4
Built and ran in directory /work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423


Experiment C96mx100_S2S_a4c413cf Completed 1 Cycles: *SUCCESS* at Fri Mar  7 13:46:01 CST 2025
Experiment C48_ATM_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 13:51:57 CST 2025
Experiment C48_ATM_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_ATM_a4c413cf/logs/2021032312/gfs_arch_vrfy.log
Experiment C96C48_hybatmDA_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 14:04:18 CST 2025
Experiment C96C48_hybatmDA_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96C48_hybatmDA_a4c413cf/logs/2021122100/gdas_fit2obs.log
Experiment C96_atm3DVar_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 14:10:17 CST 2025
Experiment C96_atm3DVar_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C96_atm3DVar_a4c413cf/logs/2021122100/gdas_fit2obs.log
Experiment C48mx500_hybAOWCDA_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 14:10:20 CST 2025
Experiment C48mx500_hybAOWCDA_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_hybAOWCDA_a4c413cf/logs/2021032500/gdas_fit2obs.log
Experiment C48mx500_3DVarAOWCDA_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 14:10:31 CST 2025
Experiment C48mx500_3DVarAOWCDA_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_a4c413cf/logs/2021032500/gdas_fit2obs.log
Experiment C48_S2SWA_gefs_a4c413cf Completed 1 Cycles: *SUCCESS* at Fri Mar  7 15:53:31 CST 2025
Experiment C48_S2SW_a4c413cf Terminated with 0
FAIL
FAIL tasks failed and 1 dead at Fri Mar  7 16:11:33 CST 2025
Experiment C48_S2SW_a4c413cf Terminated: *FAIL*
Error logs:
/work2/noaa/global/role-global/GFS_CI_CD/HERCULES/CI/3423/RUNTESTS/COMROOT/C48_S2SW_a4c413cf/logs/2021032312/gfs_arch_vrfy.log

Terry McGUinness added 4 commits February 22, 2025 01:45

fixed misspelling of launch

89d0101

updated hercules for role account and biefed up the internal document…

aa68c2b

…ation

updated path to custome working directory for running CI with Jenkins…

8cbb38c

… on Hercules role account

fixed more misspellings of lauch

bc6da13

TerrenceMcGuinness-NOAA added the CI/CD Issue related to CI/CD label Mar 6, 2025

TerrenceMcGuinness-NOAA self-assigned this Mar 6, 2025

TerrenceMcGuinness-NOAA requested a review from WalterKolczynski-NOAA as a code owner March 6, 2025 16:57

TerrenceMcGuinness-NOAA requested review from KateFriedman-NOAA, DavidHuber-NOAA and aerorahul March 6, 2025 16:58

TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Mar 6, 2025

TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Mar 6, 2025

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Mar 6, 2025

TerrenceMcGuinness-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules labels Mar 6, 2025

Terry McGuinness and others added 3 commits March 7, 2025 11:49

moving PTMP to global for now until stmp is added to the role account

dad7ff3

add echo for JENKINS_WORK_DIR being used

0c8a2a2

Merge branch 'ci_jenkins_role_hercules' of github.com:TerrenceMcGuinn…

a4c413c

…ess-NOAA/global-workflow into ci_jenkins_role_hercules

TerrenceMcGuinness-NOAA added the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Mar 7, 2025

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels Mar 7, 2025

emcbot added CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed and removed CI-Hercules-Failed **Bot use only** CI testing on Hercules for this PR has failed labels Mar 7, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migration to role account for Jenkins on Hercules #3423

Migration to role account for Jenkins on Hercules #3423

TerrenceMcGuinness-NOAA commented Mar 6, 2025 •

edited

Loading

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

TerrenceMcGuinness-NOAA commented Mar 6, 2025 •

edited

Loading

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

Migration to role account for Jenkins on Hercules #3423

Are you sure you want to change the base?

Migration to role account for Jenkins on Hercules #3423

Conversation

TerrenceMcGuinness-NOAA commented Mar 6, 2025 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

emcbot commented Mar 6, 2025

TerrenceMcGuinness-NOAA commented Mar 6, 2025 • edited Loading

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

emcbot commented Mar 7, 2025

TerrenceMcGuinness-NOAA commented Mar 6, 2025 •

edited

Loading

TerrenceMcGuinness-NOAA commented Mar 6, 2025 •

edited

Loading