build_all.sh fails when gdas.cd is present #1043

RussTreadon-NOAA · 2022-10-04T17:46:56Z

Expected behavior
./build_all.sh on Orion from develop @ 833b8f4 should run to completion when sorc/gdas.cd is present.

Current behavior
./build_all.sh fails on Orion when it gets to gdas.cd.

Orion-login-3[23] rtreadon$ ./build_all.sh -v
INFO: settings in config file: gfs_build.cfg
Building ufs_model (ufs_model) ........................ yes
Building ww3_prepost (ww3_prepost) .................... yes
Building gsi_enkf (gsi_enkf) .......................... yes
Building gsi_utils (gsi_utils) ........................ yes
Building gsi_monitor (gsi_monitor) .................... yes
Building gldas (gldas) ................................ yes
Building UPP (upp) .................................... yes
Building ufs_utils (ufs_utils) ........................ yes
Building gfs_wafs (gfs_wafs) .......................... yes
Building gfs_utils (gfs_utils) ........................ yes
INFO: partial build settings:
  Build_ufs_model: True
  Build_ww3_prepost: True
  Build_gsi_enkf: True
  Build_gsi_utils: True
  Build_gsi_monitor: True
  Build_ww3_prepost: True
  Build_reg2grb2: True
  Build_gldas: True
  Build_upp: True
  Build_ufs_utils: True
  Build_gfs_wafs: True
  Build_gfs_utils: True
 .... Building WW3 pre and post execs ....
 .... Building forecast model ....
 .... Building gsi and enkf ....
 .... Building gsi utilities ....
./build_all.sh: line 170: Build_gdas: parameter null or not set
Orion-login-3[24] rtreadon$

Machines affected
Only tested on Orion.

To Reproduce

clone g-w develop at 833b8f4
cd sorc
I did checkout.sh -g -u to clone both DA engines for comparative studies, but checkout.sh -u is sufficient to see the failure.
./build_all.sh

Possible Implementation
As a test,

add Build_gdas to Build_prg list in partial_build.sh
add Building gdas (gdas) .................................. yes to gfs_build.cfg
./build_all.sh runs to completion

Since ./checkout.sh supports both GSI (-g) and UFS (-u) DA builds, should partial_build.sh and gfs_build.cfg include both gsi_enkf and gdas?

The text was updated successfully, but these errors were encountered:

WalterKolczynski-NOAA · 2022-10-04T18:08:27Z

I've got this bug fixed, but in trying to confirm the fix I get another error during the GDASApp build:

-- Updating crtm to TAG v2.3-jedi.3...
-- git fetch --all @ /work2/noaa/global/wkolczyn/save/global-workflow/gdas_build/sorc/gdas.cd/crtm
-- git fetch --all --tags @ /work2/noaa/global/wkolczyn/save/global-workflow/gdas_build/sorc/gdas.cd/crtm
-- git checkout v2.3-jedi.3 @ /work2/noaa/global/wkolczyn/save/global-workflow/gdas_build/sorc/gdas.cd/crtm
-- ---------------------------------------------------------
-- �[32m[crtm] (2.3.0) [5c52ad0]�[m
-- Feature TESTS enabled
CMake Error at build/jedicmake/jedicmake-config.cmake:19 (message):
  File or directory
  /work2/noaa/global/wkolczyn/save/global-workflow/gdas_build/sorc/gdas.cd/jedicmake/CMakeModules/Modules
  referenced by variable jedicmake_MODULES does not exist !
Call Stack (most recent call first):
  build/jedicmake/jedicmake-config.cmake:36 (set_and_check)
  crtm/CMakeLists.txt:20 (find_package)


-- Configuring incomplete, errors occurred!
See also "/work2/noaa/global/wkolczyn/save/global-workflow/gdas_build/sorc/gdas.cd/build/CMakeFiles/CMakeOutput.log".

RussTreadon-NOAA · 2022-10-04T18:19:41Z

The GDASApp hash in checkout.sh no longer works due to changes the JEDI core team made to jedicmake. The current head of GDASApp develop, 1ef214b, get us past the jedicmake error. Replace

checkout "gdas.cd" "https://github.com/NOAA-EMC/GDASApp.git" "5952c9d"; errs=$((errs + $?))

with

checkout "gdas.cd" "https://github.com/NOAA-EMC/GDASApp.git" "1ef214b"; errs=$((errs + $?))

This alone, is not sufficient, for GDASApp to build. The ioda-converters in GDASApp require bufr/11.7.1. hpc-stack issue #489 has been opened with this request. This request now appears to be wrapped up in spack-stack issue #339.

Until bufr/11.7.1 is installed we can revert back to an older ioda-converters hash. To check out the older hash, make the following change to sorc/gdas.cd/CMakeLists.txt

   if(BUILD_IODA_CONVERTERS)
-    ecbuild_bundle( PROJECT iodaconv GIT "https://github.com/JCSDA-internal/ioda-converters.git" BRANCH develop UPDATE )
+    ecbuild_bundle( PROJECT iodaconv GIT "https://github.com/JCSDA-internal/ioda-converters.git" BRANCH a29e1247 )
   endif()

CoryMartin-NOAA · 2022-10-04T19:08:47Z

Does GDASApp no longer build using the hash checked out as part of the develop branch of g-w? EDIT: never mind, it is the case because it checks out develop of jedicmake... ugh

RussTreadon-NOAA · 2022-10-04T19:22:00Z

No, the build fails as @WalterKolczynski-NOAA documented. This is the jedicmake problem. We fixed this in GDASApp develop and so the GDASApp hash in checkout.sh needs to be updated. While updating the GDASApp develop hash will get us through the configure part of the GDASApp build, the compilation will still fail. The ioda-converters extracted with the GDASApp require bufr/11.7.1. This is not yet available. If we use an older hash for ioda-converters, we can build GDASApp from g-w.

WalterKolczynski-NOAA · 2022-10-04T19:29:51Z

The GDASApp hash in checkout.sh no longer works due to changes the JEDI core team made to jedicmake. The current head of GDASApp develop, 1ef214b, get us past the jedicmake error. Replace

checkout "gdas.cd" "https://github.com/NOAA-EMC/GDASApp.git" "5952c9d"; errs=$((errs + $?))

with

checkout "gdas.cd" "https://github.com/NOAA-EMC/GDASApp.git" "1ef214b"; errs=$((errs + $?))

This alone, is not sufficient, for GDASApp to build. The ioda-converters in GDASApp require bufr/11.7.1. hpc-stack issue #489 has been opened with this request. This request now appears to be wrapped up in spack-stack issue #339.

Until bufr/11.7.1 is installed we can revert back to an older ioda-converters hash. To check out the older hash, make the following change to sorc/gdas.cd/CMakeLists.txt
   if(BUILD_IODA_CONVERTERS)
-    ecbuild_bundle( PROJECT iodaconv GIT "https://github.com/JCSDA-internal/ioda-converters.git" BRANCH develop UPDATE )
+    ecbuild_bundle( PROJECT iodaconv GIT "https://github.com/JCSDA-internal/ioda-converters.git" BRANCH a29e1247 )
   endif()

I'm going to submit the fix for the bug that is ours, since I'm confident I have fixed that. I dislike that GDAS is using non-static versions of components by checking out develop instead of specific commits. It makes it impossible to guarantee that any specific version at our level will work in the same way (or at all, as seen here). We have a policy against using dynamic version pointers in workflow, and would hope that constituent components also follow them. It can be relaxed since GDAS is still in a preliminary stage, but as soon as people start really using it, this will be a problem.

CoryMartin-NOAA · 2022-10-04T19:30:57Z

@WalterKolczynski-NOAA agreed, NOAA-EMC/GDASApp#143 is already created. Soon we will switch to checking out specific tags/commits of JEDI repos.

When the GDAS app was added to the workflow, the corresponding build setting was not added to partial_build and the build configuration file. Refs NOAA-EMC#1043

When the GDAS app was added to the workflow, the corresponding build setting was not added to partial_build and the build configuration file. This means that after `build_all.sh` was updated to correct syntax issues, the build would fail because `$Build_gdas` was undefined. Note: the GDAS app still does not build currently due to unrelated problems within the gdas repo. Refs #1043

RussTreadon-NOAA · 2022-10-23T19:08:50Z

g-w issue #1067 updates the GDASApp hash. g-w successfully builds the GDASApp (gdas.cd) with the updated GDASApp hash.

Updates the GDASapp version to a newer version that builds correctly. The former version no longer builds because submodules point to develop branches that are no longer compatible. Moves module loads out of the j-jobs and into their appropriate place in the rocoto job. A new load module script is added to handle the different module set needed for UFSDA. Also temporarily turns off strict mode for the UFSDA jobs to avoid PS1 unbound error in conda. Fixes #1043 Fixes #1067

RussTreadon-NOAA added the bug Something isn't working label Oct 4, 2022

WalterKolczynski-NOAA self-assigned this Oct 4, 2022

WalterKolczynski-NOAA mentioned this issue Oct 4, 2022

Add GDAS to the partial build list #1050

Merged

7 tasks

WalterKolczynski-NOAA closed this as completed Oct 11, 2022

WalterKolczynski-NOAA reopened this Oct 11, 2022

RussTreadon-NOAA mentioned this issue Oct 24, 2022

Feature/updates for gdas app #1091

Merged

8 tasks

WalterKolczynski-NOAA closed this as completed in #1091 Oct 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build_all.sh fails when gdas.cd is present #1043

build_all.sh fails when gdas.cd is present #1043

RussTreadon-NOAA commented Oct 4, 2022

WalterKolczynski-NOAA commented Oct 4, 2022

RussTreadon-NOAA commented Oct 4, 2022

CoryMartin-NOAA commented Oct 4, 2022 •

edited

Loading

RussTreadon-NOAA commented Oct 4, 2022

WalterKolczynski-NOAA commented Oct 4, 2022

CoryMartin-NOAA commented Oct 4, 2022

RussTreadon-NOAA commented Oct 23, 2022

build_all.sh fails when gdas.cd is present #1043

build_all.sh fails when gdas.cd is present #1043

Comments

RussTreadon-NOAA commented Oct 4, 2022

WalterKolczynski-NOAA commented Oct 4, 2022

RussTreadon-NOAA commented Oct 4, 2022

CoryMartin-NOAA commented Oct 4, 2022 • edited Loading

RussTreadon-NOAA commented Oct 4, 2022

WalterKolczynski-NOAA commented Oct 4, 2022

CoryMartin-NOAA commented Oct 4, 2022

RussTreadon-NOAA commented Oct 23, 2022

CoryMartin-NOAA commented Oct 4, 2022 •

edited

Loading