package fit2obs and install it #12

aerorahul · 2023-04-11T20:36:38Z

This PR:

enables installing the package and loading it via a modulefile. The modulefile defines standard package variables such as HOMEfit2obs, USHfit2obsetc. See themodulefiles/fit2obs.lua.tmpl` for details.
Moves files excfs.....sh and global_hyblevs.l65.txt to scripts/ and fix/ directories to conform w/ NCO EE2 standards
Creates a README.md that describes installation instructions for the package. This may need update from @jack-woollen to accurately describe what this repository does.
Creates a VERSION file. The version number in here should be incremented with every release. This is for managing releases of this project.
Adds an option BUILD_LEGACY=YES for building and installing executables in the project cloned space.
moves contents of batrun to ush. batrun/runfits is moved to ush/runfits.sh
removes contents and directory runs following suggestions from @jack-woollen
removes batrun/subfits_dell_nems as the WCOSS Dell machine has been decomissioned

Resolves #10

jack-woollen

@aerorahul @KateFriedman-NOAA I see where you're going with this and definitely approve. There's a couple loose ends which I listed some thoughts on below.

runfits has been the entry point for running the fits software. I would promote it to scripts/runfits.sh, and keep its function the same, including propagating the fit2obs module paths into the subsequent scripts as is done now.
keep the batrun directory to contain subfits files for various machines as examples of how jobs would be submitted on each. They would be stand-alone in the sense that users would fill in various parameters in a subfits file, and then run a batch job by invoking the modified subfits script with a date argument.
probably remove the runs directory from the repo since the new subfits files will not need drivers to function.

I/we could make changes in this branch before merging it to develop, or I could go ahead and approve merging as is, and raise another issue and make another PR to tie up the loose ends. Let me know your feedback on this question. Thanks!

aerorahul · 2023-04-12T13:30:39Z

@jack-woollen
Thank you for the review.
I can work on your suggestions in this branch. May I suggest/add to yours?

move batrun/runfits -> ush/runfits.sh
move batrun/ -> ush/batrun/
remove runs

There is use of cfs in variable names e.g. HOMEcfs etc. Is this appropriate? If we are to follow EE2 naming conventions, these would be HOMEfit2obs etc. This can be addressed in another PR.

There is also a lot of repetition in the various batrun/subfits_$MACHINE that can be consolidated and only the platform (mostly scheduler) specifics be retained in the separate scripts. I have no preference since we will not be using these scripts to submit the jobs in the workflow. This is for another day, another PR.

I would wait for @KateFriedman-NOAA to approve and confirm this branch works as her expectations in the global-workflow.

Thoughts?

jack-woollen

@aerorahul in ush/CMakeLists.txt it should have runfits.sh (maybe cfs_runfits.sh) but maybe not run_fits.sh. Consistent with the runfits script name in the ush directory in any case.

aerorahul · 2023-04-12T16:02:07Z

@aerorahul in ush/CMakeLists.txt it should have runfits.sh (maybe cfs_runfits.sh) but maybe not run_fits.sh. Consistent with the runfits script name in the ush directory in any case.

apologies @jack-woollen That was a typo on my part. I have fixed the name of the script to be consistent w/ what is in ush.

KateFriedman-NOAA

Changes look good. I will build this branch and run it via global-workflow on Orion (likely tomorrow) to confirm it works for us. Stay tuned.

KateFriedman-NOAA · 2023-04-17T14:48:49Z

I made the needed updates on the global-workflow side to use this Fit2Obs PR branch and installed a copy of feature/module branch on Orion (outside of my clone copy of it). I then reran a gdasfit2obs job in the test I used for the prior updates. It worked without issue and reproduced output from the prior test.

Log: /work/noaa/stmp/kfriedma/comrot/testfit2obs/logs/2022011000/gdasfit2obs.log

Here are the few changes I had to make on the workflow side:

Orion-login-3[60] /work/noaa/global/kfriedma/git/feature-fit2obs/sorc$ git diff
diff --git a/jobs/JGDAS_FIT2OBS b/jobs/JGDAS_FIT2OBS
index f6a1b363..d6738454 100755
--- a/jobs/JGDAS_FIT2OBS
+++ b/jobs/JGDAS_FIT2OBS
@@ -58,7 +58,7 @@ if [[ ${CDATE} -gt ${SDATE} ]]; then
   # RUN FIT2OBS VERIFICATION
   ##############################################

-  "${fitdir}/batrun/excfs_gdas_vrfyfits.sh.ecf"
+  "${SCRIPTSfit2obs}/excfs_gdas_vrfyfits.sh"
   status=$?
   [[ ${status} -ne 0 ]] && exit "${status}"

diff --git a/modulefiles/module_base.orion.lua b/modulefiles/module_base.orion.lua
index 19897aaf..77a7486d 100644
--- a/modulefiles/module_base.orion.lua
+++ b/modulefiles/module_base.orion.lua
@@ -30,6 +30,9 @@ setenv("WGRIB2","wgrib2")
 prepend_path("MODULEPATH", pathJoin("/work/noaa/global/glopara/git/prepobs/v1.0.1/modulefiles"))
 load(pathJoin("prepobs", "1.0.1"))

+prepend_path("MODULEPATH", pathJoin("/work/noaa/global/glopara/git/Fit2Obs/test/modulefiles"))
+load(pathJoin("fit2obs", "1.0.0"))
+
 -- Temporary until official hpc-stack is updated
 prepend_path("MODULEPATH", "/work2/noaa/global/wkolczyn/save/hpc-stack/modulefiles/stack")
 load(pathJoin("hpc", "1.2.0"))
diff --git a/parm/config/config.fit2obs b/parm/config/config.fit2obs
index 9a904e2b..46baaa9e 100644
--- a/parm/config/config.fit2obs
+++ b/parm/config/config.fit2obs
@@ -8,13 +8,6 @@ echo "BEGIN: config.fit2obs"
 # Get task specific resources
 . "${EXPDIR}/config.resources" fit2obs

-export fit_ver="wflow.1.0"
-export fitdir="${BASE_GIT}/Fit2Obs/${fit_ver}"
-
-export HOMEcfs=${fitdir}
-export EXECcfs=${HOMEcfs}/exec
-export USHcfs=${HOMEcfs}/ush
-
 export PRVT=${HOMEgfs}/fix/gsi/prepobs_errtable.global
 export HYBLEVS=${HOMEgfs}/fix/am/global_hyblev.l${LEVS}.txt

Note, I set test for the version of this install in the module_base file. I will change this to match the version in the next Fit2Obs tag cut after this PR and after I install that tag everywhere. I will also update the other workflow module_base files with a similar module update.

Based on this successful testing this PR is good to merge and then tag.

aerorahul · 2023-04-17T15:54:37Z

Thanks for testing @KateFriedman-NOAA
@jack-woollen we await merge and tag.
Please note that I have labeled this as 1.0.0 in the VERSION file. If you need me to update that to something else, please let me know.

Please use the version in your tag e.g. v1.0.0 as the tag name.

aerorahul · 2023-04-18T14:41:10Z

Gentle reminder @jack-woollen

jack-woollen · 2023-04-18T15:55:29Z

If no objections The tag will be wflow.v1.0.0

aerorahul · 2023-04-18T15:57:24Z

@jack-woollen
In the future, when you make updates to this repository, please open a PR and add @KateFriedman-NOAA so she can make necessary updates in the global-workflow.

jack-woollen · 2023-04-18T16:03:35Z

Tagged and pushed. Will do on the update procedure.

KateFriedman-NOAA · 2023-04-19T15:47:55Z

Thanks @jack-woollen and @aerorahul !

package fit2obs and install it

a63cfa9

aerorahul requested review from KateFriedman-NOAA and jack-woollen April 11, 2023 20:36

jack-woollen reviewed Apr 12, 2023

View reviewed changes

update following suggestions from @jack-wollen

cb5cd0b

jack-woollen reviewed Apr 12, 2023

View reviewed changes

fix typo in ush/CMakeLists.txt

2f6c45d

also set HOMEcfs in the modulefile till the scripts get updated

b43b47f

KateFriedman-NOAA reviewed Apr 13, 2023

View reviewed changes

KateFriedman-NOAA approved these changes Apr 17, 2023

View reviewed changes

KateFriedman-NOAA mentioned this pull request Apr 17, 2023

Fit2Obs reorganization and add module NOAA-EMC/global-workflow#1472

Closed

3 tasks

jack-woollen merged commit 74b24dd into develop Apr 18, 2023

aerorahul deleted the feature/module branch April 18, 2023 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

package fit2obs and install it #12

package fit2obs and install it #12

aerorahul commented Apr 11, 2023 •

edited by KateFriedman-NOAA

Loading

jack-woollen left a comment

aerorahul commented Apr 12, 2023 •

edited

Loading

jack-woollen left a comment

aerorahul commented Apr 12, 2023

KateFriedman-NOAA left a comment

KateFriedman-NOAA commented Apr 17, 2023

aerorahul commented Apr 17, 2023

aerorahul commented Apr 18, 2023

jack-woollen commented Apr 18, 2023

aerorahul commented Apr 18, 2023

jack-woollen commented Apr 18, 2023

KateFriedman-NOAA commented Apr 19, 2023

package fit2obs and install it #12

package fit2obs and install it #12

Conversation

aerorahul commented Apr 11, 2023 • edited by KateFriedman-NOAA Loading

jack-woollen left a comment

Choose a reason for hiding this comment

aerorahul commented Apr 12, 2023 • edited Loading

jack-woollen left a comment

Choose a reason for hiding this comment

aerorahul commented Apr 12, 2023

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

KateFriedman-NOAA commented Apr 17, 2023

aerorahul commented Apr 17, 2023

aerorahul commented Apr 18, 2023

jack-woollen commented Apr 18, 2023

aerorahul commented Apr 18, 2023

jack-woollen commented Apr 18, 2023

KateFriedman-NOAA commented Apr 19, 2023

aerorahul commented Apr 11, 2023 •

edited by KateFriedman-NOAA

Loading

aerorahul commented Apr 12, 2023 •

edited

Loading