Add IFI to regression tests (and put regression tests in repository)

[SamuelTrahanNOAA]

This adds four regression tests to the UPP test suite related to the closed-source library IFI, which is generally not accessible to users. These tests are enabled by the presence of a sorc/libIFI.fd/src directory in the UPP clone.

• post_hrrr_ifi = generate all IFI fields for HRRR input
• post_fv3r_ifi = generate all IFI fields for FV3R (RRFS-like) input
• ifi_standalone_fv3r = run the libIFI standalone program and compare output to post_fv3r_ifi
• ifi_standalone_hrrr = run the libIFI standalone program and compare output to post_hrrr_ifi
• fv3r_ifi_missing = generate IFI fields when libIFI is missing, thus writing out a masked-out grid

Presently, the ifi_standalone_hrrr test is disabled because the output doesn't match post_hrrr_ifi.

In pursuant to that, this PR also:

1. Adds the regression test jobs-dev scripts to the repository and updates rt.sh to use that copy.
2. Adds a post config file that only generates IFI field, and updates the raphrrr available fields to know about IFI.
3. Adds a "-o name.x" option to compile_upp.sh to set the name of the upp.x executable.
4. Adds a "-a" option to compile_upp.sh to skip deletion of the exec directory (a = add to exec)
5. When BUILD_IFI_EXECUTABLES is enabled in cmake, you'll get relevant messages.

Also, emacs autosave files *~ are added back to the .gitignore. That gitignore line was removed, likely by accident, in prior PRs. If you'd prefer me to take that change out, please let me know and I'll open another PR for it.

• fixes UPP regression test scripts should be in the UPP repository #1117
• fixes Add IFI regression tests to UPP rt.sh #1111
• fixes .gitignore should ignore emacs autosave files (*~) #1120
• fixes BUILD_IFI_EXECUTABLES option needs more cmake messages #1122
• fixes Regression tests for IFI fields when libIFI is absent #1123
• fixes UPP rt.sh doesn't detect when a test is unable to run #1137

Documentation Required

Users should know how to build IFI's executables via compile_upp.sh. It's the -B option.

• IFI access documention should explain how to compile IFI executables #1124

Blocking Dependencies

Presently, only epic group members can run rt.sh on MSU HPC platforms due to permission problems.

• UPP regression test data permission problems on MSU HPC clusters (Hercules, Orion) #1118

[SamuelTrahanNOAA]

Before this is merged, I'd like other people to confirm rt.sh works for them on Hera, Hercules, and Orion.

Presently, it won't run for non-epic users, except on Hera, due to permission problems discussed in

• UPP regression test data permission problems on MSU HPC clusters (Hercules, Orion) #1118

[gspetro-NOAA]

@SamuelTrahanNOAA I just attempted to run on Hercules and Orion, and I'm getting an error in nohup.out:

+ export rundir=/work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir/upp-HERCULES
+ rundir=/work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir/upp-HERCULES
+ test -d /work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir/upp-HERCULES
+ mkdir /work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir/upp-HERCULES
mkdir: cannot create directory ‘/work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir/upp-HERCULES’: No such file or directory

[SamuelTrahanNOAA]

That error means this directory doesn't exist:

/work/noaa/epic/gpetro/hercules/RTs/upp-rts/1119a/ci/rundir

The mkdir should be mkdir -p so it makes all ancestor directories.

I updated rt.sh. Please try again.

[SamuelTrahanNOAA]

@gspetro-NOAA -

Could you remind me where your test suite locations are?

You can find the locations in ci/rt.sh. These are the paths:

 export machine=HERA
 export homedir=${homedir:-"/scratch2/BMC/wrfruc/Samuel.Trahan/UPP/test_suite"}
...
 export machine=ORION
 export homedir=${homedir:-"/work2/noaa/wrfruc/strahan/upp-ifi-regtest/upp-data"}
...
 export machine=HERCULES
 export homedir=${homedir:-"/work2/noaa/wrfruc/strahan/upp-ifi-regtest/upp-data"}

I put Hercules and Orion in the same directory because that's what epic did in their project area.

And which files need to be moved? Or do you think it's easier just to copy everything, since it won't matter for the unchanged files.

The simplest way:

1. Copy all data_out/*ifi*/ directories.
2. Copy the scripts/runtime.log.$machine file(s).

I don't expect any other tests to change output.

[WenMeng-NOAA]

@gspetro-NOAA I have no comments on this PR. We may start the final testing after you sync baselines from Sam's account to the EPIC's account.

[gspetro-NOAA]

I have updated baselines on all systems, and the RTs pass on Hera and Orion with no changes (from the new baselines). I will test on Hercules once it's back up. Then I'll be able to approve.

[WenMeng-NOAA]

I have updated baselines on all systems, and the RTs pass on Hera and Orion with no changes (from the new baselines). I will test on Hercules once it's back up. Then I'll be able to approve.

@gspetro-NOAA Great!
@SamuelTrahanNOAA Could you update the paths of 'test_suite' in rt.sh in your PR to point to the EPIC locations? We will start the final round of testing.

[SamuelTrahanNOAA]

@WenMeng-NOAA @gspetro-NOAA - I reverted the changes to test_suite paths. They now point to whatever they did before my commits.

Could you update the paths of 'test_suite' in rt.sh in your PR to point to the EPIC locations? We will start the final round of testing.

[WenMeng-NOAA]

The testing with ifi code involved on Hera: ./rt.sh -a ovp

Job cards submitted for enabled tests, waiting on timestamps for finished jobs...
00:03:50   hrrr_ifi_test 00:05:00
00:02:56   fv3r_ifi_test 00:05:00
00:02:47   ifi_standalone_fv3r_test 00:05:00
00:01:26   nmmb_test 00:01:00
00:01:23   nmmb_pe_test 00:01:00
00:00:17   fv3gefs_test 00:40:00
00:00:23   fv3gefs_pe_test 00:40:00
00:01:06   rap_test 00:02:00
00:01:17   rap_pe_test 00:02:00
00:02:24   hrrr_test 00:02:00
00:02:17   hrrr_pe_test 00:02:00
00:06:58   fv3gfs_test 00:15:00
00:07:09   fv3gfs_pe_test 00:15:00
00:01:53   fv3r_test 00:03:00
00:01:58   fv3r_pe_test 00:03:00
00:00:18   fv3r_ifi_missing 00:03:00
00:00:37   fv3hafs_test 00:03:00
00:00:36   fv3hafs_pe_test 00:03:00
00:01:53   rtma_test 00:03:00
00:01:58   rtma_pe_test 00:03:00
No changes in test results detected.

The results look good to me.

[SamuelTrahanNOAA]

This is a minor thing, but I noticed some runtimes are longer than the expected runtime. It's only a small amount, but perhaps the expected runtimes should be updated eventually.

[WenMeng-NOAA]

The testing without ifi code involved on Hera: ./rt.sh -a ovp -d

Job cards submitted for enabled tests, waiting on timestamps for finished jobs...
00:01:09   nmmb_test 00:01:00
00:01:21   nmmb_pe_test 00:01:00
00:00:27   fv3gefs_test 00:40:00
00:00:27   fv3gefs_pe_test 00:40:00
00:01:04   rap_test 00:02:00
00:01:21   rap_pe_test 00:02:00
00:02:32   hrrr_test 00:02:00
00:02:05   hrrr_pe_test 00:02:00
00:07:10   fv3gfs_test 00:15:00
00:07:43   fv3gfs_pe_test 00:15:00
00:01:49   fv3r_test 00:03:00
00:01:47   fv3r_pe_test 00:03:00
00:00:31   fv3r_ifi_missing 00:03:00
00:00:36   fv3hafs_test 00:03:00
00:00:36   fv3hafs_pe_test 00:03:00
00:01:50   rtma_test 00:03:00
00:01:49   rtma_pe_test 00:03:00
No changes in test results detected.

The results look good to me.

[WenMeng-NOAA]

This is a minor thing, but I noticed some runtimes are longer than the expected runtime. It's only a small amount, but perhaps the expected runtimes should be updated eventually.

@SamuelTrahanNOAA We could address that in the future.

[ChristopherHill-NOAA]

My regression test results on Hera using the latest updates to ci/rt.sh and without using IFI code -- complementing results from @WenMeng-NOAA

00:01:26   nmmb_test 00:01:00
00:01:23   nmmb_pe_test 00:01:00
00:00:22   fv3gefs_test 00:40:00
00:00:23   fv3gefs_pe_test 00:40:00
00:01:03   rap_test 00:02:00
00:01:14   rap_pe_test 00:02:00
00:02:23   hrrr_test 00:02:00
00:01:56   hrrr_pe_test 00:02:00
00:06:29   fv3gfs_test 00:15:00
00:07:13   fv3gfs_pe_test 00:15:00
00:01:41   fv3r_test 00:03:00
00:01:42   fv3r_pe_test 00:03:00
00:00:22   fv3r_ifi_missing 00:03:00
00:00:36   fv3hafs_test 00:03:00
00:00:32   fv3hafs_pe_test 00:03:00
00:01:54   rtma_test 00:03:00
00:01:42   rtma_pe_test 00:03:00
No changes in test results detected.

[gspetro-NOAA]

This is a minor thing, but I noticed some runtimes are longer than the expected runtime. It's only a small amount, but perhaps the expected runtimes should be updated eventually.

I have wondered about this, too. On Orion/Hercules, in particular, the hrrr_test and hrrr_pe_test always run significantly longer than baseline time and have since the beginning of the log files' existence. I will create an issue.

[WenMeng-NOAA]

The UPP RTs have been completed on WCOSS2 with no baseline changes.

[WenMeng-NOAA]

This PR is ready for merging.