PFS tests should always save timing

[jgfouca]

Test suite: pre-commit
Test baseline:
Test namelist changes:
Test status: [bit for bit, roundoff, climate changing]

Fixes [CIME Github issue #]

User interface changes?:

Update gh-pages html (Y/N)?:

[jgfouca]

I couldn't think of any reason why you'd ever want to run a PFS test and not save the timings.

[jedwards4b]

I can actually - I frequently use the PFS test for load balancing and don't want to save or compare to a baseline.

[jgfouca]

@jedwards4b , ah ok. Should I abort this PR?

[jedwards4b]

I'm not sure - if I create a PFS test and don't specify a baseline directory with --generate what happens?

[rljacob]

I'm looking at pfs.py and can't tell what its doing. Are the actual test mechanics somewhere else in CIME?

[jedwards4b]

That's the place to look - all it does is run a case for 20 days. It saves the timing files and software_environment.txt file for each run. It can save them to a baseline directory but most often I use it as a standalone tool to load balance configurations.

[jedwards4b]

I should add that it turns off history and restart to minimize IO for the run.

[jgfouca]

I think SAVE_TIMING should work fine with or without baselines. @rljacob , the mechanics of the test are very simple, basically the same as SMS, so the only interesting stuff is in CIME/data/config/config_tests.xml

[rljacob]

Yes SAVE_TIMING just saves the timing info to another directory. It doesn't reference any other runs.

[rljacob]

This is off topic but it seems like turning off history and setting the run length to 20 days should be in pfs.py. Same for other tests. Why even have CIME/data/config/config_tests.xml ?

[jgfouca]

@rljacob , test parameters are kept separate from test python mechanics. I think this approach makes sense since create_test parses the test options and handles the XML creation before the test-specific python code is run.

[billsacks]

Does this address #2918 or is this somewhat separate from what I was asking there?

Regarding the tangential discussion of config_tests.xml: I have to say that I agree with @rljacob here: I have always found it confusing that I need to look in two places to see the full picture of how a test is defined. Maybe there are a handful of things that truly can't be handled in the python, but it seems like most of what we do in config_tests.xml could be done with a case.set_value early in the test, and then we could have everything in one place.

[rljacob]

I happened to be looking at ert.py which DOES use case.set_value and thought it was easy to follow. Then I went to pfs.py and wondered how it was set to do 20 days.

[jgfouca]

@billsacks , separate, but we (E3SM) are interesting in the topic you raised in that ticket.

[rljacob]

How does the PFS test determine pass/fail without a baseline?

[jedwards4b]

It does not - you need a baseline.
I use PFS mostly for loadbalancing fully coupled configurations.

[rljacob]

What does your workflow look like when you're loadbalancing? I would build PFS once and then go to the test casedir and do cycles of edit pelayout-submit-look at timings.

[jedwards4b]

Yes, that's exactly it and the env_mach_pes.xml and software_environment.txt are saved for each run.