Updates in helper script for updated cupid workflow #176

TeaganKing · 2025-01-30T19:18:15Z

Description of changes: This PR will include additional parameters in the helper script which generates the cupid configuration file to account for a more flexible cupid workflow.

Have you followed the guidelines in our Contributor's Guide?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?
Have you made sure that the pre-commit checks passed (#8 in Adding Notebooks Guide)?
Have you added an explanation of what your changes do and why you'd like us to include them?
Have you successfully tested your changes locally?

mnlevy1981 · 2025-01-30T22:41:55Z

I'll be adding cime_config/config_tool.xml, which can be used by CIME to generate env_postprocessing.xml in a CESM case directory. @ekluzek suggested we have preview_namelists generate the config.yml file used by CUPiD in the CESM workflow in RUN_ANALYSIS=TRUE. I don't know what that looks like, but I wanted to leave a comment here to remember to look into it later (it would mean moving some of the setup from ccs_config's template.cupid to a different script that could be invoked by preview_namelists; I think that script would also be in cime_config/).

…criptions from xml vars

cime_config/config_tool.xml is used by CESM to create env_postprocessing.xml. Variables in this XML file are then used to generate the cupid config file for CESM cases

1. Bring in latest ADF commit as external 2. generate_cupid_config_for_cesm_case.py now has --adf-output-root argument that lets users specify a place other than ${CASEROOT} for ADF_output/ Also cleaned up comments about start_years and end_years in timeseries.py

Keeping with CESM convention, using STARTDATE instead of START_YEAR. This propogates into generate_cupid_config_for_cesm_case.py, which wants to drop the MM-DD portion of the date in favor of saving just the year for timeseries generation (where we assume ENDDATE = YYYY-01-01, so END_YEAR = YYYY - 1)

When you create a new case, CESM will create case.cupid, which in turn will execute the new cesm_postprocessing.sh script -- this means we can update what CUPiD does in the CESM workflow without requiring changes to ccs_config_cesm

mnlevy1981 · 2025-03-05T23:13:48Z

This is ready for a final review -- the final summary of changes:

Created cime_config/config_tool.xml, which CIME can parse to create env_postprocessing.xml
Created helper_scripts/cesm_postprocessing.py, which will be launched by case.cupid in the CESM workflow
Updated ADF to recent commit
Added several more arguments to generate_cupid_config_for_cesm_case.py because we want to pass variables from the new env_postprocessing.xml to this script
Updated one of the seaice notebooks to allow base_case_output_dir and CESM_output_dir to differ
cleaned up comments about datatypes in timeseries.py

Happy to address any concerns in the implementation, but I think this has all the features we need for cesm3_0_beta06 :)

mnlevy1981

This approval seems awfully shady, since half the commits in this PR are from me... but @TeaganKing can't approve it because she opened the PR (and also made the other half of the commits) 😄

TeaganKing · 2025-03-06T16:23:51Z

This approval seems awfully shady, since half the commits in this PR are from me... but @TeaganKing can't approve it because she opened the PR (and also made the other half of the commits) 😄

Hah! I am reviewing this right now, too. I will be commenting on it shortly...

TeaganKing

I have just a few minor change requests.

TeaganKing · 2025-03-06T16:18:04Z

helper_scripts/cesm_postprocessing.sh

+  echo "CUPID_RUN_ALL is True, running diagnostics for all components"
+fi
+
+unset PYTHONPATH


It may make more sense to move this closer to the comment on line 21 (or move the comment) so long as that doesn't mess anything else up?

Good catch! It seems safer to move the comment down than the unset command up :)

TeaganKing · 2025-03-06T16:26:11Z

helper_scripts/generate_cupid_config_for_cesm_case.py

+    cupid_startdate : str
+        The start date of the case being analyzed ("YYYY-MM-DD").
+
+    cupid_enddate : int


I think this one is a str?

Yup, another good catch :)

1. Move comment about unsetting python path to where we unset the path 2. correct comment about cupid_enddate datatype Note: I also removed the ./xmlquery MACH call because we weren't using that anywhere (I wonder if we used to only run some commands based on being on derecho, but now it's all controlled by env_postprocessing.xml)

mnlevy1981 · 2025-03-06T16:51:37Z

@TeaganKing hopefully ed9c322 addresses both of your requests (the cesm_postprocessing.sh comment doesn't show as outdated because I moved the comments you referred to rather than the actual line you commented on)

TeaganKing · 2025-03-06T16:53:03Z

Great! Looks good to me!

dabail10 · 2025-03-06T16:54:34Z

cime_config/config_tool.xml

+
+  <!-- Variables that define which components are analyzed -->
+
+  <entry id="CUPID_RUN_ALL">


Not a high priority, but do all of these need to be CUPID_RUN_STUFF?

I think the idea was that it would be clear that these are cupid-specific variables that turn on running a particular component. So, I wouldn't want to just leave the variable name as only ATM or ICE, for instance. Open to other suggestions, though!

I think we should leave the variable names as-is for now, we can always change them later :)

dabail10 · 2025-03-06T16:56:15Z

nblibrary/ice/Hemis_seaice_visual_compare_obs_lens.ipynb

    "case_name = \"\"  # \"b.e30_beta02.BLT1850.ne30_t232.104\"\n",
-    "base_case_name = \"\"  # \"b.e23_alpha17f.BLT1850.ne30_t232.092\"\n",
+    "base_case_name = None  # \"b.e23_alpha17f.BLT1850.ne30_t232.092\"\n",


I like this addition.

very preliminary updates in helper script for updated cupid workflow

99e4316

TeaganKing and others added 5 commits February 7, 2025 13:37

a few formatting things

741e8cb

Update generate_cupid_config_for_cesm_case.py to include argument des…

50ae4a5

…criptions from xml vars

remove whitespace in generate_cupid_config_for_cesm_case.py

4939a3d

Merge remote-tracking branch 'origin/main' into workflow

54ce71e

Add cime_config and update helper scripts

88d6f2c

cime_config/config_tool.xml is used by CESM to create env_postprocessing.xml. Variables in this XML file are then used to generate the cupid config file for CESM cases

TeaganKing assigned TeaganKing and mnlevy1981 Feb 11, 2025

TeaganKing added the framework label Feb 11, 2025

TeaganKing added 2 commits February 11, 2025 12:10

Update config_tool.xml to include ADF xml var

d8ecae9

Update generate_cupid_config_for_cesm_case.py formatting

e71a14e

TeaganKing marked this pull request as ready for review February 11, 2025 19:17

This was referenced Feb 11, 2025

Updates in template.cupid and config_workflow.xml for cupid workflow ESMCI/ccs_config_cesm#210

Merged

Create env_postprocessing.xml in case root ESMCI/cime#4742

Merged

mnlevy1981 mentioned this pull request Feb 19, 2025

Add RUN_POSTPROCESSING variable to env_run.xml ESCOMP/CMEPS#537

Merged

mnlevy1981 added 6 commits February 25, 2025 21:49

Merge branch 'main' into workflow

bdbabb1

Support base_case_output_dir in seaice notebook

6e85a34

ADF updates

8f06074

1. Bring in latest ADF commit as external 2. generate_cupid_config_for_cesm_case.py now has --adf-output-root argument that lets users specify a place other than ${CASEROOT} for ADF_output/ Also cleaned up comments about start_years and end_years in timeseries.py

Pass CI

35e362b

Create new script for CESM workflow

df6d41e

When you create a new case, CESM will create case.cupid, which in turn will execute the new cesm_postprocessing.sh script -- this means we can update what CUPiD does in the CESM workflow without requiring changes to ccs_config_cesm

Update config_tool.xml comments (minor typo)

884c8eb

mnlevy1981 approved these changes Mar 6, 2025

View reviewed changes

TeaganKing commented Mar 6, 2025

View reviewed changes

dabail10 approved these changes Mar 6, 2025

View reviewed changes

mnlevy1981 merged commit fc3c638 into NCAR:main Mar 6, 2025
2 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Updates in helper script for updated cupid workflow #176

Updates in helper script for updated cupid workflow #176

TeaganKing commented Jan 30, 2025 •

edited by mnlevy1981

Loading

mnlevy1981 commented Jan 30, 2025

mnlevy1981 commented Mar 5, 2025

mnlevy1981 left a comment •

edited

Loading

TeaganKing commented Mar 6, 2025

TeaganKing left a comment

TeaganKing Mar 6, 2025

mnlevy1981 Mar 6, 2025

TeaganKing Mar 6, 2025

mnlevy1981 Mar 6, 2025

mnlevy1981 commented Mar 6, 2025

TeaganKing commented Mar 6, 2025

dabail10 Mar 6, 2025

TeaganKing Mar 6, 2025

mnlevy1981 Mar 6, 2025

dabail10 Mar 6, 2025


		<!-- Variables that define which components are analyzed -->

		<entry id="CUPID_RUN_ALL">

Updates in helper script for updated cupid workflow #176

Updates in helper script for updated cupid workflow #176

Conversation

TeaganKing commented Jan 30, 2025 • edited by mnlevy1981 Loading

Description of changes: This PR will include additional parameters in the helper script which generates the cupid configuration file to account for a more flexible cupid workflow.

mnlevy1981 commented Jan 30, 2025

mnlevy1981 commented Mar 5, 2025

mnlevy1981 left a comment • edited Loading

Choose a reason for hiding this comment

TeaganKing commented Mar 6, 2025

TeaganKing left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mnlevy1981 commented Mar 6, 2025

TeaganKing commented Mar 6, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TeaganKing commented Jan 30, 2025 •

edited by mnlevy1981

Loading

mnlevy1981 left a comment •

edited

Loading