Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Improve the way fixed files are handled. (#212)
* Improve the way fixed files are handled. * Improve the way fixed files are handled so that what is happening with the fixed files is clearer. * Reduce the number of symlinks needed in each cycle directory. * Fix the EMC_CONUS_3km grid settings so that it has a valid BLOCKSIZE. * Add new WE2E test for EMC_CONUS_3km grid. List of modifications common to more than one file (these are referenced below for each file as needed): ------------------------------------------------------------------------------------------------------- (A) Instead of passing in NPROCS as an environment variable via the rocoto XML, calculate nprocs locally using workflow variables. (B) Use new function check_for_preexist_dir_file() that replaces old function check_for_preexist_dir(). The new one works for both directories and files. (C) Use the new function set_FV3_sfc_climo_filenames() to set those variables in the FV3 namelist file that specify the paths to the surface climatology files on the native grid. (D) Improve and/or clean up comments and/or informational and/or error messages. (E) Put in a check to make sure that the executable chgres_cube.exe exists. (F) Do not export variables when not necessary. (G) Remove commented-out (unused) code. (H) Remove blank lines. File-by-file modifications: -------------------------- jobs/JREGIONAL_GET_EXTRN_FILES: * For clarity, rename this to JREGIONAL_GET_EXTRN_MDL_FILES. jobs/JREGIONAL_MAKE_ICS: (A) jobs/JREGIONAL_MAKE_LBCS: (A) jobs/JREGIONAL_MAKE_SFC_CLIMO: (B) jobs/JREGIONAL_RUN_POST: (B) * Add check to make sure forecast hour contains exactly two digits. That's all EMC_post seems to be able to handle at the moment. scripts/exregional_make_grid.sh: (C), (D), (F) * Remove topo_dir as a variable since it is not needed in this script. scripts/exregional_make_ics.sh: (D), (E) scripts/exregional_make_lbcs.sh: (D), (E) scripts/exregional_make_orog.sh: (B), (F) * Remove definition of topo_dir in CHEYENNE stanza since this variable gets redefined later on anyway (for all machines). scripts/exregional_make_sfc_climo.sh: (A) scripts/exregional_run_fcst.sh: (A), (D) * Do not use the (now removed) workflow arrays FIXgsm_FILENAMES and FIXam_FILENAMES to create links in the cycle directory to fixed files in the FIXam directory. Instead, use the (new) workflow array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING that specifies the mapping between the symlinks that need to be created in the cycle directory and the corresponding files in the FIXam directory. * In community mode, make the symlinks that are created in the cycle directory to the certain files relative. These files consist of the data_table, field_table, the FV3 namelist, nems_configure, CCPP physics suite, and (for some physics suites) the CCN_ACTIVATE.BIN files. * Do not copy the forecast executable to the cycle directory. Instead, just call it using its full path in its original location (in EXECDIR). This saves disk space and reduces clutter. scripts/exregional_run_post.sh: (A), (G), (H) tests/baseline_configs/config.EMC_CONUS_3km.sh: * Add this new WE2E test configuration. This tests the workflow on EMC's 3km CONUS grid in NCO mode (i.e. RUN_ENVIR set to "nco"). tests/baseline_configs/config.regional_005.sh: * Specify non-default computational resources for this WE2E test (the 50km Alaska domain) such that the make_ics, make_lbcs, and run_post tasks do not crash (the number of MPI processes cannot be too high; otherwise, these tasks will fail). tests/baseline_configs/config.regional_009.sh: * Uncomment lines specific for Hera and comment out lines specific for Jet. This has to be changed to include an if-statement that sets parameters based on the machine. tests/baseline_configs/config.regional_015.sh: * Add this new WE2E test configuration. This tests the workflow on a coarse GFDLgrid type of grid that has a stretch factor of almost 1 (it cannot be exactly 1 because then the ufs_weather_model code assumes that it is a global grid and thus expects 6 tiles; this is a bug in the ufs_weather_model code that needs to be fixed). A more refined version of this grid (refine_ratio changed from 2 as it is here to 36) is being tested by GSL until issues with the JPgrid verification are worked out. Thus, we make sure capability of the workflow to run on this GFDLgrid doesn't break. tests/baselines_list.txt: * Update the list to include the new WE2E tests that are included in this commit (regional_014, regional_015, and EMC_CONUS_3km). This file isn't really necessary and should be deleted at some point (e.g. in a different commit that fixes up various deficiencies with the WE2E tests). ush/bash_utils/check_for_preexist_dir.sh: * Modify the function this file defines such that it now works for files as well as directories. * Rename the file and the function that it defines from check_for_preexist_dir(.sh) to check_for_preexist_dir_file(.sh). ush/compare_config_scripts.sh: * Change variable DEFAULT_EXPT_CONFIG_FN to EXPT_DEFAULT_CONFIG_FN (this variable is renamed in ush/setup.sh). ush/config_defaults.sh: * Put the variable SCHED specifying the job scheduler to use in this file so that it becomes user-specifiable (and avaiable as a workflow variable). This may be needed when adding the capability to run outside of the scheduler (in a future commit/PR). * Include detailed descriptions of the NCO mode variables (COMINgfs, STMP, NET, envir, RUN, PTMP) are used. * Add new variable (EXPT_CONFIG_FN) that specifies the name of the experiment/workflow configuration file containing user-specified values. * Add new variable (FV3_EXEC_FN) that specifies the name to use for the forecast model executable if/when it is copied from the directory in which the build step creates it to the directory in which the workflow scripts look for it (EXECDIR). This copy step should really be done in during the build. * For clarity and consistency, rename variables FV3_NML_FN and FV3_NML_CONFIG to FV3_NML_BASE_FN and FV3_NML_YAML_CONFIG_FN, respectively. * Add new array SFC_CLIMO_FIELDS containing the names of the surface climatology fields used by the sfc_climo_gen code. * Remove some variables (FNALBC, FNALBC2, etc) whose names are now specified in the new array FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING. * Add new array FIXgsm_FILES_TO_COPY_TO_FIXam that specifies the fixed files that must be copied from the system directory FIXgsm to the local directory FIXam (which is under the experiment directory). * Add new array FV3_NML_VARNAME_TO_FIXam_FILES_MAPPING that specifies the FV3 namelist variable names and the corresponding files in the FIXam directory that they are associated with (the variables need to be set to the relative or full paths to these files). * Add new array FV3_NML_VARNAME_TO_SFC_CLIMO_FIELD_MAPPING that specifies the FV3 namelist variable names and the corresponding surface climatology fields in the sfc_climo_gen code (the variables need to be set to the relative or full paths to the files containing the corresponding fields). * Add new array CYCLEDIR_LINKS_TO_FIXam_FILES_MAPPING specifying the symlinks that need to be created in each cycle directory and their corresponding target files in the FIXam directory. * Add new variables specifying the workflow task names and the corresponding values of the number of nodes, logical processes per node, and maximum wallclock time for each task. ush/generate_FV3SAR_wflow.sh: (D) * Remove variables PROC_RUN_FCST and NPROCS_RUN_FCST since they are now superceded by the variables NNODES_RUN_FCST and PPN_RUN_FCST. * Remove the variable CYCLE_DIR since it is being replaced by CYCLE_BASEDIR (which is set in ush/setup.sh; see description of changes to that file). * For clarity, reorder (and include comments between) the lines in the variable "settings" that specifies how to set the variables (entities, etc) in the template rocoto XML. * Replace code that copies files from the FIXgsm directory with code that does the same thing using (looping over) the new workflow array variable FIXgsm_FILES_TO_COPY_TO_FIXam. * Put in new code that copies the forecast model executable from the directoy in which the forecast model repository was cloned into the executable directory (EXECDIR) but ONLY IF such a copy does not already exist in EXECDIR or, if it does exist, it is older than the source executable. This copying should really be done in the build step, but it currently is not. * For clarity, reorder some of the lines in the variable "settings" that is used to specify values in the FV3 fortran namelist file (input.nml) such that they are grouped by type (e.g. grid-related, computational, etc). Also, include some comments between the lines. * Remove the hard-coded settings of the "FN..." variables from the "settings" variable. Instead: * For the "FN..." variables that specify paths to fixed files in the FIXam directory, loop through the new workflow array variable FV3_NML_VARNAME_TO_FIXam_FILES_MAPPING and, for each element of this array, create an appropriate entry in "settings". * For the "FN..." variables that specify the paths to surface climatology files: (1) If not running the make_grid task, call the new function set_FV3nml_sfc_climo_filenames that sets those FV3 namelist variables that specify the paths to surface climatology files to the names of the surface climatology files on the FV3SAR grid (these files are either pregenerated or are created by the make_sfc_climo task). (2) If running the make_grid task, call this same funtion in the make_grid task. This is because the names of the surface climatology files depend on the CRES parameter (the C-resolution of the grid), and usually, that is not known until the grid is created (by the make_grid task). ush/link_fix.sh: * Replace local variable sfc_climo_fields with new workflow array variable SFC_CLIMO_FIELDS. * Add code that creates symlinks in the FIXsar directory to surface climatology files with names that end in "tile7.nc" that have names that end in "tile1.nc". This is apparently necessary for FV3 to be able to use the surface climatology files on the native FV3SAR grid (which are either pregenerated or created by the make_sfc_climo task). ush/load_modules_run_task.sh: (D) * Fix indentation. ush/set_FV3nml_sfc_climo_filenames.sh: * This is a new file that defines a function (set_FV3nml_sfc_climo_filenames) that sets the values of the variables in the forecast model's namelist file that specify the paths to the surface climatology files on the FV3SAR native grid (which are either regenerated or created by the MAKE_SFC_CLIMO_TN task). We make this a function because it is needed in a couple of places in the scripts. ush/set_fix_filenames.sh: * Remove this file since the fixed files are being handled differently now using the new workflow array variables defined in ush/config_defaults.sh. ush/set_ozone_param.sh: * This is a new file that defines a function (set_ozone_param) that sets various workflow parameters depending on which ozone parameterization is being used by the experiment. The ozone parameterization is determined by inspecting the CCPP suite defintion file. ush/set_predef_grid_params.sh: (G) * Set BLOCKSIZE to an allowed value for the EMC_CONUS_3km grid. ush/setup.sh: (D), (G) * For clarity, rename variable DEFAULT_EXPT_CONFIG_FN to EXPT_DEFAULT_CONFIG_FN. * Remove variable EXPT_CONFIG_FN since that is now defined in config_defaults.sh. * Set SCHED to a machine-dependent value only if it is not already specified by the user in config.sh. * Remove stanza for machine THEIA (macine no longer exists). * Change SCHED to lowercase (since that is how rocoto accepts it) and Test to make sure that SCHED is set to a valid value. * Remove setting of variables that specify the task names since those are now in config_defaults.sh. * Set the new workflow variable CYCLE_BASEDIR that contains the base directory under which the cycle directories are created. This replaces CYCLE_DIR. Note that CYCLE_BASEDIR is the old CYCLE_DIR but without the last directory (which is the CDATE of the cycle). Now, CYCLE_DIR is formed within the XML by combining CYCLE_BASEDIR and the current cycle's date information from rocoto. Also, add CYCLE_BASEDIR to the variable definitions file (var_defns.sh). * For clarity, rename variable FV3_NML_CONFIG_FP to FV3_NML_YAML_CONFIG_FP. * For convenience, create variable FV3_NML_FN (that is set to the name of the namelist file that FV3 expects to read in). Then use this to set FV3_NML_FP, which is the full path to that namelist file. * Create new workflow variable FV3_EXEC_FP that conains the full path to the copy of the FV3SAR executable in the executables directory EXECDIR. Add this new variable to the variable definitions file (in a "cat" statement towards the end of the file). * Set the new variable LOAD_MODULES_RUN_TASK_FP that specifies the full path to the script that loads the modules for each task and then runs the task. This used to be hard-coded inside of the rocoto XML but is now needed as a workflow variable (because it is needed in at least a couple of places). Add this variable to the variable definitions file (var_defns.sh). * Fix bug (the way the local variable glob_pattern is set) to allow for the use of a dot as the fixed file name separator when in NCO mode (i.e. when RUN_ENVIR is set to "nco"). * Rename the variable NUM_NODES to NNODES_RUN_FCST to be consistent with the naming convention for number of nodes used for other tasks. * Remove call to function set_fix_filenames since it is no longer needed. * Remove recording of the array variables FIXgsm_FILENAMES and FIXam_FILENAMES in the variable defintions file since these arrays are no longer needed (they were previously set by the function set_fix_filenames). * Call the new function set_ozone_param that sets various workflow parameters depending on which ozone parameterization is being used (which is specified in the CCPP suite defintion file). * Clean up comments. ush/source_util_funcs.sh: (D) * Change the file name check_for_preexist_dir.sh to check_for_preexist_dir_file.sh. ush/templates/FV3SAR_wflow.xml: * For clarity, reorder the entities that contain jinja parameters by grouping similar items together. * Make LOAD_MODULES_RUN_TASK_FP an ENTITY that is set externally by the workflow generation scripts (as opposed to hard-coded in the XML) because this is now a variable that is needed in another file (the script that is used to run a task outside of rocoto). * Replace CYCLE_DIR as an externally specified ENTITY with CYCLE_BASEDIR. * Create new externally-specified ENTITYs for the number of nodes, the number of MPI processes per node, and the wallclock time for each task. These will get set by the workflow generation scripts. * Make LOAD_MODULES_RUN_TASK_FP an ENTITY that is set externally by the workflow generation scripts (as opposed to hard-coded in the XML) because this is now a variable that is needed in another file (the script that is used to run a task outside of rocoto). * Use the ENTITYs for the number of nodes, the number of MPI processes per node, and the wallclock time for each task to set the PROC_... and RSRC_... ENTITYs for each task (as opposed to hard-coding the latter). * For brevity, rename ENTITYs containing the strings "MAKE_ICS_SURF_LBC0" and "MAKE_LBC1_TO_LBCN" to "MAKE_ICS" and "MAKE_LBCS", respectively. * For consistency with other tasks, rename PROC_POST and RSRC_POST to PROC_RUN_POST and RSRC_RUN_POST. * Change name of ENTITY RSRV_RUN_FCST to RSRV_FCST to be consistent with name of ENTITY QUEUE_FCST. Names of ENTITYs related to the job scheduler need to be revisited (e.g. maybe change QUEUE_FCST to QUEPART_FCST since in slurm, there are partitions, not queues). * Remove variable PDY from pre-processing tasks (which are cycle-independent). * Fix bug where the task name "make_grid" is hard-coded as opposed to using the ENTITY MAKE_GRID_TN. * For clarity, rename file JREGIONAL_GET_EXTRN_FILES to JREGIONAL_GET_EXTRN_MDL_FILES. * Reorder environment variable settings in each task to go from least to most general. * Within each cycle-dependent task, set CYCLE_DIR based on CYCLE_BASEDIR. ush/valid_param_vals.sh: * Add valid values for the new user-specifiable workflow variable SCHED. * Remove the unused predefined grid GSD_HRRR_AK_3km since it has been renamed GSD_RRFSAK_3km. * Manual merge of Mike's changes for file ush/load_modules_run_task.sh. * Improve/streamline generation of rocoto workflow XML. Modifications made to more than one file: ---------------------------------------- (A) For uniformity among workflow tasks, split the worfklow variables that specify the number of nodes, logical processes per node, and wall clock time for the generic GET_EXTRN_MDL_FILES task into two variables each -- one for the GET_EXTRN_ICS_TN and another for the GET_EXTRN_LBCS_TN task. (B) Remove USHDIR as an ENTITY in the XML since it is not used after being set. ush/config_defaults.sh: (A) ush/generate_FV3SAR_wflow.xml: (B) ush/templates/FV3SAR_wflow.xml: (A), (B) * Clean up comments. * To reduce the number of entities rocoto has to deal with, remove the ENTITY definitions for various entities that only appear in one location in the file and replace them with the corresponding jinja2 placeholders or combinations of such placeholders. Do this for the following entities: * The per-task entities NNODES_..., PPN_..., WTIME_..., PROC_..., and RSRC_.... * EXTRN_MDL_NAME_ICS and EXTRN_MDL_NAME_LBCS * Get EMC's changes into community workflow (Ratko's PR plus other necessary changes). File-by-file description of modifications: ----------------------------------------- fix/fix_upp/nam_nests.hiresf_nn.txt: * Deleted as requested by EMC (from Ratko's original PR). fix/fix_upp/postxconfig-NT-fv3sar.txt * Modified as requested by EMC (from Ratko's original PR). scripts/exregional_get_extrn_files.sh: * If extracting the external model files from one or more HPSS archive files, put in a check to ensure that each external model file was extracted exactly once from one of the archive files. scripts/exregional_make_ics.sh: * Add parameter settings for the FV3_GFS_2017_gfdlmp_regional physics suite (from Ratko's original PR). * The FV3_GFS_2017_gfdlmp_regional physics suite requires a variable called tref (reference temperature) in the initial conditions. That variable is available only for external files coming from the FV3GFS model. tref can be extracted from the surface analysis file if the namelist variable convert_nst is set to .true. in the chgres_cube namelist file. Thus, create a new local variable named convert_nst that gets set to true for the FV3GFS model but is false otherwise and pass it to the chgres_cube namelist. scripts/exregional_make_lbcs.sh: * Add parameter settings for the FV3_GFS_2017_gfdlmp_regional physics suite (from Ratko's original PR). tests/baseline_configs/config.EMC_CONUS_3km.sh: * Rename to tests/baseline_configs/config.nco_conus.sh (see that file). tests/baseline_configs/config.nco_conus.sh: * Renamed from tests/baseline_configs/config.EMC_CONUS_3km.sh. * Change queue from debug to batch since we always want batch as the default (so that if running many tests, they are not limited by the max number of jobs a user can have in the debug queue, which on hera is just 2). * Set the physics suite for this WE2E test to FV3_GFS_2017_gfdlmp_regional because that's the one EMC runs with. * Reset the RUN variable (which is used in creating output directory if in NCO mode) to a value that is unique to this WE2E test so that it doesn't interfere with other tests that are running at the same time (and vice versa). tests/baseline_configs/config.nco_conus_c96.sh: * New WE2E test for the coarse EMC CONUS grid. For rapid testing, this can be run instead of config.nco_conus.sh. tests/baseline_configs/config.new_JPgrid.sh * New WE2E test for specifying a new (i.e. not predefined) JPgrid. tests/baseline_configs/config.regional_009.sh: * Reset the RUN variable (which is used in creating output directory if in NCO mode) to a value that is unique to this WE2E test so that it doesn't interfere with other tests that are running at the same time (and vice versa). tests/baselines_list.txt: * Update the full list of WE2E test names to the latest set. ush/config_defaults.sh: * Edit comments. ush/generate_FV3SAR_wflow.sh: * Remove FNZORC, FNTSFA, FNACNA, and FNSNOA from the "settings" variable used in generating the FV3 input namelist file. These variables are now instead specified in the base namelist file (ush/templates/input.nml.FV3) because they are the same for all physics suites considered thus far. ush/get_extrn_mdl_file_dir_info.sh: * Remove unused code. ush/link_fix.sh: * Bug fix: Change the targets of the "...tile1..." symlinks so that they point to the tile7.halo0.nc files, not the tile7.nc symlinks (which in turn point to the tile7.halo4.nc files, which is not what we want). ush/set_namelist.py: * Add option "sort=True" to nml.write method to sort the resulting FV3 namelist file in alphabetical order. This makes it easier to compare (diff) namelists. ush/setup.sh: * Add a temporary section of code that makes fixes to the suite defintion file for the FV3_GFS_2017_gfdlmp_regional physics suite. This is necessary because the version of the file in the dtc/develop branch of the NCAR fork of the fv3atm repo has not yet been updated to have these corrections. This code should be removed once that branch/repo have been updated. ush/templates/FV3.input.yml: * Add section for the new FV3_GFS_2017_gfdlmp_regional physics suite. Note that this references the preexisting FV3_GFS_2017_gfdlmp physics suite. ush/templates/input.nml.FV3: * Reorder lines so that both namelists and the variables within each namelist are lower case and in alphabetical order. This makes it easier to compare this base FV3 namelist file to other FV3 namelist files generated using the set_namelist.py script. ush/templates/diag_table.FV3_GFS_2017_gfdlmp_regional: * New diag_table file for FV3_GFS_2017_gfdlmp_regional suite. ush/templates/field_table.FV3_GFS_2017_gfdlmp_regional: * New field_table file for FV3_GFS_2017_gfdlmp_regional suite. ush/templates/model_configure.FV3_GFS_2017_gfdlmp_regional: * New model_configure file for FV3_GFS_2017_gfdlmp_regional suite. ush/valid_param_vals.sh: * Add FV3_GFS_2017_gfdlmp_regional to the list of valid physic suites (as requested by EMC). * Change compile options for EMC as requested by Ratko.
- Loading branch information
1 parent
2b82500
commit a78b7d6