Skip to content

Commit e09989b

Browse files
DavidHuber-NOAADavidHuber-NOAA
authored
Port to S4 (#1023)
Ports the global workflow to the S4 cluster. Note that this does not include support for S2S experiments. Additionally, S4 does not support C768 experiments. A couple of special notes: - S4 does not have access to rstprod data. Among other things, this means that the nsstbufr and prepbufr.acft_profiles files must be created on the fly. The way I accomplished this was by moving the `MAKE_NSSTBUFR` to and creating `MAKE_ACFTBUFR` in config.base.emc.dyn and setting them via setup_workflow.xml. This seems like a bit of a kludge and I am open to suggestions on how else to address this. Both options need to be set for the prep and analysis jobs. - S4 _can_ run S2S+ experiments, but this requires significant, and convoluted, modifications to the configuration files. Support for these are thus not enabled by default. Instead, I have placed a set of configuration files in S4:/data/users/dhuber/save/s2s_configs. Users interested in performing these experiments should contact me to set it up. Closes #138
1 parent b26a8ac commit e09989b

28 files changed

+756
-206
lines changed

‎Externals.cfg

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ protocol = git
88
required = True
99

1010
[gfs-utils]
11-
hash = 3a609ea
11+
hash = 7bf599f
1212
local_path = sorc/gfs_utils.fd
1313
repo_url = https://github.com/NOAA-EMC/gfs-utils
1414
protocol = git

‎README.md

+4-3
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,14 @@ The global-workflow depends on the following prerequisities to be available on t
77
* workflow manager - ROCOTO (https://github.com/christopherwharrop/rocoto)
88
* modules - NCEPLIBS (various), esmf v8.0.0bs48, hdf5, intel/ips v18, impi v18, wgrib2, netcdf v4.7.0, hpss, gempak (see module files under /modulefiles for additional details)
99

10-
The global-workflow current supports the following machines:
10+
The global-workflow current supports the following tier-1 machines:
1111

12-
* WCOSS-Dell
13-
* WCOSS-Cray
1412
* Hera
1513
* Orion
1614

15+
Additionally, the following tier-2 machine is supported:
16+
* S4 (Note that S2S+ experiments are not fully supported)
17+
1718
Quick-start instructions are below. Full instructions are available in the [wiki](https://github.com/NOAA-EMC/global-workflow/wiki/Run-Global-Workflow)
1819

1920
## Build global-workflow:

‎env/S4.env

+295
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,295 @@
1+
#!/bin/bash -x
2+
3+
if [ $# -ne 1 ]; then
4+
5+
echo "Must specify an input argument to set runtime environment variables!"
6+
echo "argument can be any one of the following:"
7+
echo "atmanalrun atmensanalrun"
8+
echo "anal sfcanl fcst post vrfy metp"
9+
echo "eobs eupd ecen efcs epos"
10+
echo "postsnd awips gempak"
11+
exit 1
12+
13+
fi
14+
15+
step=$1
16+
PARTITION_BATCH=${PARTITION_BATCH:-"s4"}
17+
18+
if [[ ${PARTITION_BATCH} = "s4" ]]; then
19+
export npe_node_max=32
20+
elif [[ ${PARTITION_BATCH} = "ivy" ]]; then
21+
export npe_node_max=20
22+
fi
23+
export launcher="srun -l --export=ALL"
24+
25+
# Configure MPI environment
26+
export OMP_STACKSIZE=2048000
27+
export NTHSTACK=1024000000
28+
29+
ulimit -s unlimited
30+
ulimit -a
31+
32+
export job=${PBS_JOBNAME:-${step}}
33+
export jobid=${job}.${PBS_JOBID:-$$}
34+
35+
36+
if [[ ${step} = "prep" || ${step} = "prepbufr" ]]; then
37+
38+
npe_node_prep=${npe_node_prep:-${npe_node_max}}
39+
nth_max=$((npe_node_max / npe_node_prep))
40+
41+
export POE="NO"
42+
export BACK="NO"
43+
export sys_tp="S4"
44+
export launcher_PREP="srun"
45+
46+
elif [[ ${step} = "waveinit" || ${step} = "waveprep" || ${step} = "wavepostsbs" || ${step} = "wavepostbndpnt" || ${step} = "wavepostbndpntbll" || ${step} = "wavepostpnt" ]]; then
47+
48+
export mpmd="--multi-prog"
49+
export CFP_MP="YES"
50+
if [[ ${step} = "waveprep" ]]; then export MP_PULSE=0 ; fi
51+
export wavempexec=${launcher}
52+
export wave_mpmd=${mpmd}
53+
54+
elif [[ ${step} = "atmanalrun" ]]; then
55+
56+
export CFP_MP=${CFP_MP:-"YES"}
57+
export USE_CFP=${USE_CFP:-"YES"}
58+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
59+
60+
npe_node_atmanalrun=${npe_node_atmanalrun:-${npe_node_max}}
61+
nth_max=$((npe_node_max / npe_node_atmanalrun))
62+
63+
export NTHREADS_ATMANAL=${nth_atmanalrun:-${nth_max}}
64+
[[ ${NTHREADS_ATMANAL} -gt ${nth_max} ]] && export NTHREADS_ATMANAL=${nth_max}
65+
export APRUN_ATMANAL="${launcher} -n ${npe_atmanalrun:-0}"
66+
67+
elif [[ ${step} = "atmensanalrun" ]]; then
68+
69+
export CFP_MP=${CFP_MP:-"YES"}
70+
export USE_CFP=${USE_CFP:-"YES"}
71+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
72+
73+
npe_node_atmensanalrun=${npe_node_atmensanalrun:-${npe_node_max}}
74+
nth_max=$((npe_node_max / npe_node_atmensanalrun))
75+
76+
export NTHREADS_ATMENSANAL=${nth_atmensanalrun:-${nth_max}}
77+
[[ ${NTHREADS_ATMENSANAL} -gt ${nth_max} ]] && export NTHREADS_ATMENSANAL=${nth_max}
78+
export APRUN_ATMENSANAL="${launcher} -n ${npe_atmensanalrun:-0}"
79+
80+
elif [[ ${step} = "aeroanlrun" ]]; then
81+
82+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
83+
84+
npe_node_aeroanlrun=${npe_node_aeroanlrun:-${npe_node_max}}
85+
nth_max=$((npe_node_max / npe_node_aeroanlrun))
86+
87+
export NTHREADS_AEROANL=${nth_aeroanlrun:-${nth_max}}
88+
[[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max}
89+
export APRUN_AEROANL="${launcher} -n ${npe_aeroanlrun:-0}"
90+
91+
elif [[ ${step} = "anal" || ${step} = "analcalc" ]]; then
92+
93+
export MKL_NUM_THREADS=4
94+
export MKL_CBWR=AUTO
95+
96+
export CFP_MP=${CFP_MP:-"YES"}
97+
export USE_CFP=${USE_CFP:-"YES"}
98+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
99+
100+
npe_node_anal=${npe_node_anal:-${npe_node_max}}
101+
nth_max=$((npe_node_max / npe_node_anal))
102+
103+
export NTHREADS_GSI=${nth_anal:-${nth_max}}
104+
[[ ${NTHREADS_GSI} -gt ${nth_max} ]] && export NTHREADS_GSI=${nth_max}
105+
export APRUN_GSI=${launcher}
106+
107+
export NTHREADS_CALCINC=${nth_calcinc:-1}
108+
[[ ${NTHREADS_CALCINC} -gt ${nth_max} ]] && export NTHREADS_CALCINC=${nth_max}
109+
export APRUN_CALCINC=${launcher}
110+
111+
export NTHREADS_CYCLE=${nth_cycle:-12}
112+
[[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
113+
npe_cycle=${ntiles:-6}
114+
export APRUN_CYCLE="${launcher} -n ${npe_cycle:-0}"
115+
116+
117+
export NTHREADS_GAUSFCANL=1
118+
npe_gausfcanl=${npe_gausfcanl:-1}
119+
export APRUN_GAUSFCANL="${launcher} -n ${npe_gausfcanl:-0}"
120+
121+
elif [[ ${step} = "sfcanl" ]]; then
122+
npe_node_sfcanl=${npe_node_sfcanl:-${npe_node_max}}
123+
nth_max=$((npe_node_max / npe_node_sfcanl))
124+
125+
export NTHREADS_CYCLE=${nth_sfcanl:-14}
126+
[[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
127+
npe_sfcanl=${ntiles:-6}
128+
export APRUN_CYCLE="${launcher} -n ${npe_sfcanl:-0}"
129+
130+
elif [[ ${step} = "gldas" ]]; then
131+
132+
npe_node_gldas=${npe_node_gldas:-${npe_node_max}}
133+
nth_max=$((npe_node_max / npe_node_gldas))
134+
135+
export NTHREADS_GLDAS=${nth_gldas:-${nth_max}}
136+
[[ ${NTHREADS_GLDAS} -gt ${nth_max} ]] && export NTHREADS_GLDAS=${nth_max}
137+
export APRUN_GLDAS="${launcher} -n ${npe_gldas:-0}"
138+
139+
export NTHREADS_GAUSSIAN=${nth_gaussian:-1}
140+
[[ ${NTHREADS_GAUSSIAN} -gt ${nth_max} ]] && export NTHREADS_GAUSSIAN=${nth_max}
141+
export APRUN_GAUSSIAN="${launcher} -n ${npe_gaussian:-0}"
142+
143+
# Must run data processing with exactly the number of tasks as time
144+
# periods being processed.
145+
146+
gldas_spinup_hours=${gldas_spinup_hours:-0}
147+
npe_gldas_data_proc=$((gldas_spinup_hours + 12))
148+
export APRUN_GLDAS_DATA_PROC="${launcher} -n ${npe_gldas_data_proc} --multi-prog"
149+
150+
elif [[ ${step} = "eobs" ]]; then
151+
152+
export MKL_NUM_THREADS=4
153+
export MKL_CBWR=AUTO
154+
155+
export CFP_MP=${CFP_MP:-"YES"}
156+
export USE_CFP=${USE_CFP:-"YES"}
157+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
158+
159+
npe_node_eobs=${npe_node_eobs:-${npe_node_max}}
160+
nth_max=$((npe_node_max / npe_node_eobs))
161+
162+
export NTHREADS_GSI=${nth_eobs:-${nth_max}}
163+
[[ ${NTHREADS_GSI} -gt ${nth_max} ]] && export NTHREADS_GSI=${nth_max}
164+
export APRUN_GSI=${launcher}
165+
166+
elif [[ ${step} = "eupd" ]]; then
167+
168+
export CFP_MP=${CFP_MP:-"YES"}
169+
export USE_CFP=${USE_CFP:-"YES"}
170+
export APRUNCFP="${launcher} -n \$ncmd --multi-prog"
171+
172+
npe_node_eupd=${npe_node_eupd:-${npe_node_max}}
173+
nth_max=$((npe_node_max / npe_node_eupd))
174+
175+
export NTHREADS_ENKF=${nth_eupd:-${nth_max}}
176+
[[ ${NTHREADS_ENKF} -gt ${nth_max} ]] && export NTHREADS_ENKF=${nth_max}
177+
export APRUN_ENKF=${launcher}
178+
179+
elif [[ ${step} = "fcst" ]]; then
180+
181+
#PEs and PEs/node can differ for GFS and GDAS forecasts if threading differs
182+
if [[ ${CDUMP:-gdas} == "gfs" ]]; then
183+
npe_fcst=${npe_fcst_gfs:-0}
184+
npe_node_fcst=${npe_node_fcst_gfs:-${npe_node_max}}
185+
nth_fv3=${nth_fv3_gfs:-1}
186+
fi
187+
188+
npe_node_fcst=${npe_node_fcst:-${npe_node_max}}
189+
nth_max=$((npe_node_max / npe_node_fcst))
190+
191+
export NTHREADS_FV3=${nth_fv3:-${nth_max}}
192+
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
193+
export cores_per_node=${npe_node_max}
194+
export APRUN_FV3="${launcher} -n ${npe_fcst:-0}"
195+
196+
export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
197+
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
198+
export APRUN_REGRID_NEMSIO=${launcher}
199+
200+
export NTHREADS_REMAP=${nth_remap:-2}
201+
[[ ${NTHREADS_REMAP} -gt ${nth_max} ]] && export NTHREADS_REMAP=${nth_max}
202+
export APRUN_REMAP=${launcher}
203+
export I_MPI_DAPL_UD="enable"
204+
205+
elif [[ ${step} = "efcs" ]]; then
206+
207+
npe_node_efcs=${npe_node_efcs:-${npe_node_max}}
208+
nth_max=$((npe_node_max / npe_node_efcs))
209+
210+
export NTHREADS_FV3=${nth_efcs:-${nth_max}}
211+
[[ ${NTHREADS_FV3} -gt ${nth_max} ]] && export NTHREADS_FV3=${nth_max}
212+
export cores_per_node=${npe_node_max}
213+
export APRUN_FV3="${launcher} -n ${npe_efcs:-0}"
214+
215+
export NTHREADS_REGRID_NEMSIO=${nth_regrid_nemsio:-1}
216+
[[ ${NTHREADS_REGRID_NEMSIO} -gt ${nth_max} ]] && export NTHREADS_REGRID_NEMSIO=${nth_max}
217+
export APRUN_REGRID_NEMSIO="${launcher} ${LEVS:-128}"
218+
219+
elif [[ ${step} = "post" ]]; then
220+
221+
npe_node_post=${npe_node_post:-npe_node_max}
222+
nth_max=$((npe_node_max / npe_node_post))
223+
224+
export NTHREADS_NP=${nth_np:-1}
225+
[[ ${NTHREADS_NP} -gt ${nth_max} ]] && export NTHREADS_NP=${nth_max}
226+
export APRUN_NP=${launcher}
227+
228+
export NTHREADS_DWN=${nth_dwn:-1}
229+
[[ ${NTHREADS_DWN} -gt ${nth_max} ]] && export NTHREADS_DWN=${nth_max}
230+
export APRUN_DWN=${launcher}
231+
232+
elif [[ ${step} = "ecen" ]]; then
233+
234+
npe_node_ecen=${npe_node_ecen:-${npe_node_max}}
235+
nth_max=$((npe_node_max / npe_node_ecen))
236+
237+
export NTHREADS_ECEN=${nth_ecen:-${nth_max}}
238+
[[ ${NTHREADS_ECEN} -gt ${nth_max} ]] && export NTHREADS_ECEN=${nth_max}
239+
export APRUN_ECEN=${launcher}
240+
241+
export NTHREADS_CHGRES=${nth_chgres:-12}
242+
[[ ${NTHREADS_CHGRES} -gt ${npe_node_max} ]] && export NTHREADS_CHGRES=${npe_node_max}
243+
export APRUN_CHGRES="time"
244+
245+
export NTHREADS_CALCINC=${nth_calcinc:-1}
246+
[[ ${NTHREADS_CALCINC} -gt ${nth_max} ]] && export NTHREADS_CALCINC=${nth_max}
247+
export APRUN_CALCINC=${launcher}
248+
249+
elif [[ ${step} = "esfc" ]]; then
250+
251+
npe_node_esfc=${npe_node_esfc:-${npe_node_max}}
252+
nth_max=$((npe_node_max / npe_node_esfc))
253+
254+
export NTHREADS_ESFC=${nth_esfc:-${nth_max}}
255+
[[ ${NTHREADS_ESFC} -gt ${nth_max} ]] && export NTHREADS_ESFC=${nth_max}
256+
export APRUN_ESFC="${launcher} -n ${npe_esfc:-0}"
257+
258+
export NTHREADS_CYCLE=${nth_cycle:-14}
259+
[[ ${NTHREADS_CYCLE} -gt ${npe_node_max} ]] && export NTHREADS_CYCLE=${npe_node_max}
260+
export APRUN_CYCLE="${launcher} -n ${npe_esfc:-0}"
261+
262+
elif [[ ${step} = "epos" ]]; then
263+
264+
npe_node_epos=${npe_node_epos:-${npe_node_max}}
265+
nth_max=$((npe_node_max / npe_node_epos))
266+
267+
export NTHREADS_EPOS=${nth_epos:-${nth_max}}
268+
[[ ${NTHREADS_EPOS} -gt ${nth_max} ]] && export NTHREADS_EPOS=${nth_max}
269+
export APRUN_EPOS=${launcher}
270+
271+
elif [[ ${step} = "init" ]]; then
272+
273+
export APRUN=${launcher}
274+
275+
elif [[ ${step} = "postsnd" ]]; then
276+
277+
npe_node_postsnd=${npe_node_postsnd:-${npe_node_max}}
278+
nth_max=$((npe_node_max / npe_node_postsnd))
279+
280+
export NTHREADS_POSTSND=${nth_postsnd:-1}
281+
[[ ${NTHREADS_POSTSND} -gt ${nth_max} ]] && export NTHREADS_POSTSND=${nth_max}
282+
export APRUN_POSTSND=${launcher}
283+
284+
export NTHREADS_POSTSNDCFP=${nth_postsndcfp:-1}
285+
[[ ${NTHREADS_POSTSNDCFP} -gt ${nth_max} ]] && export NTHREADS_POSTSNDCFP=${nth_max}
286+
export APRUN_POSTSNDCFP=${launcher}
287+
288+
elif [[ ${step} = "awips" ]]; then
289+
290+
echo "WARNING: ${step} is not enabled on S4!"
291+
292+
elif [[ ${step} = "gempak" ]]; then
293+
294+
echo "WARNING: ${step} is not enabled on S4!"
295+
fi

‎jobs/JGDAS_ENKF_DIAG

+15-2
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ export pgmerr=errfile
5656
export CDATE=${CDATE:-${PDY}${cyc}}
5757
export CDUMP=${CDUMP:-${RUN:-"gdas"}}
5858
export COMPONENT=${COMPONENT:-atmos}
59+
export MAKE_NSSTBUFR=${MAKE_NSSTBUFR:-"NO"}
60+
export MAKE_ACFTBUFR=${MAKE_ACFTBUFR:-"NO"}
5961

6062

6163
##############################################
@@ -104,9 +106,20 @@ export PREPQC="$COMIN_OBS/${OPREFIX}prepbufr"
104106
if [ ! -f $PREPQC ]; then
105107
echo "WARNING: Global PREPBUFR FILE $PREPQC MISSING"
106108
fi
107-
export PREPQCPF="$COMIN_OBS/${OPREFIX}prepbufr.acft_profiles"
108109
export TCVITL="$COMIN_ANL/${OPREFIX}syndata.tcvitals.tm00"
109-
[[ $DONST = "YES" ]] && export NSSTBF="$COMIN_OBS/${OPREFIX}nsstbufr"
110+
if [[ $DONST = "YES" ]]; then
111+
if [[ $MAKE_NSSTBUFR == "YES" ]]; then
112+
export NSSTBF="${COMOUT}/${OPREFIX}nsstbufr"
113+
else
114+
export NSSTBF="${COMIN_OBS}/${OPREFIX}nsstbufr"
115+
fi
116+
fi
117+
118+
if [[ $MAKE_ACFTBUFR == "YES" ]]; then
119+
export PREPQCPF="${COMOUT}/${OPREFIX}prepbufr.acft_profiles"
120+
else
121+
export PREPQCPF="${COMIN_OBS}/${OPREFIX}prepbufr.acft_profiles"
122+
fi
110123

111124
# Guess Bias correction coefficients related to control
112125
export GBIAS=${COMIN_GES_CTL}/${GPREFIX}abias

‎jobs/JGDAS_ENKF_SELECT_OBS

+15-2
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,8 @@ export pgmerr=errfile
5656
export CDATE=${CDATE:-${PDY}${cyc}}
5757
export CDUMP=${CDUMP:-${RUN:-"gdas"}}
5858
export COMPONENT=${COMPONENT:-atmos}
59+
export MAKE_NSSTBUFR=${MAKE_NSSTBUFR:-"NO"}
60+
export MAKE_ACFTBUFR=${MAKE_ACFTBUFR:-"NO"}
5961

6062

6163
##############################################
@@ -107,9 +109,20 @@ export PREPQC="$COMIN_OBS/${OPREFIX}prepbufr"
107109
if [ ! -f $PREPQC ]; then
108110
echo "WARNING: Global PREPBUFR FILE $PREPQC MISSING"
109111
fi
110-
export PREPQCPF="$COMIN_OBS/${OPREFIX}prepbufr.acft_profiles"
111112
export TCVITL="$COMIN_ANL/${OPREFIX}syndata.tcvitals.tm00"
112-
[[ $DONST = "YES" ]] && export NSSTBF="$COMIN_OBS/${OPREFIX}nsstbufr"
113+
if [[ $DONST = "YES" ]]; then
114+
if [[ $MAKE_NSSTBUFR == "YES" ]]; then
115+
export NSSTBF="${COMOUT}/${OPREFIX}nsstbufr"
116+
else
117+
export NSSTBF="${COMIN_OBS}/${OPREFIX}nsstbufr"
118+
fi
119+
fi
120+
121+
if [[ $MAKE_ACFTBUFR == "YES" ]]; then
122+
export PREPQCPF="${COMOUT}/${OPREFIX}prepbufr.acft_profiles"
123+
else
124+
export PREPQCPF="${COMIN_OBS}/${OPREFIX}prepbufr.acft_profiles"
125+
fi
113126

114127
# Guess Bias correction coefficients related to control
115128
export GBIAS=${COMIN_GES_CTL}/${GPREFIX}abias

0 commit comments

Comments
 (0)