Skip to content

Commit

Permalink
Updates to run "base" s2s-prototype4 (#80)
Browse files Browse the repository at this point in the history 
* Pointing to v3.1 tag for ufs-s2s-model
making readme consistant with user.yaml.default

* Squashing commits to wave2global

* Fix to JWAVE_PREP to look back a day for rtofs

* Moving standalone fv3 model_config exglobal_fcst block into if/else/fi cplwav model_config block. Reinstating config.wave block in JGLOBAL_FORECAST. Pointing EXECwave to HOMEgfs/exec directory for WW3 util executables (changed link_fv3gfs.sh accordingly). Removing debug options from compile.sh line in build_fv3.sh.

* Clean up exwave_post_sbs

* updates to checkout, build and link to run prototype 4

* making Externals_coupled.cfg consistent with checkout.sh

* removing fix from .gitignore

* fixed error when linking on hera

* replace MOM6IC with BASE_CPLIC for base directory location of
IC for s2s model and made it platform dependent.
(Orion location is temporary and needs to be replaced, can also be
changed in cases/coupled_free_forecast.yaml to users prefered location)

* updates for copying ICs

* more updates for ICs

* updates for new fix (which includes prep for multiple resolutions) and some clean-up

* renaming exe to reflect updated build

* update diag_table for ocean

* updates for ice_in to match prototype 4

* updates for CCPP suite used in p4

* making a fix dir of all available ccpp suite files
copying suite xml file to rundirectory

* updating for reorganized tar files, pulling in changes from wcoss-p2 workflow

* last of archive updates for p4

* updating module files to match s2s model and ncep post modules

* we don't want to cycle/have 4 runs per day so setting the cases to be just 1 IC

* updating module files on hera

* updates to readme for orion

* mediator cold start succeeds on hera and forecast job starts
post jobs for atmosphere are no longer triggered on cold starts

* adding ncks, updating ocean post, some fixes for tar files

* updates for tar files and do not keep data

* fix typo

* updating module_base.orion that allows cold start to run.
Not sure if other jobs will now fail though or if what is in your
.bashrc file will effect the success/failure of the run.

* both forecast and atm post seem to be running okay on orion

* adding nems.configure for waves

Co-authored-by: Henrique Alves <henrique.alves@noaa.gov>
Co-authored-by: kate.friedman <kate.friedman@noaa.gov>
Co-authored-by: Jose-Henrique Alves <47567389+ajhenrique@users.noreply.github.com>
  • Loading branch information
@JessicaMeixner-NOAA @ajhenrique @KateFriedman-NOAA
4 people authored Jun 5, 2020
1 parent 4d815e9 commit bf34181
Show file tree
Hide file tree
Showing 61 changed files with 7,538 additions and 523 deletions.
2 changes: 1 addition & 1 deletion .gitignore
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
__pycache__
exec/
fix/
fix
fix/0readme
fix/fix_am
fix/fix_chem
Expand Down
4 changes: 2 additions & 2 deletions Externals_coupled.cfg
View file
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
[ufs-s2s-model]
tag = v3.1
tag = s2s_prototype4.0
local_path = sorc/fv3_coupled.fd
repo_url = https://github.com/ufs-community/ufs-s2s-model
protocol = git
required = True

[EMC_post]
hash = ba7e59b2
tag = upp_gfsv16_release.v1.0.8
local_path = sorc/gfs_post.fd
repo_url = https://github.com/NOAA-EMC/EMC_post.git
protocol = git
Expand Down
12 changes: 12 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,10 @@ sh build_ncep_post.sh #This command will build ncep_post.
sh build_fv3_coupled.sh #This command will build ufs-s2s-model
To link fixed files and executable programs for the coupled application:
On Hera:
sh link_fv3gfs.sh emc hera coupled
On Orion:
sh link_fv3gfs.sh emc orion coupled
```

## Create CROW user file
Expand Down Expand Up @@ -56,6 +59,15 @@ mkdir -p $EXPROOT
or
./setup_case.sh -p HERA ../cases/coupled_free_forecast.yaml test_3d
For Orion:
First make sure you have python loaded:
module load contrib
module load rocoto
module load intelpython3
and then replace ORION with HERA in the commands above.
```
This will create a experiment directory ($EXPERIMENT_DIRECTORY). In the current example, $EXPERIMENT_DIRECTORY=$EXPROOT/test_3d.

Expand Down
93 changes: 93 additions & 0 deletions jobs/JWAVE_INIT
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
#!/bin/bash

date
export PS4=' $SECONDS + '
set -x -e

#############################
# Source relevant config files
#############################
configs="base wave waveinit"
export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
for config in $configs; do
. $config_path/config.$config
status=$?
[[ $status -ne 0 ]] && exit $status
done

##########################################
# Source machine runtime environment
##########################################
. $HOMEgfs/env/${machine}.env waveinit
status=$?
[[ $status -ne 0 ]] && exit $status

# PATH for working directory
export NET=${NET:-gfs}
export RUN=${RUN:-gfs}
export COMPONENTwave=${COMPONENTwave:-${RUN}wave}

export HOMEgfs=${HOMEgfs:-$NWROOT/${NET}.${gfs_ver}}

# Add default errchk = err_chk
export errchk=${errchk:-err_chk}

# Set HOMEwave to HOMEgfs
HOMEwave=${HOMEwave:-${HOMEgfs}}

# Create and go to DATA directory
export DATA=${DATA:-${DATAROOT:?}/${jobid}}
mkdir -p $DATA
cd $DATA

cyc=${cyc:-00}
export cycle=${cycle:-t${cyc}z}

# Set PDY
setpdy.sh
sh ./PDY

export pgmout=OUTPUT.$$

export MP_PULSE=0

# Set resources to propagate NTASKS across cfp call
NTASKS=${NTASKS:-${npe_node_waveinit}}
export NTASKS=${NTASKS:?NTASKS required to be set}

# Path to HOME Directory
export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
export EXECwave=${EXECwave:-$HOMEwave/exec}
export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
export USHwave=${USHwave:-$HOMEwave/ush}
export EXECcode=${EXECcode:-$CODEwave/exec}

# Set COM Paths and GETGES environment
export COMINwave=${COMINwave:-${ROTDIR:?}}
export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
export COMOUT=${COMOUT:-${COMOUTwave}/${COMPONENTwave}.${PDY}/${cyc}}
[[ ! -d $COMOUT ]] && mkdir -m 775 -p $COMOUT

if [ $SENDCOM = YES ]; then
mkdir -p $COMOUT/rundata
fi

export wavelog=${COMOUTwave}/wave.log

# Set mpi serial command
export wavempexec=${wavempexec:-"mpirun -n"}
export wave_mpmd=${wave_mpmd:-"cfp"}

# Execute the Script
$HOMEwave/scripts/exwave_init.sh

# Remove temp directories
if [ "$KEEPDATA" != "YES" ]; then
cd $DATAROOT
rm -rf $DATA
fi
date

117 changes: 117 additions & 0 deletions jobs/JWAVE_POST_SBS
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
#!/bin/bash

date
export PS4=' $SECONDS + '
set -x -e

#############################
# Source relevant config files
#############################
configs="base wave wavepostsbs"
export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
for config in $configs; do
. $config_path/config.$config
status=$?
[[ $status -ne 0 ]] && exit $status
done

##########################################
# Source machine runtime environment
##########################################
. $HOMEgfs/env/${machine}.env wavepostsbs
status=$?
[[ $status -ne 0 ]] && exit $status

# PATH for working directory
export NET=${NET:-gfs}
export RUN=${RUN:-gfs}
export COMPONENTwave=${COMPONENTwave:-${RUN}wave}

export HOMEgefs=${HOMEgefs:-$NWROOT/$NET.${gefs_ver}}
export HOMEgfs=${HOMEgfs:-$NWROOT/$NET.${gfs_ver}}

# Add default errchk = err_chk
export errchk=${errchk:-err_chk}

# Set HOMEwave to HOMEgefs
HOMEwave=${HOMEwave:-${HOMEgfs}}

# Set resources to propagate NTASKS across cfp call
NTASKS=${NTASKS:-${npe_node_waveprep}}
export NTASKS=${NTASKS:?NTASKS required to be set}

# Create and go to DATA directory
export DATA=${DATA:-${DATAROOT:?}/${jobid}}
mkdir -p $DATA
cd $DATA

export cyc=${cyc:-00}
export cycle=${cycle:-t${cyc}z}

# Set PDY
setpdy.sh
sh ./PDY

export pgmout=OUTPUT.$$

export MP_PULSE=0

# Path to HOME Directory
export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
export EXECwave=${EXECwave:-$HOMEwave/exec}
export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
export USHwave=${USHwave:-$HOMEwave/ush}
export EXECcode=${EXECcode:-$CODEwave/exec}

# Set COM Paths and GETGES environment
export COMINwave=${COMINwave:-${ROTDIR:?}}
export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
export COMOUT=${COMOUT:-${COMOUTwave}/${COMPONENTwave}.${PDY}/${cyc}}

export COMINice=${COMINice:-${COMROOTp2}/omb/prod}
export COMINwnd=${COMINwnd:-${COMROOT}/gfs/prod}
export COMIN_WAV_CUR=${COMIN_WAV_CUR:-${COMROOTp2}/rtofs/prod}

mkdir -p $COMOUT/gridded
mkdir -p $COMOUT/station
mkdir -p $COMOUT/stats

export wavelog=${COMOUTwave}/wave.log

# Set mpi serial command
export wavempexec=${wavempexec:-"mpirun -n"}
export wave_mpmd=${wave_mpmd:-"cfp"}

env | sort

# Set wave model ID tag to include member number
# if ensemble; waveMEMB var empty in deterministic
# Set wave model ID tag to include member number
# if ensemble; waveMEMB var empty in deterministic
membTAG='p'
if [ "${waveMEMB}" == "00" ]; then membTAG='c'; fi
export membTAG
export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}

export CFP_VERBOSE=1

# Execute the Script
$HOMEwave/scripts/exwave_post_sbs.sh
err=$?
if [ $err -ne 0 ]; then
msg="FATAL ERROR: ex-script of GWES_POST failed!"
else
msg="$job completed normally!"
fi
postmsg "$jlogfile" "$msg"

# Remove temp directories
if [ "$KEEPDATA" != "YES" ]; then
cd $DATAROOT
rm -rf $DATA
fi
date

126 changes: 126 additions & 0 deletions jobs/JWAVE_PREP
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
#!/bin/bash

date
export PS4=' $SECONDS + '
set -x -e

#############################
# Source relevant config files
#############################
configs="base wave waveprep"
export EXPDIR=${EXPDIR:-$HOMEgfs/parm/config}
config_path=${EXPDIR:-$NWROOT/gfs.${gfs_ver}/parm/config}
for config in $configs; do
. $config_path/config.$config
status=$?
[[ $status -ne 0 ]] && exit $status
done

##########################################
# Source machine runtime environment
##########################################
. $HOMEgfs/env/${machine}.env waveprep
status=$?
[[ $status -ne 0 ]] && exit $status

# PATH for working directory
export NET=${NET:-gfs}
export RUN=${RUN:-gfs}
export COMPONENTwave=${COMPONENTwave:-${RUN}wave}

export HOMEgfs=${HOMEgfs:-$NWROOT/gfs.${gfs_ver}}

# Add default errchk = err_chk
export errchk=${errchk:-err_chk}

# Set HOMEwave to HOMEgfs
HOMEwave=${HOMEwave:-${HOMEgfs}}

# Set resources to propagate NTASKS across cfp call
NTASKS=${NTASKS:-${npe_node_waveprep}}
export NTASKS=${NTASKS:?NTASKS required to be set}

# Create and go to DATA directory
export DATA=${DATA:-${DATAROOT:?}/${jobid}}
mkdir -p $DATA
cd $DATA

cyc=${cyc:-00}
export cycle=${cycle:-t${cyc}z}

# Set PDY
setpdy.sh
sh ./PDY
# Set rtofs PDY
export RPDY=$PDY

export pgmout=OUTPUT.$$

export MP_PULSE=0

# CDO required for processing RTOFS currents
# export CDO=${COMROOTp2}/nwprod/rtofs_glo.v1.2.0/bin/cdo
export CDO=/gpfs/dell2/emc/verification/noscrub/Todd.Spindler/CDO/bin/cdo

# Path to HOME Directory
export CODEwave=${CODEwave:-${HOMEfv3gfs}/WW3}
export EXECwave=${EXECwave:-$HOMEwave/exec}
export FIXwave=${FIXwave:-$HOMEwave/fix/fix_wave_${NET}}
export PARMwave=${PARMwave:-$HOMEwave/parm/wave}
export USHwave=${USHwave:-$HOMEwave/ush}
export EXECcode=${EXECcode:-$CODEwave/exec}

###################################
# Set COM Paths and GETGES environment
###################################
# Set COM Paths and GETGES environment
export COMINwave=${COMINwave:-${ROTDIR:?}}
export COMOUTwave=${COMOUTwave:-${ROTDIR:?}}
export COMIN=${COMIN:-${COMINwave}/${COMPONENTwave}.${PDY}/${cyc}}
export COMOUT=${COMOUT:-${COMOUTwave}/${COMPONENTwave}.${PDY}/${cyc}}
[[ ! -d $COMOUT ]] && mkdir -m 775 -p $COMOUT

if [ $RUN_ENVIR = "nco" ]; then
export COMIN_WAV_ICE=${COMIN_WAV_ICE:-$(compath.py gfs/prod)}/${CDUMP}.${PDY}/${cyc}
export COMIN_WAV_WND=${COMIN_WAV_WND:-$(compath.py gfs/prod)}/${CDUMP}.${PDY}/${cyc}
export COMIN_WAV_CUR=${COMIN_WAV_CUR:-$(compath.py ${WAVECUR_DID}/prod)}/${WAVECUR_DID}.${RPDY}
if [ ! -d $COMIN_WAV_CUR ]; then
export RPDY=`$NDATE -24 ${PDY}00 | cut -c1-8`
export COMIN_WAV_CUR=$(compath.py ${WAVECUR_DID}/prod)/${WAVECUR_DID}.${RPDY}
fi
else
if [ ! -d $DMPDIR/${WAVECUR_DID}.${RPDY} ]; then export RPDY=`$NDATE -24 ${PDY}00 | cut -c1-8`; fi
if [ ! -L $ROTDIR/${WAVECUR_DID}.${RPDY} ]; then # Check if symlink already exists in ROTDIR
$NLN $DMPDIR/${WAVECUR_DID}.${RPDY} $ROTDIR/${WAVECUR_DID}.${RPDY}
fi
$NLN $DMPDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE} $ROTDIR/$CDUMP.${PDY}/$cyc/${WAVICEFILE}
export COMIN_OBS=${COMIN_OBS:-$ROTDIR/$RUN.$PDY/$cyc}
export COMIN_WAV_ICE=${COMIN_OBS}
export COMIN_WAV_WND=${COMIN_OBS}
export COMIN_WAV_CUR=${ROTDIR}/${WAVECUR_DID}.${RPDY}
fi

if [ $SENDCOM = YES ]; then
mkdir -p $COMOUT
fi

export wavelog=${COMOUTwave}/wave.log

# Set mpi serial command
export wavempexec=${wavempexec:-"mpirun -n"}
export wave_mpmd=${wave_mpmd:-"cfp"}

# Set wave model ID tag to include member number
# if ensemble; waveMEMB var empty in deterministic
export WAV_MOD_TAG=${COMPONENTwave}${waveMEMB}

# Execute the Script
$HOMEwave/scripts/exwave_prep.sh

# Remove temp directories
if [ "$KEEPDATA" != "YES" ]; then
cd $DATAROOT
rm -rf $DATA
fi
date

Loading

0 comments on commit bf34181

Please sign in to comment.