Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update WAFS scripts and file names #24

Merged
merged 3 commits into from
May 24, 2024
Merged

Update WAFS scripts and file names #24

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4307cac
1. Remove gfs from all script filenames
May 24, 2024
87b37c1
Add sorc/checkout_upp.sh
May 24, 2024
00547ae
Update sorc/checkout_upp.sh
YaliMao-NOAA May 24, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 10 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,15 +3,22 @@ Software necessary to generate WAFS products

To install:

Clone repository (recursively with UPP, if desired):
Clone repository
```bash
git clone --recursive https://github.com/NOAA-EMC/WAFS
git clone https://github.com/NOAA-EMC/WAFS
```

Move into desired branch and then run:

Clone submodule and sub-submodule repository (including upp and upp/sorc/post_gtg):
(gtg code is UCAR private, access needs to be authorized)
```bash
./sorc/build_all.sh
sh sorc/checkout_upp.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you forget to add, commit and push this file?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry. I thought I would modify it later when adding UPP. Just added it.

```

Compile the executable files:
```bash
sh sorc/build_all.sh
```

`build_all.sh` will build WAFS executables and offline UPP executable.
Expand Down
147 changes: 70 additions & 77 deletions dev/driver/driver_WAFS.README
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,111 +2,104 @@
How to use the drivers for WAFS jobs
*********************************************************************

There are 4 WAFS jobs and their corresponding drivers
JGFS_ATMOS_WAFS <-> run_JGFS_ATMOS_WAFS.dell
JGFS_ATMOS_WAFS_GRIB2 <-> run_JGFS_ATMOS_WAFS_GRIB2.dell
JGFS_ATMOS_WAFS_BLENDING <-> run_JGFS_ATMOS_WAFS_BLENDING.dell
JGFS_ATMOS_WAFS_GCIP <-> run_JGFS_ATMOS_WAFS_GCIP.dell
JGFS_ATMOS_WAFS_GRIB2_0P25 <-> run_JGFS_ATMOS_WAFS_GRIB2_0P25.dell
JGFS_ATMOS_WAFS_BLENDING_0P25 <-> run_JGFS_ATMOS_WAFS_BLENDING_0P25.dell
There are 1 standalone UPP job and 5 WAFS jobs
run_JWAFS_UPP.wcoss2
run_JWAFS_GRIB.wcoss2
run_JWAFS_GCIP.wcoss2
run_JWAFS_GRIB2.wcoss2
run_JWAFS_GRIB2_0P25.wcoss2
run_JWAFS_BLENDING_0P25.wcoss2

JWAFS_UPP is a downstream of GFS model outputs in netcdf

JGFS_ATMOS_WAFS_BLENDING is a downstream of JGFS_ATMOS_WAFS_GRIB2 and
should run after JGFS_ATMOS_WAFS_GRIB2
JWAFS_GRIB is a downstream of GFS pgrb outputs

JGFS_ATMOS_WAFS_BLENDING_0P25 is a downstream of JGFS_ATMOS_WAFS_GRIB2_0P25 and
should run after JGFS_ATMOS_WAFS_GRIB2_0P25
JWAFS_GCIP is a downstream of GFS master files, also has satellite and
radar observation data inputs, and METAR SHIP PIREPs data which are dumped
from DCOMROOT by using 'dumpjb'

JGFS_ATMOS_WAFS_GCIP has satellite and radar data inputs, and METAR SHIP PIREPs
data which are dumped from DCOMROOT by using 'dumpjb'
JWAFS_GRIB2 and JWAFS_GRIB2_0P25 are both a downstream of GFS master files
and JWAFS_UPP outputs

Step 1:
=====================
After a user downloads/checkouts the project tag.
1) cd sorc
2) sh build_wafs.sh

Project location, defined by HOMEgfs
export NWROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/git
export HOMEgfs=${HOMEgfs:-${NWROOT}/EMC_wafs_branch}
JWAFS_BLENDING_0P25 is a downstream of UK data at DCOM and JWAFS_GRIB2_0P25,
and should be run after JWAFS_GRIB2_0P25

A user needs to modify NWROOT or HOMEgfs to the project location
Step 1: source code
=====================
Project location, defined by HOMEwafs
export NWROOT=(the location holds your WAFS local inventory)
export HOMEwafs=${HOMEwafs:-${NWROOT}/{the folder of your WAFS local inventory)}

A user needs to modify NWROOT or HOMEwafs to the project location

Step 2:
Step 2: output and working folder
=====================
Canned model data:
/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/gfs/prod/gfs.20180725/00
Specify COMOUT output folder by setting COMROOT
export COMROOT=(a folder ends with $envir/com)

Change working folder:
export DATA=

So first set the specific data:
export PDY=20180725
export cyc=00
Change file names of error messages and script printout:
#PBS -j oe
#PBS -o (your job log file with full path)

1) For run_JGFS_ATMOS_WAFS.dell, run_JGFS_ATMOS_WAFS_GRIB2.dell,
It's optional for a user to modify the input data location, COMROOT or COMIN:
export COMROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/datainput
export COMIN=$COMROOT/gfs.$PDY/$cyc
Step 3: input data
=====================
Specify date and cycle:
export PDY=(date as YYYYMMDD)
export cyc=(cycle as CC)

2) JGFS_ATMOS_WAFS_BLENDING has two inputs, UK and US.
US data is a downstream product of JGFS_ATMOS_WAFS_GRIB2, so COMINus needs to point to
$COMOUT/wmo of run_JGFS_ATMOS_WAFS_GRIB2.dell:
export COMINus=
Specify COMIN by either setting COMIN or by setting COMPATH used by compath.py
export COMPATH=(location of canned data ending with $envir/com/$RUN structure)
export COMIN=(folder with a full path)

UK data is from DCOM, currently copied DCOM from Phase 1. For canned data, it's at:
/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/dcom/us007003/20180725/wgrbbul/ukmet_wafs
export DCOMROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/dcom
export COMINuk=${COMINuk:-$DCOMROOT/us007003/$PDY/wgrbbul/ukmet_wafs}
1) run_JWAFS_GRIB2.WCOSS2 and run_JWAFS_GRIB2_0P25.WCOSS2
Ensure COMPATH includes both GFS and WAFS

It's optional for a user to change either DCOMROOT or COMINuk to where the input UK data is.
2) run_JWAFS_BLENDING_0P25.wcoss2 has two inputs, UK and US.
US data is a downstream product of JWAFS_GRIB2_0P25,
so either:
export COMPATH=($COMROOT/wafs where COMROOT is of run_JWAFS_GRIB2_0P25.wcoss2)
or:
export COMINus=($COMOUT of run_JWAFS_GRIB2_0P25.wcoss2)

3) JGFS_ATMOS_WAFS_GCIP has 4 inputs: model data, bufr, satellite and radar
UK data is from DCOM,
either:
export DCOMROOT=(a folder canned data with structure of $PDY/wgrbbul/ukmet_wafs)
or:
export COMINuk=${COMINuk:-$DCOMROOT/$PDY/wgrbbul/ukmet_wafs}

Canned model data location:
/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/gfs/prod/gfs.20180725/00
It's optional for a user to modify the input data location, COMROOT or COMINgfs:
export COMROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput
export COMINgfs=$COMROOT/gfs.$PDY/$cyc
3) run_JWAFS_GCIP.wcoss2 has 4 inputs: gfs master files of f000 and f003, bufr, satellite and radar
Ensure COMPATH includes GFS

BUFR data is dumped from DCOM.
export DCOMROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/dcom
dumpjb searches data in b000 b001 b004 b007 under $DCOMROOT/us007003/20180725.
dumpjb searches data in b000 b001 b003 b004 b007 under $DCOMROOT/$PDY
Though it's optional to modify DCOMROOT for dumping BUFR data, it should be mentioned
DCOMROOT of satellite data may be affected.

Satellite data, currently copied from DCOM on Phase1. For canned data, it's at:
/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/dcom/us007003/20161101/mcidas/
It's optional for a user to modify the input data location, DCOMROOT or COMINsat:
export DCOMROOT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/dcom
export COMINsat=${COMINsat:-$DCOMROOT/us007003/$PDY/mcidas}
Satellite data,
either:
export DCOMROOT=(a folder canned data with structure of $PDY/mcidas)
or:
export COMINsat=${COMINsat:-$DCOMROOT/$PDY/mcidas}

Radar data, currently copied from COM on Phase1. For canned data, it's at:
/gpfs/hps/emc/global/noscrub/Yali.Mao/datainput/com/hourly/prod/radar.20161101/
It's optional for a user to modify the input data location, COMROOTp1 or COMINradar
export COMROOTp1=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/datainput/com1
export COMINradar=${COMINradar:-$COMROOTp1/hourly/prod/radar.$PDY}
Radar data,
either ensure COMPATH includes radar,
or:
export COMINradar=${COMINradar:-$COMROOT/radarl2/$radarl2_ver)/radar.$PDY}

Step 3:
=====================
Change working folder:
export DATA=

Change product output folder:
export COMOUT=

Change file names of error messages and script printout:
#BSUB -oo
#BSUB -eo
4) run_JWAFS_UPP.wcoss2 and run_JWAFS_GRIB.wcoss2
Ensure COMPATH includes GFS


*********************************************************************
How to test and test results
*********************************************************************
Test by comparing outputs to operational products in different ways.
1) Use wgrib2 to check date, cycle and forecast hour are correct.
2) Check the number of fields and records are correct by comparing the
1) Compared outputs to the operational.
2) Use wgrib2 to check date, cycle and forecast hour are correct.
3) Check the number of fields and records are correct by comparing the
control files generated by g2ctl
3) Plot and compare by using GrADS to have a sanity check whether
4) Plot and compare by using GrADS to have a sanity check whether
the differences are reasonable.

The test products are saved at:
COMOUT=/gpfs/dell2/emc/modeling/noscrub/Yali.Mao/2018impl/dataprod/gfs.20180725/00
Loading