Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Limited RAVE Emissions for NA13km Domain on WCOSS2 #20

Closed
drnimbusrain opened this issue Oct 12, 2022 · 57 comments
Closed

Limited RAVE Emissions for NA13km Domain on WCOSS2 #20

drnimbusrain opened this issue Oct 12, 2022 · 57 comments

Comments

@drnimbusrain
Copy link

drnimbusrain commented Oct 12, 2022 •
edited
Loading

@chan-hoo @JianpingHuang-NOAA @KaiWang-NOAA

The ability to run Online-CMAQ on WCOSS2 is indeed great news! Thank you!

However, I noticed that there is only a very limited RAVE emissions dataset available on WCOSS2 for running the NA13 km domain:  
/lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT
20190805 20190806 20190807

Can the entire dataset for the NA13 domain be brought over from Hera to WCOSS2 soon for longer testing? Specifically, the data for August-September 2020 case would be excellent.

Thank you.
@drnimbusrain drnimbusrain changed the title Missing RAVE Emissions for NA13km Domain on WCOSS2 Limited RAVE Emissions for NA13km Domain on WCOSS2 Oct 12, 2022
@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Oct 12, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA Thank you for your reply. I am running a base test case for September 2020, and have moved the data manually to my own WCOSS2 location. However, it of course would be good to similarly house all Online-CMAQ inputs on WCOSS2 the same as on Hera.

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 13, 2022 via email

@drnimbusrain
Copy link
Author

@KaiWang-NOAA Thank you, as we indeed did not have the latest version of RAVE data. Can those files please be made available at the WCOSS location?

AQM_FIRE_DIR: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT
AQM_FIRE_FILE: Hourly_Emissions_regrid_NA_13km
AQM_FIRE_FILE_SUFFIX: _h72.nc

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 13, 2022 via email

@drnimbusrain
Copy link
Author

Thank you @KaiWang-NOAA

@drnimbusrain
Copy link
Author

@KaiWang-NOAA It seems your RAVE emissions have been moved or deleted from this location, and our Sep 2020 runs are failing on WCOSS2 again. Can you update us on this?

@bbakernoaa

@bbakernoaa
Copy link
Contributor

@KaiWang-NOAA @HaixiaLiu-NOAA We really need to get this sorted out. We cannot have emission files moving directories without letting everyone know.

We really need to think about moving all emissions (anthropogenic) to a fix directory and the fire emissions to a static directory and/or HPSS and retrieve them in the workflow. This would cut down on errors like this drastically.

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 24, 2022 via email

@drnimbusrain
Copy link
Author

@KaiWang-NOAA Why is it in "RAVE_CONUS"? We are ONLY running the larger NA domain. Where are those files?

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 24, 2022 via email

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 24, 2022 via email

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Oct 25, 2022 via email

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 1, 2022 •
edited
Loading

@KaiWang-NOAA Thank you for putting all RAVE emissions there.

However, at some point hopefully @chan-hoo (or whoever has write access) can move all RAVE emissions to the following Online-CMAQ WCOSS2 location for common emissions: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT We really shouldn't have to do extra step to pull down emissions from HPSS and store locally for the official workflow on each system, correct?

FYI. The current workflow and aqm.rc template needs the emissions stored in YYYYMMDD sub-directory format here too. See: https://github.com/ufs-community/ufs-srweather-app/blob/online-cmaq/ush/create_aqm_rc_file.py#L67

@chan-hoo
Copy link

chan-hoo commented Nov 1, 2022

@drnimbusrain, due to the space issue, I put some specific data for the engineering test (sample_config/WE2E_test) in the 'noscrub/RRFS_CMAQ' directory. Otherwise, I'll receive a warning from the system administrator. I don't have permission to write data files in the centralized locations on HPCs.

@drnimbusrain
Copy link
Author

@chan-hoo Thank you for clarifying. Seems this is a larger issue then with WCOSS2 for us then. @bbakernoaa

@drnimbusrain
Copy link
Author

@KaiWang-NOAA @JianpingHuang-NOAA @chan-hoo @HaixiaLiu-NOAA

We have been continuing our online-cmaq NA domain run for Sep 2020 with latest model and RAVE fire emissions provided by @KaiWang-NOAA . While most of the areas of CONUS look OK after spin-up, there seems to be an issue with the input fire emissions magnitude and extremely low PM2.5 in the west. You can see our regional analysis here:
https://docs.google.com/presentation/d/1AKhSlbVjPsy9aIBVyPRUYwQfoEgtCe4tFwxtBT4neLE/edit?usp=sharing

For example, while the non-fire regions look OK in the northeast:
image

The fire regions in the west and CA appear to account for the RAVE emissions, but they are much too low:
image

The spatial mean bias plots for PM2.5 also indicate an issue in the western fire regions:
image

RAVE fire emissions are being read from: /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_NA/202009

@bbakernoaa
Copy link
Contributor

@KaiWang-NOAA from the figures you can see that the emissions are being read in and processed but the magnitude is very low. This indicates that the units may be different than indicated in the aqm.rc. We need to get to the bottom of this for the NEW fire emission product and not the test data previously used.

@KaiWang-NOAA
Copy link

@bbakernoaa @drnimbusrain @JianpingHuang-NOAA I compiled several slides that I did before when QA the RAVE data regridding. https://docs.google.com/presentation/d/1Xq8rwSXsqwRwOujKaypH8VbM9JtA0y70Vfa5XwBKxSI/edit#slide=id.g1841e488be7_0_193. See slides 6 and 8 for the fluxes with the units consistent with the model ready emissions. It looks to be ok for me. We may need to look further at what exactly cause the low PM2.5

@KaiWang-NOAA
Copy link

@bbakernoaa @bbakernoaa Here are some slides that Jianping presented at the end of Sept. which covers the same Sept. 2020 episode. https://docs.google.com/presentation/d/12Jse29OvM4eqcB6Dkev1U_N2SCkSh0Pw/edit#slide=id.g15c34760966_0_118
c8 used the same set of RAVE emissions with cold start from Sept. 1, 2020. As you can see on slide 4, the PM2.5 concentration kept building up, which is much different than your results.

I'm not sure if @JianpingHuang-NOAA still has the run folder with related configure/aqm.rc files. It's worth looking into it.

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA This would be very helpful if you could provide any information on your Sample Config or aqm.rc file for your previous run. While you PM2.5 is still very low (purple line CONUS ave ~ 10 ug/m3 by Sep 18) and likely not doing well in the fire regions (although there is no regional analysis for R9 for example):
image

It is indeed about 2x higher than my run (green line CONUS ave ~ 5 ug/m3 by Sep 18)
image

However @KaiWang-NOAA , I wouldn't say it is "much different than our results", as the 2x higher PM2.5 is still likely much lower than the fire regions PM2.5 during this period, and thus indicates potentially a similar problem with RAVE fire emissions in both our runs. We really should discuss to get to the bottom of this.

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 7, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA Yes, thank you for your comments. I agree there is some issue in RAVE fire emissions during this time, but not sure of the cause. It seems a larger issue than what has initially been suggested. If you look at the region R9 or R10, or really any of our analysis, it is not even realistic:

image

image

@bbakernoaa
Copy link
Contributor

@JianpingHuang-NOAA it would be helpful if you could repeat this experiment using the latest code and files to make sure that we are not setting this up incorrectly. What we are seeing here is a massive under prediction of fires that is beyond what has been shown in any of our simulations with online or offline versions. Again we do see a signal but it is far below what we expect and far away from the source region everything looks more reasonable. Please repeat even a small section of this experiment. It would drastically help settle this issue

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 8, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA Thank you for having a check. My current test run continuing on in Sep 18-20, 2020, is found here: /lfs/h2/oar/ptmp/Patrick.C.Campbell/expt_dirs/aqm_cold_aqmna13_1day_arl_base_wetdepfix_firex1000_20200918_20200920

Please ignore in the aqm.rc file that I am testing an arbitrary increase (x1000) in fire emissions to gauge the impact on the extremely low PM2.5 concentrations.

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 9, 2022 •
edited
Loading

@JianpingHuang-NOAA @KaiWang-NOAA You can see how much better our operational model with GBBEPx is doing compared to my online-cmaq run with RAVE for R9 and R10:

image

@JianpingHuang-NOAA Do you see any issues with my aqm.rc or config.yaml files for my run? From my vantage point, it seems like the RAVE 2020 emissions do not even see the major fires in the west, as increasing the PM2.5 emissions by 1000 (after Sep 18th in my run) does not even make an impact on PM2.5 concentrations.

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 9, 2022 via email

@Ho-ChunHuang-NOAA
Copy link

is opr NAM-CMAQ or current operational CMAQ? in Sept 2020 NAM-CMAQ is the operational AQM. GFS-CMAQ is the operational AQM after July 2021.

If NAm-CMAQ then it is HMS-based Bluesky fire emission
If GFS-CMAQ then it is daily GBBEPx fire emissions with prescribed diurnal profile for PM25 and FRP+ wild-fire only with spatial filter.

@Ho-ChunHuang-NOAA
Copy link

@JianpingHuang-NOAA @drnimbusrain
Sorry I reply in github issue directly, hope you received notice.

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 9, 2022 •
edited
Loading

@JianpingHuang-NOAA @Ho-ChunHuang-NOAA The "opr" here is the current operational model, NACC-CMAQ. This as you know uses GBBEPx fire emissions with prescribed diurnal profile.

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 9, 2022 •
edited
Loading

@JianpingHuang-NOAA I also do not think this issue necessarily has to do with the anthropogenic emissions, as other regions outside of fires look OK for PM2.5, and the ozone looks reasonable in our runs.
Outside of the west the PM2.5 looks reasonable, such as the non-fire northeast R2 (albeit slightly underpredicted)
image
image
Ozone CONUS looks reasonable (albeit overpredicted)
image

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 9, 2022 via email

@bbakernoaa
Copy link
Contributor

@JianpingHuang-NOAA please put the problem here so that everyone can see it. if you know what the issue is we all need to know.

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 9, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA This is not clear as to what the issue may be with anthropogenic emissions in my run, and does not seem related to potential wildfire emissions issue. Can you please further clarify/show here this issue you found?

@bbakernoaa
Copy link
Contributor

@drnimbusrain I completely agree. This does not seem to have anything related to the anthropogenic emissions. It is clearly something with the wildfire emissions

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 9, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA There is no figure here to reference. Please further clarify.

@bbakernoaa
Copy link
Contributor

@KaiWang-NOAA can you help out out and repeat this simulation? We really need an independent verification of this.

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 10, 2022 via email

@bbakernoaa
Copy link
Contributor

@JianpingHuang-NOAA Nothing has really changed in the nexus step. I don't understand what you are showing here or what you think should be happening. There is no details at all to what you mention may be a problem.

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 10, 2022 via email

@bbakernoaa
Copy link
Contributor

@JianpingHuang-NOAA please create a separate issue for this. It is not answering problems related to the issue described above

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Nov 10, 2022 via email

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 10, 2022 •
edited
Loading

@KaiWang-NOAA

I agree there are two separate issues, and disagreed with @JianpingHuang-NOAA adding the NEXUS anthropogenic issue here. Hopefully, he will open up a separate issue on that.

For the main issue (1) about RAVE emissions test in our Sep 2020 retro case run. I understand that you have done QA/QC on the units/regridding. However, if they are correct, and the online-cmaq workflow reads them in "correctly", I disagree with the following consensus with your discussion with Fangjun and Xiaoyang: "The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation."

I think we would all agree that the online-cmaq PM2.5 predictions in the west during the strong fires are not even realistic in my runs, and such underpredictions cannot be attributed to long-range transport issues. For example, as shown below, the operational (NACC-CMAQ) model with GBBEPx appears to do much better than RAVE (which appears to not even see the fires):
image

If either you or @JianpingHuang-NOAA can do an independent test of the same RAVE Sep 2020 emissions for this case, and you still see such an unrealistic PM2.5 prediction in the west, I think we really need to involve others (including Fangjun and Xiaoyang) again for another discussion.

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Nov 11, 2022 via email

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 11, 2022 via email

@KaiWang-NOAA
Copy link

KaiWang-NOAA commented Nov 14, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA @KaiWang-NOAA Thank you for performing your independent Sep 2020 case test. I agree that your run looks much better for PM2.5 in the western fire regions compared to our online-cmaq run, with the understanding that the major PM2.5 from fires in the west are still underpredicted (but not as bad as our run).

I assume Jianping's run is also a cold start, where the PM2.5 does quickly spin-up and reach background pretty quickly (~10 days). Our run does not spin-up in 10-days, and the PM2.5 stays flat for some reason. Can I see the aqm.rc and config.yaml files for your run? I am suspicious about how the runs are setup, which may suggest a separate problem in retro runs.

For the NACC-CMAQ comparison in your analysis, I would use the Para6d runs for this case (identical to the operational) found here on Hera: /scratch1/NCEPDEV/stmp4/Patrick.C.Campbell/emc-para6d

Thanks again.

@bbakernoaa
Copy link
Contributor

@JianpingHuang-NOAA Also if any changes were made to any of the input.nml files etc we need to know that. It should all be committed back to the repository so that all of us can do the simulations.

@bbakernoaa
Copy link
Contributor

bbakernoaa commented Nov 14, 2022 •
edited
Loading

@KaiWang-NOAA @JianpingHuang-NOAA Are these using the exact same fire emissions that @drnimbusrain did? or is this using the old conus only input files. There is no context here and we need to understand what is happening. Only one of us can actually run this model at this point and its extremely worrying

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 14, 2022 •
edited
Loading

@bbakernoaa Yes, that is another excellent point I didn't mention. Please @JianpingHuang-NOAA @KaiWang-NOAA , please point us to the locations of the RAVE 2020 NA emissions and config.yaml/aqm.rc configurations used in this run so we can compare them.

Patrick's Online-CMAQ RAVE 2020 Emissions for NA Domain (WCOSS Run): /lfs/h2/oar/arl/noscrub/Patrick.C.Campbell/RAVE_RT/G793
config.yaml file:
/lfs/h2/oar/arl/noscrub/Patrick.C.Campbell/run_na_test_ufs-srw-app-arl-base_wetdepfix/ush/config.yaml
aqm.rc file:
/lfs/h2/oar/arl/noscrub/Patrick.C.Campbell/run_na_test_ufs-srw-app-arl-base_wetdepfix/parm/aqm.rc

Jianping's Online-CMAQ RAVE 2020 Emissions for NA Domain:
?

@JianpingHuang-NOAA
Copy link
Collaborator

JianpingHuang-NOAA commented Nov 14, 2022 via email

@drnimbusrain
Copy link
Author

@JianpingHuang-NOAA Thank you. I do not fully disagree, and yes it certainly is possible the NEXUS-generated issue that persists has some impact, but we thought it best to document as a separate issue. I also just wanted to make sure we have the latest directories from you to compare these runs, since we are seeing different results. Thank you for simulating the case and for providing more details.

@KaiWang-NOAA
Copy link

Are these using the exact same fire emissions that @drnimbusrain did?
@bbakernoaa @drnimbusrain, the RAVE fire files that Jianping used are in /lfs/h2/emc/physics/noscrub/jianping.huang/data/RRFS_CMAQ/emissions/GSCE/RAVE.in.C793/RAVE_RT/202009*, which are identical to those under my folder at /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_NA/202009/

@drnimbusrain
Copy link
Author

drnimbusrain commented Nov 14, 2022 •
edited
Loading

@JianpingHuang-NOAA @KaiWang-NOAA @bbakernoaa

I confirm that our RAVE 2020 emissions inputs are identical, and our aqm.rc files are the same for the appropriate settings.

There are some differences in the config.yaml file, but I notice that you pointed me to your NRT config.yaml. Is this identical to @JianpingHuang-NOAA 's retro Sep 2020 run case just performed? Below is the diff for our config.yaml files:

config.yaml-diff.txt

BrianCurtis-NOAA pushed a commit to BrianCurtis-NOAA/AQM that referenced this issue Feb 25, 2025
@drnimbusrain
Merge pull request NOAA-EMC#20 from lwcugb/feature/cmaq54_li
5803ed7 
Feature/cmaq54 li
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@bbakernoaa @drnimbusrain @Ho-ChunHuang-NOAA @JianpingHuang-NOAA @chan-hoo @KaiWang-NOAA