Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INSTALL] Install hpc-stack with Intel 2021.3.0 and impi 2021.3.0 #378

Open
junwang-noaa opened this issue Feb 3, 2022 · 59 comments
Open

[INSTALL] Install hpc-stack with Intel 2021.3.0 and impi 2021.3.0 #378

junwang-noaa opened this issue Feb 3, 2022 · 59 comments
Assignees
@kgerheiser @climbfuji @Hang-Lei-NOAA

Comments

@junwang-noaa
Copy link

junwang-noaa commented Feb 3, 2022 •
edited
Loading

Note: Please search to see if an issue already exists for the software you are requesting to be installed.

Which software in the stack would you like installed?
The hpc-stack used by ufs-weather-model.

What is the version/tag of the software?
hpc/1.1.0
with Intel 2021.3.0 and impi 2021.3.0, following libraries are required:

jasper/2.0.22
zlib/1.2.11
png/1.6.35

hdf5/1.10.6
netcdf/4.7.4
pio/2.5.2
esmf/8_2_1bs04
fms/2021.03

bacio/2.4.1
crtm/2.3.0
g2/3.4.2
g2tmpl/1.10.0
ip/3.3.3
sp/2.3.3
w3nco/2.4.1

gftl-shared/v1.3.3
yafyaml/v0.5.1
mapl/2.11.0-esmf-8_2_0

Please provide the version/tag of the software to be installed.

What compilation options would you like set?
fms/2021.03 needs to build with avx on hera (see fms/2021.03-avx under hpc/1.1.0 in /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/stack

Which machines would you like to have the software installed?
On hera first, we will test it and confirm it is working. Then it will be installed on jet, gaea, orion, dell, cray and wcoss2

Any other relevant information that we should know to correctly install the software??
Please describe in as much detail as possible.

Additional context
Add any other context or screenshots about the install request here.
@kgerheiser
Copy link
Contributor

FYI I don't see an intel/2021.3 on Orion, just intel/2021.2.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Feb 4, 2022 via email

@junwang-noaa
Copy link
Author

Thanks. Intel 2021.2 is fine, we need to move to Intel 2021.x version.

@junwang-noaa
Copy link
Author

@jkbk2004 FYI.

@jkbk2004
Copy link

Here is the test result of intel2021.2 with the control/butterfly case of -DAPP=ATM. Package options in the intel22021.2 run can be stayed only for the updates on the compiler and cmake. Will continue to follow up.
control-intel21.pptx

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA May I ask if the disk issue is resolved? When hpc-stack with intel/impi 2021 can be installed?

@kgerheiser
Copy link
Contributor

@Hang-Lei-NOAA is on leave this week

@kgerheiser
Copy link
Contributor

@junwang-noaa it was recommended by an RDHPCS admin that we should use Intel 2022. 2021 contains the log4j vulnerability.

@junwang-noaa
Copy link
Author

Sure. If you have the library installed with Intel 2022, we can do some testing with ufs-weather-model. Currently we have one platform (cheyenne) running with Intel 2021.02 and it works fine, we haven't tried Intel 2022 yet.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Feb 28, 2022 via email

@jkbk2004
Copy link

@kgerheiser @junwang-noaa @Hang-Lei-NOAA let me know if intel2022 option is available on orion
. I can quickly launch out RT test.

@kgerheiser
Copy link
Contributor

@Hang-Lei-NOAA just FYI, Hera will be undergoing maintenance tomorrow.

@kgerheiser
Copy link
Contributor

The 2022 compilers aren't available on Orion, but I put in a request to have them installed.

@kgerheiser
Copy link
Contributor

Orion admins are in the process of installing the new compilers.

@junwang-noaa junwang-noaa mentioned this issue Mar 14, 2022
Closed
@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA May I ask if the library installed with Intel 2022 is available on hera? Thanks.

@jkbk2004
Copy link

@junwang-noaa @Hang-Lei-NOAA I tried to install hpc-stack with intel22 on /work/noaa/epic-ps/jongkim/stacks/libs-intel2022.1.2. I may go ahead to do butterfly test on orion first. I am trying to install hpc-stack with intel2022 on Cheynne as well.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 15, 2022 via email

@junwang-noaa
Copy link
Author

junwang-noaa commented Mar 15, 2022 •
edited
Loading

@Hang-Lei-NOAA I got the following error when loading the libraries with intel 2022 on hera in ufs-weather-model develop branch:

module use /scratch2/NCEPDEV/nwprod/hpc-stack/libs/hpc-stack/modulefiles/stack
module load hpc/1.1.0
module load hpc-intel/2022.1.2
module load hpc-impi/2022.1.2

error:
Lmod has detected the following error: These module(s) or extension(s) exist
but cannot be loaded as requested: "jasper/2.0.22",
"mapl/2.11.0-esmf-8.2.1b04", "g2/3.4.2", "esmf/8.2.1b04", "pio/2.5.2",
"gftl-shared/v1.3.3", "crtm/2.3.0"

Would you please install those libraries with intel 2022? Those modules are currently used in ufs-weather-model develop branch, we hope to have a clean compiler update before changing any of the libraries. Thanks

@jkbk2004
Copy link

@junwang-noaa intel2022 hpc-stack on Cheyenne: /glade/scratch/jongkim/stacks/libs-intel2022.1.2/modulefiles/stack. I will run the butterfly test on both Orion and Cheyenne.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 15, 2022 via email

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA We need to evaluate the impact of intel compiler version change before we can use the modules in ufs-weather-model. I do see new versions of jasper, g2, pio and crtm, do you know if they will change results compared to previous versions? If yes, then we still need the previous versions.

Also the gftl-shared/v1.3.3, esmf/8_2_1bs04 and MAPL/2.11.o are still missing.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 15, 2022 via email

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 16, 2022 via email

@jkbk2004
Copy link

@junwang-noaa @Hang-Lei-NOAA I am checking the difference of intel2021 and intel2022 on Cheyenne. The result seems to be identical. It is interesting. I will double check. Orion que is very slow again.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Mar 16, 2022 via email

@jkbk2004
Copy link

@Hang-Lei-NOAA Yeah, it seems like that's the case.

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA May I ask if the Intel 2022 is available on dell?

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Apr 18, 2022 via email

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA This means we will drop ufs-weather-model develop branch on dell any more when we move to Intel 2022. @arunchawla-NOAA FYI.

@arunchawla-NOAA
Copy link

Intel 2022 is not available on wcoss2, we will only have intel 19 there initially
Does ufs-weather-model need intel 2022 in develop ?

@junwang-noaa
Copy link
Author

junwang-noaa commented Apr 19, 2022 via email

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Apr 19, 2022 via email

@junwang-noaa
Copy link
Author

@arunchawla-NOAA Can we ask GRIT to install Intel/2022.1 on wcoss2?

@edwardhartnett
Copy link
Contributor

@kgerheiser and @climbfuji did you have any trouble building HDF5 with the intel compilers? One user has reported problems with this on the netCDF site...

Also, what versions of HDF5 do you build? I assume 1.10.7 and 1.12.1?

@kgerheiser
Copy link
Contributor

@edwardhartnett, no I don't think so. We build using Intel just about everywhere normally with NetCDF 4.7.4 and HDF5 1.10.6. And recently we've also tried NetCDF 4.8.1 and HDF5 1.12.1.

@climbfuji
Copy link
Contributor

No problem at all building those versions on any of the platforms, with Intel versions 18 up to oneapi 2022.0.2 (note: we are still using the classic compilers, not icx/ifx).

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Apr 25, 2022 via email

@arunchawla-NOAA
Copy link

GDIT has a huge backlog. I will make a help desk request but do not expect that to happen any time soon till the transition to operations happens formally

@junwang-noaa
Copy link
Author

junwang-noaa commented Apr 25, 2022 •
edited
Loading

@Hang-Lei-NOAA The mpiall2all slowness in Intel 2018/2019 has nothing to do with Netcdf/HDF5, It was tested that the mpiall2all is fast in impi 2021/2022. We need impi 2021/2022 to continue the ESMF managed threading capability in UFS.

@junwang-noaa
Copy link
Author

@kgerheiser May I ask if the hpc-stack with Intel 2022 is installed on jet and gaea? Thanks

@kgerheiser
Copy link
Contributor

Hey @junwang-noaa I am working on that today.

@kgerheiser
Copy link
Contributor

Just about done. A few more libraries to install because of an issue with the firewall on Jet.

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented May 3, 2022 via email

@kgerheiser
Copy link
Contributor

Jet and Orion have 2022.1.2, and Gaea with Intel 2021.3.0 has been completed.

@junwang-noaa
Copy link
Author

@kgerheiser Thank you very much! I will run RT test on these platforms.

@jkbk2004
Copy link

jkbk2004 commented May 6, 2022

@kgerheiser @junwang-noaa I will confirm on Cheyenne as well: module use /glade/work/epicufsrt/GMTB/tools/intel/2022.1/hpc-stack-v1.2.0_6eb6/modulefiles/stack

@EdwardSnyder-NOAA EdwardSnyder-NOAA mentioned this issue May 6, 2022
Closed
@junwang-noaa
Copy link
Author

junwang-noaa commented May 20, 2022 •
edited
Loading

@kgerheiser with gnu compiler on hera, the g2/3.4.5 and pio/2.5.3 are not available,vwould you pelase install them? I got following error:

but cannot be loaded as requested: "g2/3.4.5", "pio/2.5.3"

@jkbk2004 @MinsukJi-NOAA FYI.

@kgerheiser
Copy link
Contributor

@Hang-Lei-NOAA Hera

@Hang-Lei-NOAA
Copy link
Contributor

g2/3.4.5 and pio/2.5.3 have been added. Thanks!

@junwang-noaa
Copy link
Author

@KyleNevins-NOAA @Hang-Lei-NOAA Would you please install hpc-stack with impi 2021.1.3 on hera? We had some issue with a
ufs-weather-model test using impi 2022.1.2, please see ufs-community/ufs-weather-model#1228. Thank you very much for helping!

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented May 25, 2022 via email

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA Thanks. Here is the list:
esmf/8.3.0b09
esmf/8.3.0b09-debug
mapl/2.11.0-esmf-8.3.0b09
hdf5/1.10.6
netcdf/4.7.4
pio/2.5.3

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented May 25, 2022 via email

@junwang-noaa
Copy link
Author

@Hang-Lei-NOAA Thank you very much! The issue with impi 2022 was identified, we are trying to see if a workaround will resolve the issue. I will let you know if we need the intel/2021.3 soon. Thanks!

@Hang-Lei-NOAA
Copy link
Contributor

Hang-Lei-NOAA commented Oct 11, 2022 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kgerheiser kgerheiser

@climbfuji climbfuji

@Hang-Lei-NOAA Hang-Lei-NOAA

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@kgerheiser @climbfuji @junwang-noaa @jkbk2004 @edwardhartnett @Hang-Lei-NOAA @arunchawla-NOAA @KyleNevins-NOAA