Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider · 2025-01-22T14:31:27Z

Additional changes :

PR Unit test job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-unittests) - add GPU_FLAVOR parameter, select node based on it.
PR Relval job (https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/) - select node based on TEST_FLAVOR
Monitoring job (https://cmssdt.cern.ch/jenkins/job/jenkins-monitor-queue/) needs to be changed - kill pr test jobs that are stuck due to lumi node being offline
If ROCm job is killed due to node being offline, mark commit status!

cmsbuild · 2025-01-22T14:31:53Z

cms-bot internal usage

iarspider · 2025-01-22T14:55:00Z

please test with cms-sw/cmssw#46579

to check that cpu tests are not broken

cmsbuild · 2025-01-22T15:00:14Z

-1

Failed Tests: ClangBuild
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43916/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43916/install.sh to create a dev area with all the needed externals and cmssw changes.

CMS deprecated warnings: 1 CMS deprecated warnings found, see summary page for details.

Clang Build

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

iarspider · 2025-01-22T15:07:25Z

please test with cms-sw/cmssw#47163

cmsbuild · 2025-01-22T17:13:41Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/43917/summary.html
COMMIT: 118dd7e
CMSSW: CMSSW_15_0_X_2025-01-22-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2418/43917/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 1664 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3819085
DQMHistoTests: Total failures: 149
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3818916
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: found differences in 1 / 47 workflows

iarspider · 2025-01-23T08:39:49Z

test parameters:

enable = rocm
workflows_gpu = 141.044406,141.044408,141.044412,141.044414,141.044422,141.044424,160.03502

iarspider · 2025-01-23T08:39:53Z

please test

cmsbuild · 2025-01-23T17:05:19Z

Pull request #2418 was updated.

iarspider · 2025-01-23T17:19:57Z

please test

iarspider · 2025-02-24T12:41:14Z

please test

cmsbuild · 2025-02-24T12:41:27Z

Pull request #2418 was updated.

cmsbuild · 2025-02-24T15:28:23Z

-1

Failed Tests: RelVals-ROCM
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b157ff/44603/summary.html
COMMIT: f0974fa
CMSSW: CMSSW_15_1_X_2025-02-24-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cms-bot/2418/44603/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-ROCM

141.044412141.044412_Run3-2023_JetMET2023D_RecoECALOnlyGPU/step3_Run3-2023_JetMET2023D_RecoECALOnlyGPU.log
141.044408141.044408_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling/step3_Run3-2023_JetMET2023D_RecoPixelOnlyTripletsGPU_Profiling.log
141.044414141.044414_Run3-2023_JetMET2023D_RecoECALOnlyGPU_Profiling/step3_Run3-2023_JetMET2023D_RecoECALOnlyGPU_Profiling.log

Expand to see more relval errors ...

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 69
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920211
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

CUDA Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 1
DQMHistoTests: Total histograms compared: 0
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 0
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0 KiB( 0 files compared)
Checked 0 log files, 0 edm output root files, 1 DQM output files

iarspider · 2025-02-25T14:39:44Z

please test

cmsbuild · 2025-02-25T14:41:56Z

Pull request #2418 was updated.

…T_TESTS

cmsbuild · 2025-02-26T08:48:04Z

Pull request #2418 was updated.

iarspider · 2025-02-26T08:48:15Z

please test

cmsbuild · 2025-02-26T15:35:48Z

Pull request #2418 was updated.

iarspider · 2025-02-26T21:32:14Z

pr_testing/test_multiple_prs.sh

@@ -1326,7 +1342,9 @@ if [ "X$BUILD_OK" = Xtrue -a "$RUN_TESTS" = "true" ]; then
  fi
  if [ $(echo ${ENABLE_BOT_TESTS} | tr ',' ' ' | tr ' ' '\n' | grep '^GPU$' | wc -l) -gt 0 -a X"${DISABLE_GPU_TESTS}" != X"true" ] ; then
    DO_GPU_TESTS=true


TODO: figure out how to set DO_GPU_TESTS now that "GPU" is not in EXTRA_RELVAL_TESTS. Maybe keep it and skip when scheduling relvals? L385, L1482,

cmsbuild added orp-pending pending-signatures externals-pending tests-pending labels Jan 22, 2025

cmsbuild added tests-started requires-external and removed tests-pending labels Jan 22, 2025

cmsbuild added tests-rejected and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started and removed tests-rejected labels Jan 22, 2025

cmsbuild added tests-approved and removed tests-started labels Jan 22, 2025

cmsbuild added tests-started tests-pending and removed requires-external tests-approved tests-started labels Jan 23, 2025

cmsbuild added tests-started and removed tests-pending tests-started labels Jan 23, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 24, 2025

Pass ENABLE_GPU_FLAVORS from ib-schedule-pr-tests to ib-run-pr-tests

f0974fa

iarspider mentioned this pull request Feb 24, 2025

Document 'enable rocm' and 'enable cuda' cms-sw/cms-sw.github.io#127

Open

cmsbuild added tests-rejected and removed tests-started labels Feb 24, 2025

iarspider requested a review from smuzaffar February 24, 2025 21:13

iarspider added 2 commits February 25, 2025 15:25

Avoid running maxmem profiling for ROCm RelVals

5b59afe

Fix

2ddc6ae

cmsbuild added tests-started and removed tests-rejected labels Feb 25, 2025

iarspider mentioned this pull request Feb 25, 2025

Memory profiling causes rocmIsEnabled to segfault cms-sw/cmssw#47450

Open

Always add 'GPU' to ENABLE_BOT_TESTS; remove gpu types from ENABLE_BO…

e7c5ac8

…T_TESTS

iarspider force-pushed the pr-rocm-tests branch from 820f148 to e7c5ac8 Compare February 26, 2025 08:46

cmsbuild added tests-pending and removed tests-started labels Feb 26, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 26, 2025

Changes from review

4c3133e

cmsbuild added tests-pending and removed tests-started labels Feb 26, 2025

iarspider commented Feb 26, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

iarspider commented Jan 23, 2025 •

edited

Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

iarspider commented Feb 24, 2025

cmsbuild commented Feb 24, 2025

cmsbuild commented Feb 24, 2025

iarspider commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 26, 2025

iarspider commented Feb 26, 2025

cmsbuild commented Feb 26, 2025

iarspider Feb 26, 2025

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Are you sure you want to change the base?

Run PR gpu utests/relvals on both CUDA and ROCm GPUs #2418

Conversation

iarspider commented Jan 22, 2025 • edited Loading

cmsbuild commented Jan 22, 2025 • edited Loading

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Clang Build

iarspider commented Jan 22, 2025

cmsbuild commented Jan 22, 2025

Comparison Summary

iarspider commented Jan 23, 2025 • edited Loading

iarspider commented Jan 23, 2025

cmsbuild commented Jan 23, 2025

iarspider commented Jan 23, 2025

iarspider commented Feb 24, 2025

cmsbuild commented Feb 24, 2025

cmsbuild commented Feb 24, 2025

RelVals-ROCM

Comparison Summary

CUDA Comparison Summary

iarspider commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 26, 2025

iarspider commented Feb 26, 2025

cmsbuild commented Feb 26, 2025

iarspider Feb 26, 2025

Choose a reason for hiding this comment

iarspider commented Jan 22, 2025 •

edited

Loading

cmsbuild commented Jan 22, 2025 •

edited

Loading

iarspider commented Jan 23, 2025 •

edited

Loading