Add support for the gpu modifier at HLT

[fwyzard]

PR description:

When the gpu modifier is enabled, make the Pixel, ECAL and HCAL reconstruction running at HLT offloadable to a GPU.

PR validation:

Tested with

cmsDriver.py reHLT \
    --processName reHLT \
    -s HLT:@relval2021 \
    --conditions 120X_mcRun3_2021_realistic_v4 \
    --datatier GEN-SIM-DIGI-RAW \
    -n 10 \
    --eventcontent FEVTDEBUGHLT \
    --geometry DB:Extended \
    --era Run3 \
    --procModifiers gpu \
    --filein /store/relval/CMSSW_12_0_0_pre6/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU_120X_mcRun3_2021_realistic_v4_JIRA_129-v1/00000/79c06ed5-929b-4a57-a4f2-1ae90e6b38c5.root

With this change, the customizeHLTforPatatrack() customisation will be tested as part of the .502, .512 and .522 GPU workflows, and customizeHLTforPatatrackTriplets() with the .506 workflow.

[fwyzard]

enable gpu

[fwyzard]

please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34978/24821

• This PR adds an extra 16KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File HLTrigger/Configuration/python/customizeHLTforCMSSW.py modified in PR(s): Add GEM unpacker in L1Repack FullMC and temporary load of GEM geometry to hltGetConfiguration #34788

[cmsbuild]

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

• HLTrigger/Configuration (hlt)

@Martin-Grunewald can you please review it and eventually sign? Thanks.
@Martin-Grunewald this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5f5f9/17966/summary.html
COMMIT: 1cdd51f
CMSSW: CMSSW_12_1_X_2021-08-22-2300/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34978/17966/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19735
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19729
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: no differences found

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3000352
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3000324
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[fwyzard]

Mhm, no, this didn't work as intended.
Looks like the gpu modifier is only applied to the RECO stesp, not the HLT step, so nothing happened.

[fwyzard]

enable gpu

[fwyzard]

please test

[cmsbuild]

Pull request #34978 was updated. @jordan-martins, @chayanit, @bbilin, @wajidalikhan, @Martin-Grunewald, @AdrianoDee, @srimanob, @kskovpen can you please check and sign again.

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5f5f9/18699/summary.html
COMMIT: dd84d38
CMSSW: CMSSW_12_1_X_2021-09-16-2300/slc7_amd64_gcc900
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/34978/18699/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 4
• DQMHistoTests: Total histograms compared: 19735
• DQMHistoTests: Total failures: 6
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 19729
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
• Checked 12 log files, 9 edm output root files, 4 DQM output files
• TriggerResults: no differences found

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 3000833
• DQMHistoTests: Total failures: 12
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3000799
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[fwyzard]

OK, should be ready for review and integration.

[Martin-Grunewald]

+1

[srimanob]

+Upgrade

[qliphy]

@cms-sw/pdmv-l2 Do you have any comment?

[kskovpen]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

@fwyzard Just to make sure, I see some differences on OfflinePixelPV from GPU comparisons, are they understood? :

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisonsGPU/CMSSW_12_1_X_2021-09-16-2300+f5f5f9/45534/10824.502_TTbar_13+2018_Patatrack_PixelOnlyGPU+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+RecoFakeHLT+HARVESTFakeHLT/OfflinePixelPV_Alignment.html

[fwyzard]

If it is in the GPU (with PR) vs GPU (without PR), yes, it is understood that the parallel processing done by the GPUs (or by the CPUs, if we were to run multithreded in the same way) can introduce some small changes, from run to run.

So, the differences are very likely not due to the PR, but to the software itself.

[qliphy]

+1