HLT farm crash in Run 379530 #44759

trtomei · 2024-04-17T10:23:14Z

Reporting a crash in Run 379530

Here's the recipe how to reproduce the crashes (tested with CMSSW_14_0_5_patch1 on lxplus8-gpu):

cmsrel CMSSW_14_0_5_patch1
cd CMSSW_14_0_5_patch1/src
cmsenv

then prepare the reproducer as:

#!/bin/bash -ex

# CMSSW_14_0_5_patch1

hltGetConfiguration run:379530 \
  --globaltag 140X_dataRun3_HLT_v3 \
  --data \
  --no-prescale \
  --no-output \
  --max-events -1 \
  --input /store/group/tsg/FOG/debug/2024-04-17_run379530/run379530_ls0556.root  > hlt.py
  
cat <<@EOF >> hlt.py
process.options.wantSummary = True

process.options.numberOfThreads = 1
process.options.numberOfStreams = 0
@EOF

cmsRun hlt.py &> hlt.log

In the log, we will find the following message:

----- Begin Fatal Exception 17-Apr-2024 12:13:39 CEST-----------------------
An exception of category 'StdException' occurred while
   [0] Processing  Event run: 379530 lumi: 556 event: 1073222806 stream: 0
   [1] Running path 'DST_PFScouting_DoubleMuon_v1'
   [2] Calling method for module MuonHLTEcalPFClusterIsolationProducer/'hltMuonEcalMFPFClusterIsoForMuons'
Exception Message:
A std::exception was thrown.
vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 2)
----- End Fatal Exception -------------------------------------------------

@cms-sw/hlt-l2 FYI
@cms-sw/muon-pog-l2 FYI (sorry, tagged the DPG by mistake)

The text was updated successfully, but these errors were encountered:

cmsbuild · 2024-04-17T10:23:32Z

cms-bot internal usage

cmsbuild · 2024-04-17T10:23:32Z

A new Issue was created by @trtomei.

@smuzaffar, @Dr15Jones, @rappoccio, @sextonkennedy, @antoniovilela, @makortel can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

mmusich · 2024-04-17T10:30:07Z

assign hlt

mmusich · 2024-04-17T10:30:20Z

@cms-sw/muon-pog-l2 FYI

cmsbuild · 2024-04-17T10:30:27Z

New categories assigned: hlt

@Martin-Grunewald,@mmusich you have been requested to review this Pull request/Issue and eventually sign? Thanks

mmusich · 2024-04-17T10:33:51Z

the affected module actually resides in e/gamma package

cmssw/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc

Line 193 in 6e82cc8

    
           typedef HLTEcalPFClusterIsolationProducer<reco::RecoChargedCandidate> MuonHLTEcalPFClusterIsolationProducer;

mmusich · 2024-04-17T10:34:13Z

@cms-sw/egamma-pog-l2 FYI

mmusich · 2024-04-17T12:35:27Z

So this happens because we have a edm::Ref<reco::RecoChargedCandidate> with eta exactly equal to 0.
In this loop:

cmssw/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc

Lines 173 to 181 in 6e82cc8

    
           if (doRhoCorrection_) { 
        
             int iEA = -1; 
        
             auto cEta = std::abs(candRef->eta()); 
        
             for (int bIt = absEtaLowEdges_.size() - 1; bIt > -1; bIt--) { 
        
               if (cEta > absEtaLowEdges_.at(bIt)) { 
        
                 iEA = bIt; 
        
                 break; 
        
               } 
        
             }

we start with iEA = -1 and since cEta is exactly equal to 0, this condition if (cEta > absEtaLowEdges_.at(bIt)) is never verified and iEA stays -1 when hitting this:

cmssw/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc

Line 183 in 6e82cc8

sum = sum - rho * effectiveAreas_.at(iEA);

resulting in the bound-check failure for effectiveAreas_.

I tested that this simple change:

diff --git a/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc b/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc
index b4c4c8b9ff7..8c878698508 100644
--- a/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc
+++ b/RecoEgamma/EgammaHLTProducers/plugins/HLTEcalPFClusterIsolationProducer.cc
@@ -174,7 +174,7 @@ void HLTEcalPFClusterIsolationProducer<T1>::produce(edm::Event& iEvent, const ed
       int iEA = -1;
       auto cEta = std::abs(candRef->eta());
       for (int bIt = absEtaLowEdges_.size() - 1; bIt > -1; bIt--) {
-        if (cEta > absEtaLowEdges_.at(bIt)) {
+        if (cEta >= absEtaLowEdges_.at(bIt)) {
           iEA = bIt;
           break;
         }
diff --git a/RecoEgamma/EgammaHLTProducers/plugins/HLTHcalPFClusterIsolationProducer.cc b/RecoEgamma/EgammaHLTProducers/plugins/HLTHcalPFClusterIsolationProducer.cc
index 0b195f62b7a..b23763c10a3 100644
--- a/RecoEgamma/EgammaHLTProducers/plugins/HLTHcalPFClusterIsolationProducer.cc
+++ b/RecoEgamma/EgammaHLTProducers/plugins/HLTHcalPFClusterIsolationProducer.cc
@@ -200,7 +200,7 @@ void HLTHcalPFClusterIsolationProducer<T1>::produce(edm::StreamID sid,
       int iEA = -1;
       auto cEta = std::abs(candRef->eta());
       for (int bIt = absEtaLowEdges_.size() - 1; bIt > -1; bIt--) {
-        if (cEta > absEtaLowEdges_.at(bIt)) {
+        if (cEta >= absEtaLowEdges_.at(bIt)) {
           iEA = bIt;
           break;
         }

prevents the crash.

mmusich · 2024-04-17T12:36:02Z

assign reconstruction

cmsbuild · 2024-04-17T12:36:12Z

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

mmusich · 2024-04-17T12:48:01Z

Incidentally I am wondering if we might shave off some (probably tiny) reconstruction time by using the [] operator instead that the slower variant .at() it the same classes.

mandrenguyen · 2024-04-17T12:55:02Z

@mmusich The proposed solution makes sense. Would you like reco to make a PR or you'll take care of it?

swagata87 · 2024-04-17T12:57:15Z

So this happens because we have a edm::Refreco::RecoChargedCandidate with eta exactly equal to 0.

thanks for finding this, Marco. As egamma modules are written in similar fashion, similar unprotected codes are there as well.. so while fixing it maybe one can fix other such unprotected cases as well, if possible..

like for example

cmssw/RecoEgamma/EgammaHLTProducers/plugins/EgammaHLTHcalVarProducerFromRecHit.cc

Line 232 in 6e82cc8

if (scEta > absEtaLowEdges_.at(bIt)) {

cmssw/RecoEgamma/EgammaHLTProducers/plugins/HLTHcalPFClusterIsolationProducer.cc

Line 203 in 6e82cc8

if (cEta > absEtaLowEdges_.at(bIt)) {

mmusich · 2024-04-17T12:57:31Z

Would you like reco to make a PR or you'll take care of it?

feel free to make the PR (provided @cms-sw/egamma-pog-l2 agrees). I am wondering though - since these classes were not touched in a long time, if the triggering factor (candidate eta=0) is just a matter of chance or there is some other underlying problem upstream.

RSalvatico · 2024-04-17T12:59:15Z

Incidentally I am wondering if we might shave off some (probably tiny) reconstruction time by using the [] operator instead that the slower variant .at() it the same classes.

Since we are checking the size of absEtaLowEdges one line above, switching to [] should be safe in my opinion.

mandrenguyen · 2024-04-17T13:01:19Z

Would you like reco to make a PR or you'll take care of it?

feel free to make the PR (provided @cms-sw/egamma-pog-l2 agrees). I am wondering though - since these classes were not touched in a long time, if the triggering factor (candidate eta=0) is just a matter of chance or there is some other underlying problem upstream.

Ok, since (current and former) egamma experts are reacting in this thread, I'll leave it in their hands, unless someone asks me to do something :-)

swagata87 · 2024-04-17T13:19:48Z

i mean since Marco have it set up already, may be he can do the PR.
Since its created a crash I'm assuming its urgent, so immediate fix can go in;
while

since these classes were not touched in a long time, if the triggering factor (candidate eta=0) is just a matter of chance or there is some other underlying problem upstream.

this needs some more thinking probably.. this is a good point

mmusich · 2024-04-17T14:01:47Z

i mean since Marco have it set up already, may be he can do the PR.

OK, here's a minimal fix: #44764.

mmusich · 2024-04-17T14:02:55Z

unassign reconstruction

I realize now that the code is actually only within HLT purview - sorry for the noise @mandrenguyen

mmusich · 2024-04-18T06:49:08Z

proposed fixes:

fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44764 (master, CMSSW_14_1_X)
[14.0.X] fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44768 (backport, CMSSW_14_0_X)

VinInn · 2024-04-18T12:44:46Z

A binary-search may even be faster and safer

mmusich · 2024-05-07T07:38:43Z

+hlt

issue fixed by [14.0.X] fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44768 included in CMSSW_14_0_5_patch2
deployed online via CMSSW_14_0_6_MULTIARCHS (intermediate patches were not used online) together with HLT menu GRun 2024 V1.1. The first stable-beams run CMSSW_14_0_6_MULTIARCHS, using the "v1.1.X" HLT menu (see CMSHLT-3164), was run-380306 (Run2024D). No further issues were spotted.

cmsbuild · 2024-05-07T07:39:04Z

This issue is fully signed and ready to be closed.

cmsbuild added the pending-assignment label Apr 17, 2024

cmsbuild added hlt-pending pending-signatures and removed pending-assignment labels Apr 17, 2024

cmsbuild added the reconstruction-pending label Apr 17, 2024

mmusich mentioned this issue Apr 17, 2024

fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44764

Merged

cmsbuild removed the reconstruction-pending label Apr 17, 2024

mmusich mentioned this issue Apr 18, 2024

[14.0.X] fixup eta lookup in absEtaLowEdges_ in miscellaneous RecoEgamma/EgammaHLTProducers plugins #44768

Merged

cmsbuild closed this as completed in #44764 Apr 18, 2024

cmsbuild added hlt-approved fully-signed and removed hlt-pending pending-signatures labels May 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HLT farm crash in Run 379530 #44759

HLT farm crash in Run 379530 #44759

trtomei commented Apr 17, 2024 •

edited

Loading

cmsbuild commented Apr 17, 2024 •

edited

Loading

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024 •

edited

Loading

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024 •

edited

Loading

mandrenguyen commented Apr 17, 2024

swagata87 commented Apr 17, 2024

mmusich commented Apr 17, 2024

RSalvatico commented Apr 17, 2024

mandrenguyen commented Apr 17, 2024

swagata87 commented Apr 17, 2024 •

edited

Loading

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 18, 2024

VinInn commented Apr 18, 2024

mmusich commented May 7, 2024

cmsbuild commented May 7, 2024

HLT farm crash in Run 379530 #44759

HLT farm crash in Run 379530 #44759

Comments

trtomei commented Apr 17, 2024 • edited Loading

cmsbuild commented Apr 17, 2024 • edited Loading

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024 • edited Loading

mmusich commented Apr 17, 2024

cmsbuild commented Apr 17, 2024

mmusich commented Apr 17, 2024 • edited Loading

mandrenguyen commented Apr 17, 2024

swagata87 commented Apr 17, 2024

mmusich commented Apr 17, 2024

RSalvatico commented Apr 17, 2024

mandrenguyen commented Apr 17, 2024

swagata87 commented Apr 17, 2024 • edited Loading

mmusich commented Apr 17, 2024

mmusich commented Apr 17, 2024

mmusich commented Apr 18, 2024

VinInn commented Apr 18, 2024

mmusich commented May 7, 2024

cmsbuild commented May 7, 2024

trtomei commented Apr 17, 2024 •

edited

Loading

cmsbuild commented Apr 17, 2024 •

edited

Loading

mmusich commented Apr 17, 2024 •

edited

Loading

mmusich commented Apr 17, 2024 •

edited

Loading

swagata87 commented Apr 17, 2024 •

edited

Loading