Possible misconfiguration of TrackingPOGFilters #37738

perrotta · 2022-04-29T10:23:08Z

The PR tests of (now merged) #37134 for wfs 10804.31 and 10805.31 are showing at every event the following error messagges:

%MSG-w NoModule:   ByClusterSummaryMultiplicityPairEventFilter:toomanystripclus53X  28-Apr-2022 20:02:03 CEST Run: 1 Event: 1
No information for requested module 5. Please check in the Provinence Infomation for proper modules.
%MSG
%MSG-w NoModule:  ByClusterSummaryMultiplicityPairEventFilter:manystripclus53X  28-Apr-2022 20:02:03 CEST Run: 1 Event: 1
No information for requested module 5. Please check in the Provinence Infomation for proper modules.
%MSG

I did not verify whether this also happens in other workflows.

As pointed out by @mmusich in #37134 (comment), it looks like there's something wrong with how the "tracking POG" filters in:

cmssw/RecoMET/METFilters/python/trackingPOGFilters_cfi.py

Lines 44 to 58 in 6d2f660

    
           manystripclus53X = cms.EDFilter('ByClusterSummaryMultiplicityPairEventFilter', 
        
                                           multiplicityConfig = cms.PSet( 
        
                                                                firstMultiplicityConfig = cms.PSet( 
        
                                                                clusterSummaryCollection = cms.InputTag("clusterSummaryProducer"), 
        
                                                                subDetEnum = cms.int32(5), 
        
                                                                varEnum = cms.int32(0) 
        
                                                                ), 
        
                                                                secondMultiplicityConfig = cms.PSet( 
        
                                                                clusterSummaryCollection = cms.InputTag("clusterSummaryProducer"), 
        
                                                                subDetEnum = cms.int32(0), 
        
                                                                varEnum = cms.int32(0) 
        
                                                                ), 
        
                                                                ), 
        
                                                                cut = cms.string("( mult2 > 20000+7*mult1)") 
        
                                                                )

are configured.

The text was updated successfully, but these errors were encountered:

cmsbuild · 2022-04-29T10:23:28Z

A new Issue was created by @perrotta Andrea Perrotta.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

perrotta · 2022-04-29T10:23:47Z

assign reconstruction

FYI @cms-sw/tracking-pog-l2

cmsbuild · 2022-04-29T10:24:10Z

New categories assigned: reconstruction

@jpata,@slava77,@clacaputo you have been requested to review this Pull request/Issue and eventually sign? Thanks

jpata · 2022-04-29T13:35:30Z

type trk

slava77 · 2022-04-29T14:02:58Z

this is probably a tracker issue (clusterSummaryProducer is under the tracker )

there was a PR that drops empty detsets in the cluster collections
#37035
@ferencek
I guess this then leads to the clustersummary not seeing BPIX or other dets in the photon gun events, which then in turn leads to the warning.

@cms-sw/trk-dpg-l2

slava77 · 2022-04-29T14:05:32Z

a fix could be to still fill a summary for empty dets, but perhaps a different functionality will need to be added to check empties

ferencek · 2022-04-29T20:48:07Z

this is probably a tracker issue (clusterSummaryProducer is under the tracker )

there was a PR that drops empty detsets in the cluster collections #37035 @ferencek I guess this then leads to the clustersummary not seeing BPIX or other dets in the photon gun events, which then in turn leads to the warning.

@cms-sw/trk-dpg-l2

Hi @slava77, #37035 drops empty DetSets from the pixel digi collection so the multiplicity of pixel clusters remains unchanged (after all, there is nothing to cluster from empty DetSets, more details in #37035 (comment)). Hence, I don't see how the issue reported here could be related to #37035.

slava77 · 2022-04-29T21:32:59Z

Hi @slava77, #37035 drops empty DetSets from the pixel digi collection so the multiplicity of pixel clusters remains unchanged (after all, there is nothing to cluster from empty DetSets, more details in #37035 (comment)). Hence, I don't see how the issue reported here could be related to #37035.

but doesn't the clusterizer preserve the DetSets dimension? If so, for an empty BPIX before #37035 the clusterSummaryProducer would see BPIX detSets and would collect some (zeroes ) info for them, while after that PR it will not see them.

ferencek · 2022-04-29T23:35:14Z

Hi @slava77, #37035 drops empty DetSets from the pixel digi collection so the multiplicity of pixel clusters remains unchanged (after all, there is nothing to cluster from empty DetSets, more details in #37035 (comment)). Hence, I don't see how the issue reported here could be related to #37035.

but doesn't the clusterizer preserve the DetSets dimension? If so, for an empty BPIX before #37035 the clusterSummaryProducer would see BPIX detSets and would collect some (zeroes ) info for them, while after that PR it will not see them.

Based on my observations described in #37035 (comment), the clusterizer does not preserve the DetSetVector size. When it encounter an empty digi DetSet, it does not produce an empty cluster DetSet. This also seems to be confirmed by https://github.com/cms-sw/cmssw/blob/CMSSW_12_2_0/RecoLocalTracker/SiPixelClusterizer/plugins/SiPixelClusterProducer.cc#L191--L197.

qliphy · 2022-05-02T03:15:11Z

Also to add here: since CMSSW_12_4 2022-04-29-2300 , there is an IB issue with 10805.31
@ssrothman Please have a check.

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc10/CMSSW_12_4_X_2022-05-01-2300/pyRelValMatrixLogs/run/10804.31_SingleGammaPt10+2018_photonDRN+SingleGammaPt10_pythia8_GenSimINPUT+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano/step3_SingleGammaPt10+2018_photonDRN+SingleGammaPt10_pythia8_GenSimINPUT+Digi+RecoFakeHLT+HARVESTFakeHLT+ALCA+Nano.log#/

Starting python2 /data/cmsbld/jenkins/workspace/ib-run-relvals/cms-bot/monitor_workflow.py timeout --signal SIGTERM 9000 cmsRun -j JobReport3.xml step3_RAW2DIGI_L1Reco_RECO_RECOSIM_PAT_VALIDATION_DQM.py
sh: nvidia-smi: command not found
%MSG-i ThreadStreamSetup: (NoModuleName) 02-May-2022 01:20:31 CEST pre-events
setting # threads 4
setting # streams 4
%MSG
02-May-2022 01:20:48 CEST Initiating request to open file file:step2.root
02-May-2022 01:20:50 CEST Successfully opened file file:step2.root
2022-05-02 01:21:27.078436: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
----- Begin Fatal Exception 02-May-2022 01:25:37 CEST-----------------------
An exception of category 'FallbackFailed' occurred while
[0] Calling beginJob
Exception Message:
Starting the fallback server failed with exit code 256

makortel · 2022-05-02T14:11:39Z

Also to add here: since CMSSW_12_4 2022-04-29-2300 , there is an IB issue with 10805.31
@ssrothman Please have a check.

Let's move this to a separate issue as the Triton issue has nothing to do with TrackingPOGFilters. I opened #37767.

mmusich · 2022-05-06T09:45:04Z

so, I 've started to have a look and here is some other evidence I gathered:

as anticipated by @ferencek if I run wf 10804.31 while reverting the two commits from Avoid storing empty DetSet containers in the pixel digi collection #37035 (d951d12 and 8ce818f) I still observe the warnings, so that cannot be the culprit of the issue.
snooping around in the repository for similar issues, I see that it was already flagged once in this PR Update the ADC->fC table, simplify the charge reconstruction algorith… #22974 (comment)
the warning though are indeed triggered by missing pixel information, 5 is the position of PIXEL in the CMSTracker enum:

cmssw/DataFormats/TrackerCommon/interface/ClusterSummary.h

Lines 76 to 87 in 62ebaf9

    
           enum CMSTracker { 
        
             STRIP = 0, 
        
             TIB = 1, 
        
             TOB = 2, 
        
             TID = 3, 
        
             TEC = 4, 
        
             PIXEL = 5, 
        
             FPIX = 6, 
        
             BPIX = 7, 
        
             NVALIDENUMS = 8, 
        
             NTRACKERENUMS = 100 
        
           };

more specifically it gets triggered here when checking the cluster count:

cmssw/DataFormats/TrackerCommon/interface/ClusterSummary.h

Lines 109 to 112 in 62ebaf9

    
           int getNClus(const CMSTracker mod) const { 
        
             int pos = getModuleLocation(mod); 
        
             return pos < 0 ? 0. : nClus[pos]; 
        
           }

mmusich · 2022-05-10T08:06:22Z

Upon further look, as far as I can tell, the printout works as intended.

in this particular workflow (10804.31) for some events (but not for every event) there aren't any pixel clusters generated.
ClusterSummaryProducer copies a trimmed a version of the modules array (only for the subdetectors with non-zero entries)

cmssw/RecoLocalTracker/SubCollectionProducers/src/ClusterSummaryProducer.cc

Line 123 in 586c400

result->copyNonEmpty(cCluster);

cmssw/DataFormats/TrackerCommon/src/ClusterSummary.cc

Lines 73 to 79 in 6d2f660

    
           for (unsigned int iM = 0; iM < src_nClus.size(); ++iM) { 
        
             if (src.nClus[iM] != 0) { 
        
               modules.push_back(src_modules[iM]); 
        
               nClus.push_back(src_nClus[iM]); 
        
               clusSize.push_back(src_clusSize[iM]); 
        
               clusCharge.push_back(src_clusCharge[iM]); 
        
             }

the filters manystripclus53X and toomanystripclus53X require explicitly the presence of pixel clusters in order to come to a decision:

cmssw/RecoMET/METFilters/python/trackingPOGFilters_cfi.py

Line 48 in 6d2f660

subDetEnum = cms.int32(5),

cmssw/RecoMET/METFilters/python/trackingPOGFilters_cfi.py

Line 65 in 6d2f660

subDetEnum = cms.int32(5),

hence the warning gets triggered here:

cmssw/DataFormats/TrackerCommon/src/ClusterSummary.cc

Lines 50 to 53 in 6d2f660

    
           if (warn) 
        
             edm::LogWarning("NoModule") << "No information for requested module " << mod 
        
                                         << ". Please check in the Provinence Infomation for proper modules.";

mmusich · 2022-05-16T12:59:01Z

@cms-sw/reconstruction-l2 given the analysis above #37738 (comment) do you think is this issue still relevant?

jpata · 2022-05-16T13:59:06Z

+reconstruction

it looks like the printout warnings are as intended

cmsbuild · 2022-05-16T13:59:27Z

This issue is fully signed and ready to be closed.

perrotta · 2022-05-16T14:08:21Z

@mmusich @jpata: ok, we could even close it.
But still it looks to me that those messages could be demoted, if really to be considered "as intended" for the normal workflows. If it is intended, why to issue a warning?
Alternatively, some limitation to the number of such messages that can be issued during a run could also be useful to avoid polluting the outputs and let them clean for other, possibli more relevant. warnings

mmusich · 2022-05-16T14:13:36Z

@perrotta the problem is about the sanity of running the MET filter in a single photon, no PU wf. In normal collision workflows it makes sense to check because there are always pixel clusters.
About how the wf is set up I think it's not trk responsibility (another issue could be opened)

jpata · 2022-05-16T14:17:28Z

I agree with Marco above. Do we have ways to ignore warnings in some wfs (per-wf logging level)? Or maybe extending the warning message "this should not happen in normal collision events" or such?

makortel · 2022-05-16T15:14:43Z

It should be straightforward to suppress messages from particular category (NoModule in this case) already with something along

cmsDriver.py ... --customise_commands="process.MessageLogger.NoModule = dict(limit=0)"

(see https://github.com/cms-sw/cmssw/blob/master/FWCore/MessageService/Readme.md#suppressing-a-particular-category), but if the demand for such suppression starts to spread, perhaps some more specific structure in runTheMatrix.py infrastructure could be useful (@cms-sw/pdmv-l2).

tvami · 2022-12-20T16:08:31Z

assign pdmv

see Possible misconfiguration of TrackingPOGFilters #37738 (comment)

cmsbuild · 2022-12-20T16:08:48Z

New categories assigned: pdmv

@bbilin,@sunilUIET,@kskovpen you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild added the pending-assignment label Apr 29, 2022

cmsbuild added pending-signatures reconstruction-pending and removed pending-assignment labels Apr 29, 2022

perrotta mentioned this issue Apr 29, 2022

Dynamic Reduction Network for photon energy regression in miniAOD using the SonicTriton service #37134

Merged

cmsbuild added the tracking label Apr 29, 2022

cmsbuild added trk and removed tracking labels Apr 29, 2022

makortel mentioned this issue May 2, 2022

Triton fallback server failure in workflow 10805.31 step 3 #37767

Open

This was referenced May 6, 2022

unrunnable configuration file obtained with edmConfigDump #37837

Closed

Minor fixes in ClusterSummary(Producer) #37881

Merged

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels May 16, 2022

cmsbuild added pdmv-pending pending-signatures and removed fully-signed labels Dec 20, 2022

mmusich mentioned this issue May 16, 2023

Add 2023 data relval workflows #41629

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Possible misconfiguration of TrackingPOGFilters #37738

Possible misconfiguration of TrackingPOGFilters #37738

perrotta commented Apr 29, 2022

cmsbuild commented Apr 29, 2022

perrotta commented Apr 29, 2022

cmsbuild commented Apr 29, 2022

jpata commented Apr 29, 2022 •

edited

Loading

slava77 commented Apr 29, 2022

slava77 commented Apr 29, 2022

ferencek commented Apr 29, 2022

slava77 commented Apr 29, 2022 •

edited

Loading

ferencek commented Apr 29, 2022

qliphy commented May 2, 2022 •

edited

Loading

makortel commented May 2, 2022

mmusich commented May 6, 2022

mmusich commented May 10, 2022

mmusich commented May 16, 2022

jpata commented May 16, 2022

cmsbuild commented May 16, 2022

perrotta commented May 16, 2022

mmusich commented May 16, 2022

jpata commented May 16, 2022

makortel commented May 16, 2022

tvami commented Dec 20, 2022

cmsbuild commented Dec 20, 2022

Possible misconfiguration of TrackingPOGFilters #37738

Possible misconfiguration of TrackingPOGFilters #37738

Comments

perrotta commented Apr 29, 2022

cmsbuild commented Apr 29, 2022

perrotta commented Apr 29, 2022

cmsbuild commented Apr 29, 2022

jpata commented Apr 29, 2022 • edited Loading

slava77 commented Apr 29, 2022

slava77 commented Apr 29, 2022

ferencek commented Apr 29, 2022

slava77 commented Apr 29, 2022 • edited Loading

ferencek commented Apr 29, 2022

qliphy commented May 2, 2022 • edited Loading

makortel commented May 2, 2022

mmusich commented May 6, 2022

mmusich commented May 10, 2022

mmusich commented May 16, 2022

jpata commented May 16, 2022

cmsbuild commented May 16, 2022

perrotta commented May 16, 2022

mmusich commented May 16, 2022

jpata commented May 16, 2022

makortel commented May 16, 2022

tvami commented Dec 20, 2022

cmsbuild commented Dec 20, 2022

jpata commented Apr 29, 2022 •

edited

Loading

slava77 commented Apr 29, 2022 •

edited

Loading

qliphy commented May 2, 2022 •

edited

Loading