[JME NanoAOD] Changes for event size reduction #32722

nurfikri89 · 2021-01-23T12:26:38Z

PR description:

This PR is to reduce the size of custom JME NanoAODs. The following changes are made:

Check that npuProd[ibx].getPU_pT_hats() size is more than 0 in PhysicsTools/NanoAOD/plugins/NPUTablesProducer.cc.
Switch on storing Parton Shower (PS) weights.
b-tagging algorithm outputs are saved for AK4 jets with pT > 15 GeV.
Low-level input variables for PileUp jet ID and Quark-Gluon Likelihood are saved for AK4 jets with pT > 10 GeV.
Remove AK4 PF and AK8 PF jet collections.
Add separate AK8 GEN and AK8 Puppi jet collections for JEC studies. Minimal information are saved.
For reclustered GEN jet collections, the minimum pT at the jet clustering level uses the default value (3 GeV) in RecoJets/JetProducers/python/ak4GenJets_cfi.py.
For reclustered reco-level jet collections, the minimum pT at the jet clustering level is set to be 0. Then, only jets with (JEC-corrected) pT > 8 GeV are saved. Additionally for MC samples, jets with pT < 8 GeV are saved if they are matched to GEN jets.

With this PR, the event size is now reduced to 6.42 kb/event from 9.32 kb/event. The comparison was made using 10K events from a TTJets RunIISummer19UL17MiniAOD sample. The changes in this PR was discussed in the 13/01/2021 XPOG meeting [1].

[1] https://indico.cern.ch/event/978436/

if this PR is a backport please specify the original PR and why you need to backport that PR:

Once merged, this PR needs to be backported to 10_6_X, for the ultra legacy campaigns.

…or cleanup.

…ets. Build and add separate AK8 Gen and AK8 Puppi jet collections for JEC studies. Some cleanup.

nurfikri89 · 2021-01-23T12:27:01Z

FYI JMAR convenors @camclean @alefisico

cmsbuild · 2021-01-23T12:34:48Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20859

This PR adds an extra 32KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20859/code-checks.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20859/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20859/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20859/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2021-01-23T12:49:28Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32722/20860

This PR adds an extra 32KB to repository

cmsbuild · 2021-01-23T12:49:55Z

A new Pull Request was created by @nurfikri89 (Nurfikri Norjoharuddeen) for master.

It involves the following packages:

PhysicsTools/NanoAOD
PhysicsTools/PatAlgos

@perrotta, @gouskos, @cmsbuild, @fgolf, @slava77, @jpata, @mariadalfonso can you please review it and eventually sign? Thanks.
@jdamgov, @emilbols, @gouskos, @swertz, @jdolen, @ahinzmann, @smoortga, @schoef, @rappoccio, @mariadalfonso, @JyothsnaKomaragiri, @nhanvtran, @gkasieczka, @clelange, @hatakeyamak, @ferencek, @gpetruc, @andrzejnovak, @peruzzim, @seemasharmafnal, @mmarionncern this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

gouskos · 2021-01-23T12:57:59Z

please test

cmsbuild · 2021-01-23T15:30:36Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-43558c/12493/summary.html
COMMIT: 6e83ab4
CMSSW: CMSSW_11_3_X_2021-01-23-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/32722/12493/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716961
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2716938
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

mariadalfonso · 2021-01-25T09:33:56Z

please test workflow 25202.15,10224.15,11024.15

(to test JMEnano workflow)

mariadalfonso · 2021-01-25T09:37:11Z

@andrzejnovak
please confirm that this update is good too BTV as well

cmsbuild · 2021-01-25T12:09:40Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-43558c/12507/summary.html
COMMIT: 6e83ab4
CMSSW: CMSSW_11_3_X_2021-01-24-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/32722/12507/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-43558c/10224.15_TTbar_13+2017PU_JMENano+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-43558c/11024.15_TTbar_13+2018PU_JMENano+TTbar_13TeV_TuneCUETP8M1_GenSim+DigiPU+RecoFakeHLTPU+HARVESTFakeHLTPU+Nano
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-43558c/25202.15_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15MC_PU25_JME

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2716961
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2716932
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 156 log files, 37 edm output root files, 37 DQM output files

gouskos · 2021-01-25T14:53:31Z

PhysicsTools/NanoAOD/python/custom_jme_cff.py

-    "enabled" : False, 
-  },  
+  {
+    "jet"     : "ak6gen",


What's the motivation for the change from ak8 -> ak6 in the naming ?

I had set the new AK8 GEN jet collection reclustering in a function here. The change in the line above is just to put a dummy example using ak6gen. The ak6gen jet clustering is switched off.

andrzejnovak · 2021-01-25T15:31:33Z

@mariadalfonso afaik btv is not using this JME Nano

gouskos · 2021-01-28T08:22:58Z

+xpog

All tests passed, and changes to the code are consistent with the PR description
Please prepare the corresponding backports for CMSSW_11_2_X and CMSSW_10_6_X

nurfikri89 · 2021-01-28T09:20:36Z

@gouskos The backport PRs are #32759 (10_6_X) and #32760 (11_2_X)

slava77 · 2021-01-28T18:44:39Z

+1

for #32722 6e83ab4

code changes are in line with the PR description; reco review covers just PhysicsTools/PatAlgos/python/tools/jetCollectionTools.py , which has some extension in functionality without changing the defaults
jenkins tests pass and comparisons with the baseline show no relevant differences

cmsbuild · 2021-01-28T18:45:06Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2021-01-29T01:04:41Z

+1

…reduction [JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 10_6_X

…reduction [JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 11_2_X

nurfikri89 added 6 commits January 19, 2021 21:01

Add check for pT_hats collection to be more than 0

f415df4

Add option to specify minimum jet pt at Fastjet clustering level. Min…

fe82f32

…or cleanup.

Save Parton Shower weights. Remove PF jets. Revised pt cuts for all j…

b7af07b

…ets. Build and add separate AK8 Gen and AK8 Puppi jet collections for JEC studies. Some cleanup.

Remove pt cut

4d0a383

Put back QGL variables. Accidentally removed previously

cc53ba1

Minor documentation update

3d706d6

cmsbuild added this to the CMSSW_11_3_X milestone Jan 23, 2021

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending xpog-pending labels Jan 23, 2021

cmsbuild added code-checks-rejected and removed code-checks-pending labels Jan 23, 2021

Apply code-checks and code-format patches

6e83ab4

cmsbuild added code-checks-pending and removed code-checks-rejected labels Jan 23, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 23, 2021

cmsbuild added tests-started and removed tests-pending labels Jan 23, 2021

nurfikri89 mentioned this pull request Jan 23, 2021

JMEnano oversized cms-nanoAOD/cmssw#549

Closed

cmsbuild removed the tests-started label Jan 23, 2021

cmsbuild added the tests-approved label Jan 23, 2021

cmsbuild added tests-started and removed tests-approved labels Jan 25, 2021

mariadalfonso mentioned this pull request Jan 25, 2021

features for V9 nanoAOD cms-nanoAOD/cmssw#555

Closed

55 tasks

cmsbuild added tests-approved and removed tests-started labels Jan 25, 2021

gouskos reviewed Jan 25, 2021

View reviewed changes

cmsbuild added xpog-approved and removed xpog-pending labels Jan 28, 2021

This was referenced Jan 28, 2021

[JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 10_6_X #32759

Merged

[JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 11_2_X #32760

Merged

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Jan 28, 2021

cmsbuild added orp-approved and removed orp-pending labels Jan 29, 2021

cmsbuild merged commit 3aa05aa into cms-sw:master Jan 29, 2021

cmsbuild added a commit that referenced this pull request Feb 1, 2021

Merge pull request #32759 from nurfikri89/port113XTo106X_nanojme_size…

c39a1d7

…reduction [JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 10_6_X

cmsbuild added a commit that referenced this pull request Feb 3, 2021

Merge pull request #32760 from nurfikri89/port113XTo112X_nanojme_size…

d104174

…reduction [JME NanoAOD] Backport of #32722 (Changes for event size reduction) to 11_2_X

nurfikri89 deleted the from113X_202101191100_nanojme_sizereduction branch May 25, 2021 08:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[JME NanoAOD] Changes for event size reduction #32722

[JME NanoAOD] Changes for event size reduction #32722

nurfikri89 commented Jan 23, 2021

nurfikri89 commented Jan 23, 2021 •

edited

Loading

cmsbuild commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

gouskos commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

mariadalfonso commented Jan 25, 2021

mariadalfonso commented Jan 25, 2021

cmsbuild commented Jan 25, 2021

gouskos Jan 25, 2021

nurfikri89 Jan 25, 2021

andrzejnovak commented Jan 25, 2021

gouskos commented Jan 28, 2021

nurfikri89 commented Jan 28, 2021

slava77 commented Jan 28, 2021

cmsbuild commented Jan 28, 2021

qliphy commented Jan 29, 2021

[JME NanoAOD] Changes for event size reduction #32722

[JME NanoAOD] Changes for event size reduction #32722

Conversation

nurfikri89 commented Jan 23, 2021

PR description:

if this PR is a backport please specify the original PR and why you need to backport that PR:

nurfikri89 commented Jan 23, 2021 • edited Loading

cmsbuild commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

gouskos commented Jan 23, 2021

cmsbuild commented Jan 23, 2021

Comparison Summary

mariadalfonso commented Jan 25, 2021

mariadalfonso commented Jan 25, 2021

cmsbuild commented Jan 25, 2021

Comparison Summary

gouskos Jan 25, 2021

Choose a reason for hiding this comment

nurfikri89 Jan 25, 2021

Choose a reason for hiding this comment

andrzejnovak commented Jan 25, 2021

gouskos commented Jan 28, 2021

nurfikri89 commented Jan 28, 2021

slava77 commented Jan 28, 2021

cmsbuild commented Jan 28, 2021

qliphy commented Jan 29, 2021

nurfikri89 commented Jan 23, 2021 •

edited

Loading