Migrate CMSSW from HepMC2 to HepMC3

[mkirsano]

HepMC is a HEP event record used in CMSSW. HepMC2 is currently used.
HepMC3 is a completely rewritten version. Many methods are different from HepMC2. These two records can live together in one program. I have a partly working prototype (working at ~10% functionality, pythia8 only) in my private area.
The evolving plan follows

1. Cope with library structure. By default the HepMC3 libraries structures are lib64/libHepMC3.so->libHepMC3.so.3. CMSSW does not understand it, I have to copy this structure in CMSSW_12_3_0_pre2/external/slc7_amd64_gcc10/lib by hands. - done
2. Rewrite code created for ExternalDecays. Seems still to be needed for the photos++ package (tauola++?)
3. Create GenEventInfoProduct3 class similarly to GenEventInfoProduct. Some methods need rewriting. - done privately
4. Create HepMC3Product similarly to HepMCProduct - done privately
5. Rewrite the code where the HepMC record is fed in G4. Seems to be here (Siewyan):https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Generators/src/Generator.cc#L112. Problem: SimG4Core compiled locally does not work. In contact with V. Ivanchenko
6. At some point a special CMSSW branch will be needed?

[cmsbuild]

A new Issue was created by @mkirsano Mikhail Kirsanov.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[mkirsano]

assign generators

[cmsbuild]

New categories assigned: generators

@mkirsano,@alberto-sanchez,@SiewYan,@GurpreetSinghChahal,@Saptaparna,@agrohsje you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mkirsano]

assign simulation

[cmsbuild]

New categories assigned: simulation

@mdhildreth,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

[mkirsano]

I solved the problem with library structure, preparing PR to add external

[mkirsano]

PR cmsdist #7655

[mkirsano]

PR to include hepmc3 merged

[mkirsano]

I found that it is impossible to include both pythia8 - HepMC2 and pythia8 - HepMC3 converters in the same code.
To have a possibility to switch I will have to create another plugin: Pythia8HepMC3Hadroniser

[civanch]

@mkirsano , do you make a survey of all places inside CMSSW, which require a change? Likely it will be the same situation - new alternative code will be required.

[mkirsano]

No, this concerns only generator -> HepMC conversion. In other codes, such as analysis, using "if" can be sufficient.

[mkirsano]

This is a preliminary list of packages concerned. In addition, there is a class HepMCCandidate, but it seems to be internal in CMSSW (does not depend on hepmc)
listhepmc.txt

[smuzaffar]

@mkirsano , note that many externals are build using hepmc (https://github.com/cms-sw/cmsdist/search?q=hepmc ) including pythia8 . so if you use pythia8 then you automatically get hepmc.

[mkirsano]

I made a prototype of Pythia8Interface that can produce either HepMC2 or HepMC3 record.
Does it have sense to make a PR?

[civanch]

I think if by default it would implement HepMC2 it would be fine.

[mkirsano]

I will probably need help with adding HepMC3 products to edm. The instruction here https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCreatingNewProducts is rather old, probably not up-to-date

[makortel]

I will probably need help with adding HepMC3 products to edm. The instruction here https://twiki.cern.ch/twiki/bin/view/CMSPublic/SWGuideCreatingNewProducts is rather old, probably not up-to-date

That twiki page is pretty much up to date. But if you find something there which is not (or suspect or are unsure), let us know.

[civanch]

@mkirsano , HepMC3 migration was discussed at the today ORP meeting. @smuzaffar , explain us that several GEN packages should be compiled versus HepMC3. This can be achieved normally if a new git BRANCH will be created. So, the plan may be following:

1. we use a switch HepM2/HepMC3 as a python parameter defined somewhere upstream
2. development is ongoing within master, depending on the switch HepMC2 or HepMC3 is called inside CMSSW
3. validation will mean comparison of the new branch results versus master results

The link to the new product is correct (from my point of view). What is essential that depending on the switch producers will consume HepMC2 or HepMC3

[smuzaffar]

The externals packages which uses HepMC are https://github.com/cms-sw/cmsdist/search?q=hepmc

• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/photospp.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/tauolapp.spec
• https://github.com/cms-sw/cmsdist/blob/15eb9c93fe06e67de060ae6860cdedc3d652f37a/madgraph5amcatnlo.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/thepeg.spec
• https://github.com/cms-sw/cmsdist/blob/15eb9c93fe06e67de060ae6860cdedc3d652f37a/rivet.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/pythia8.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/herwig7.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/mctester.spec
• https://github.com/cms-sw/cmsdist/blob/1f4e6a9b7b19f23345c791dcd361230a52b7c87f/evtgen.spec
• https://github.com/cms-sw/cmsdist/blob/15eb9c93fe06e67de060ae6860cdedc3d652f37a/sherpa.spec

We still need to see if all of these can build/run with new HepMC3 :-)

[SiewYan]

@smuzaffar, @mkirsano , i can help to check on those.

[mkirsano]

All these packages can be compiled against HepMC3. I tried photospp, tauolapp. Another question is if they can be compiled against both. But the critical path is the integration of HepMC3Product in edm. Then we can start to look at all the chain (G4, analysis, validation etc.)

[mkirsano]

I am ready for the next PRs, adding HepMC3 products to GeneratorInterface/Core BaseHadronizer and adding Pythia8HepMC3Hadronizer plugin.
Work on adding HepMC3 products to edm is to be forced.

[mkirsano]

GeneratorInterface/Core : PR #37689

[mkirsano]

GeneratorInterface/Pythia8Interface : PR #37753
Depends on #37689

[SiewYan]

To reiterate the plan:

1. Cope with library structure. By default the HepMC3 libraries structures are lib64/libHepMC3.so->libHepMC3.so.3. CMSSW does not understand it, I have to copy this structure in CMSSW_12_3_0_pre2/external/slc7_amd64_gcc10/lib by hands.
2. Rewrite code created for ExternalDecays. Seems still to be needed for the photos++ package (tauola++?)
3. Create GenEventInfoProduct3 class similarly to GenEventInfoProduct. Some methods need rewriting. PR: Add generator hepmc3 products #37187
4. Create HepMC3Product similarly to HepMCProduct. PR: Add generator hepmc3 products #37187
5. Rewrite the code where the HepMC record is fed in G4. Seems to be here (Siewyan):https://github.com/cms-sw/cmssw/blob/master/SimG4Core/Generators/src/Generator.cc#L112. Problem: SimG4Core compiled locally does not work. In contact with V. Ivanchenko
6. At some point a special CMSSW branch will be needed?

PR #38033 opened to address EDMProduct definition effort from #37187

[mkirsano]

Current status: HepMC3Product now is added to EDM and can be put into the event (apparently the work is done by Markus Seidel for Rivet). Next steps:

1. Putting HepMC3 particles into G4 for tracing
2. HepMC3 in other generator interfaces?
3. Adding code to work with PHOTOS

[cmsbuild]

cms-bot internal usage

[mkirsano]

Information for further development:

1. GeneratorInterface/Pythia8Interface/test/pythia8hmc3ex2_cfg.py - configuration that produces gen event in HepMC3 format
2. The gen interface plugin used there sets the data member of the class GeneratorFilter ivhepmc to 3. According to this, the code in GeneratorInterface/Core/interface/GeneratorFilter.h puts in the event genEventInfo3 and HepMC3Product instead of genEventInfo and HepMCProduct. These are to be consumed by the Sim plugin. Is there a way to transfer ivhepmc to Sim plugin?
3. Is it possible to consume both variants of gen products without creating another plugin along with OscarMTProducer? May be not.

[mkirsano]

PR #46797 allows to transfer HepMC3Product from generators (e.g. pythia8) up to the Sim step and load particles into Geant4 for tracing. Some pieces, like vertex smearing, are still missing.

[mkirsano]

PR #46891 allows to smear the event vertex in HepMC3Product as well as in HepMCProduct. Some cleanup of the code is performed in IOMC/EventVertexGenerators

[mkirsano]

Next steps:

1. Check that evtgen works as pythia8 plugin
2. Check that photos++ works through ExternalDecays, add hepmc3 interface creation to photos++
3. Fix ResidualDecays in Pythia8Interface for the case of hepmc3
4. evtgen through ExternalDecays? Can be needed for hadronizers other than pythia8
5. herwig7 hepmc3 interface (did not look yet)
6. ResidualDecay in herwig7 interface?
7. sherpa?
8. hepmc3 GenParticle objects?
9. GenJets?

[mkirsano]

Item 2 : PR cms-sw/cmsdist/pull/9613 merged
Items 2, 3 : PR #47101 merged