Implement a `GenericProduct` and `GenericCloner` test module #46821

fwyzard · 2024-11-28T07:57:57Z

PR description:

Implement additional framework functionality used by the MPI-based modules developed in #32632:

edm::GenericProduct can be used to produce any collection using its ROOT dictionary;
edmtest::GenericCloner can be used to clone the event products declared by its configuration, using their ROOT dictionaries.

Extend few core features:

edm::Wrapper<T> can be populated moving from an object using a type-erased void* and the object's std::type_info; edm::WrapperBase provides the moveFrom() public interface.

PR validation:

New unit test pass.

cmsbuild · 2024-11-28T07:58:26Z

cms-bot internal usage

fwyzard · 2024-11-28T07:58:34Z

please test

cmsbuild · 2024-11-28T08:00:42Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46821/42824

There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/GenericProduct.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Modules/src/GenericConsumer.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/README.md modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Utilities/interface/BranchPattern.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632

cmsbuild · 2024-11-28T08:01:03Z

A new Pull Request was created by @fwyzard for master.

It involves the following packages:

FWCore/Framework (core)
FWCore/Modules (core)
FWCore/TestModules (core)
FWCore/Utilities (core)

@Dr15Jones, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@felicepantaleo, @makortel, @missirol, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2024-11-28T09:43:26Z

-1

Failed Tests: HeaderConsistency
Size: This PR adds an extra 12KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23eeae/43128/summary.html
COMMIT: 339cf8b
CMSSW: CMSSW_15_0_X_2024-11-27-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/46821/43128/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3484682
DQMHistoTests: Total failures: 401
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3484261
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 45 files compared)
Checked 202 log files, 172 edm output root files, 46 DQM output files
TriggerResults: no differences found

fwyzard · 2024-11-28T15:06:46Z

Rebased to include the fix from #46825 so the tests should now pass.

cmsbuild · 2024-11-28T15:07:25Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46821/42832

There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/interface/GenericProduct.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Modules/src/GenericConsumer.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/README.md modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Utilities/interface/BranchPattern.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632

cmsbuild · 2024-11-28T15:07:47Z

Pull request #46821 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again.

fwyzard · 2024-11-28T15:17:37Z

@Dr15Jones @makortel this is very preliminary, but id does seem to work.
Could you let me know what you think of this approach?

Some random questions:

should we consider some integration with edm::GenericHandle ?
does the way I used to build objects make sense ?
is there a way to get rid of the memcpy at GenericProduct.h
, line 41 ?

fwyzard · 2024-11-28T16:16:00Z

please test

cmsbuild · 2025-02-25T15:12:04Z

Pull request #46821 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again.

cmsbuild · 2025-02-25T15:17:24Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46821/43868

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Provenance/interface/ProductNamePattern.h modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/src/ProductNamePattern.cc modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/test/BuildFile.xml modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/test/ProductNamePattern_t.cpp modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/interface/ThingWithDoNotSort.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/interface/ThingWithPostInsert.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/src/classes.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/src/classes_def.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Framework/interface/GenericProduct.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/plugins/PostInsertProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/testPostInsertProducer_cfg.py modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/testStdProducts_cfg.py modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Modules/src/GenericConsumer.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement edm::ProductNamePattern #47444
- File FWCore/Reflection/interface/ObjectWithDict.h modified in PR(s): Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Reflection/src/ObjectWithDict.cc modified in PR(s): Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/README.md modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/plugins/GlobalFloatAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalFloatProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalIntAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalIntProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalStringAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalStringProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalVectorAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalVectorProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632

cmsbuild · 2025-02-25T15:17:55Z

Pull request #46821 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again.

Implement a generic product type that can be constructed based on the object's ObjectWithDict and on the wrapper's TypeWithDict.

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

cmsbuild · 2025-02-25T15:52:18Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-46821/43872

There are other open Pull requests which might conflict with changes you have proposed:
- File DataFormats/Provenance/BuildFile.xml modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/interface/ProductNamePattern.h modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/src/ProductNamePattern.cc modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/test/BuildFile.xml modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/Provenance/test/ProductNamePattern_t.cpp modified in PR(s): Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/interface/ThingWithDoNotSort.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/interface/ThingWithPostInsert.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/src/classes.h modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File DataFormats/TestObjects/src/classes_def.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Framework/interface/GenericProduct.h modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/Integration/plugins/BuildFile.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/plugins/PostInsertProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/BuildFile.xml modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/testPostInsertProducer_cfg.py modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Integration/test/testStdProducts_cfg.py modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Modules/src/GenericConsumer.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement edm::ProductNamePattern #47444
- File FWCore/Reflection/interface/ObjectWithDict.h modified in PR(s): Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/Reflection/src/ObjectWithDict.cc modified in PR(s): Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/README.md modified in PR(s): Extend CMSSW to a distributed application over MPI #32632, Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GenericCloner.cc modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/plugins/GlobalFloatAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalFloatProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalIntAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalIntProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalStringAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalStringProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalVectorAnalyzer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/plugins/GlobalVectorProducer.cc modified in PR(s): Implement additional framework integration tests #47425, Modernise the interface of ObjectWithDict #47442, Implement edm::ProductNamePattern #47444
- File FWCore/TestModules/test/BuildFile.xml modified in PR(s): Extend CMSSW to a distributed application over MPI #32632
- File FWCore/TestModules/test/testGenericCloner_cfg.py modified in PR(s): Extend CMSSW to a distributed application over MPI #32632

cmsbuild · 2025-02-25T15:52:40Z

Pull request #46821 was updated. @Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please check and sign again.

fwyzard · 2025-02-25T21:08:33Z

please test

🤷🏻‍♂️

cmsbuild · 2025-02-26T01:23:33Z

+1

Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23eeae/44665/summary.html
COMMIT: 8ed3144
CMSSW: CMSSW_15_1_X_2025-02-25-1200/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/46821/44665/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially removed 1 lines from the logs
Reco comparison results: 55 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3920300
DQMHistoTests: Total failures: 134
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3920146
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

cmsbuild added this to the CMSSW_15_0_X milestone Nov 28, 2024

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Nov 28, 2024

cmsbuild added tests-started and removed tests-pending labels Nov 28, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 28, 2024

cmsbuild added tests-rejected and removed tests-started labels Nov 28, 2024

fwyzard force-pushed the implement_generic_product_tools branch from 339cf8b to e486093 Compare November 28, 2024 15:05

cmsbuild added tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Nov 28, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 28, 2024

cmsbuild removed the tests-pending label Nov 28, 2024

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 25, 2025

fwyzard force-pushed the implement_generic_product_tools branch from bdd190c to 3b462b2 Compare February 25, 2025 15:15

cmsbuild added code-checks-pending and removed code-checks-approved labels Feb 25, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 25, 2025

fwyzard added 6 commits February 25, 2025 16:48

Implement edm::GenericProduct

f3fe02e

Implement a generic product type that can be constructed based on the object's ObjectWithDict and on the wrapper's TypeWithDict.

Implement an integration test for GenericProduct

a72baa8

Implement edmtest::GenericCloner

5bd0828

This EDProducer will clone all the event products declared by its configuration, using their ROOT dictionaries.

Specialize Handle and OprhanHandle for WrapperBase

a976b28

Implement integration tests for WrapperBase specialisations

eb697b7

Implement edmtest::GenericCloner (alt version)

8ed3144

fwyzard force-pushed the implement_generic_product_tools branch from 3b462b2 to 8ed3144 Compare February 25, 2025 15:49

cmsbuild added code-checks-pending and removed code-checks-approved labels Feb 25, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 25, 2025

cmsbuild removed the changes-dataformats label Feb 25, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 25, 2025

cmsbuild added tests-approved and removed tests-started labels Feb 26, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement a `GenericProduct` and `GenericCloner` test module #46821

Implement a `GenericProduct` and `GenericCloner` test module #46821

fwyzard commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024 •

edited

Loading

fwyzard commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

fwyzard commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

fwyzard commented Nov 28, 2024 •

edited

Loading

fwyzard commented Nov 28, 2024

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

fwyzard commented Feb 25, 2025

cmsbuild commented Feb 26, 2025

Implement a GenericProduct and GenericCloner test module #46821

Are you sure you want to change the base?

Implement a GenericProduct and GenericCloner test module #46821

Conversation

fwyzard commented Nov 28, 2024 • edited Loading

PR description:

PR validation:

cmsbuild commented Nov 28, 2024 • edited Loading

fwyzard commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

Comparison Summary

fwyzard commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

cmsbuild commented Nov 28, 2024

fwyzard commented Nov 28, 2024 • edited Loading

fwyzard commented Nov 28, 2024

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

fwyzard commented Feb 25, 2025

cmsbuild commented Feb 26, 2025

Comparison Summary

Implement a `GenericProduct` and `GenericCloner` test module #46821

Implement a `GenericProduct` and `GenericCloner` test module #46821

fwyzard commented Nov 28, 2024 •

edited

Loading

cmsbuild commented Nov 28, 2024 •

edited

Loading

fwyzard commented Nov 28, 2024 •

edited

Loading