Use HardwareResourcesDescription in ProcessConfiguration #47355

makortel · 2025-02-14T20:50:13Z

PR description:

This PR is part of #30044. It continues the work of #47175 and #47280 by replacing the never-used "PassID" in ProcessConfiguration with the HardwareResourcesDescription that is filled by ResourceInformation service.

The storage ProcessConfiguration doesn't technically change, as the "Pass ID" was stored as a string, and the HardwareResourcesDescription is serialized manually as a string. Empty string de-serializing as an empty HardwareResourcesDescription, and an empty HardwareResourcesDescription serializing as an empty string should guarantee backwards and forwards compatibility (even if the latter is not formally supported, but just in case future Run3 MC production would need it), also for purposes of ProcessHistory and its ID.

The next PR in this series will improve the printouts of edmProvDump.

Resolves cms-sw/framework-team#1185

PR validation:

Framework unit tests pass

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

To be backported to 15_0_X

cmsbuild · 2025-02-14T20:50:43Z

cms-bot internal usage

makortel · 2025-02-14T20:51:09Z

IOPool/Common/bin/EdmProvDump.cc

@@ -116,8 +126,9 @@ std::ostream& operator<<(std::ostream& os, edm::ProcessHistory& iHist) {
  std::string const indentDelta("  ");
  std::string indent = indentDelta;
  for (auto const& process : iHist) {
-    os << indent << process.processName() << " '" << process.passID() << "' '" << process.releaseVersion() << "' ("
-       << process.parameterSetID() << ")" << std::endl;
+    // TODO: add printout of HardwareResourcesDescription


Next PR will address this TODO

cmsbuild · 2025-02-14T20:52:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47355/43702

cmsbuild · 2025-02-14T20:53:14Z

A new Pull Request was created by @makortel for master.

It involves the following packages:

DataFormats/Provenance (core)
FWCore/AbstractServices (core)
FWCore/Framework (core)
FWCore/Integration (core)
FWCore/Sources (core)
FWCore/TestProcessor (core)
IOPool/Common (core)
IOPool/Input (core)
IOPool/SecondaryInput (core)
IOPool/Streamer (core)

@Dr15Jones, @cmsbuild, @makortel, @smuzaffar can you please review it and eventually sign? Thanks.
@missirol, @mmusich, @rovere, @wddgit this is something you requested to watch as well.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

makortel · 2025-02-14T21:08:46Z

enable gpu

makortel · 2025-02-14T21:08:51Z

@cmsbuild, please test

cmsbuild · 2025-02-14T22:55:36Z

-1

Failed Tests: Build
Size: This PR adds an extra 256KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3a0d8d/44398/summary.html
COMMIT: ec658cd
CMSSW: CMSSW_15_1_X_2025-02-14-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/47355/44398/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation error when building:

>> Compiling edm plugin src/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=120301 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -DCMSSW_GIT_HASH='CMSSW_15_1_X_2025-02-14-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_15_1_X_2025-02-14-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-02-14-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/classlib/3.1.3-2827e9dbeb7dd2aced5a46b8a966dfab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/pcre/8.43-2d141998cfe5424b8f7aff48035cc2da/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/boost/1.80.0-96a02191111b66819e07de179cb25a0e/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/bz2lib/1.0.6-d065ccd79984efc6d4660f410e4c81de/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/libuuid/2.34-27ce4c3579b5b1de2808ea9c4cd8ed29/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/protobuf/3.21.9-3b4dc3892e5c8da1829fdf0bfe570820/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/lcg/root/6.32.09-0a945d6e24dcaabe218b38fa8292785d/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/tbb/v2021.9.0-9b0f135342cfe979b6500c59f501774e/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/zlib/1.2.13-d217cdbdd8d586e845e05946de2796be/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/fmt/10.2.1-e35fd1db5eb3abc8ac0452e8ee427196/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/md5/1.0.0-5b594b264e04ae51e893b1d69a797ec6/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/tinyxml2/6.2.0-f99ae2781d074227d47e8a3e7c8ec87e/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc12/src/DQMServices/FwkIO/plugins/DQMServicesFwkIOPlugins/DQMRootOutputModule.cc.d src/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc -o tmp/el8_amd64_gcc12/src/DQMServices/FwkIO/plugins/DQMServicesFwkIOPlugins/DQMRootOutputModule.cc.o
>> Compiling edm plugin src/DQMServices/FwkIO/plugins/DQMRootSource.cc
/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/bin/c++ -c -DCMS_MICRO_ARCH='x86-64-v3' -DGNU_GCC -D_GNU_SOURCE -DTBB_USE_GLIBCXX_VERSION=120301 -DTBB_SUPPRESS_DEPRECATED_MESSAGES -DTBB_PREVIEW_RESUMABLE_TASKS=1 -DTBB_PREVIEW_TASK_GROUP_EXTENSIONS=1 -DBOOST_SPIRIT_THREADSAFE -DPHOENIX_THREADSAFE -DBOOST_MATH_DISABLE_STD_FPCLASSIFY -DBOOST_UUID_RANDOM_PROVIDER_FORCE_POSIX -D__STDC_LIMIT_MACROS -D__STDC_FORMAT_MACROS -DCMSSW_GIT_HASH='CMSSW_15_1_X_2025-02-14-1100' -DPROJECT_NAME='CMSSW' -DPROJECT_VERSION='CMSSW_15_1_X_2025-02-14-1100' -Isrc -Ipoison -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_15_1_X_2025-02-14-1100/src -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/classlib/3.1.3-2827e9dbeb7dd2aced5a46b8a966dfab/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/pcre/8.43-2d141998cfe5424b8f7aff48035cc2da/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/boost/1.80.0-96a02191111b66819e07de179cb25a0e/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/bz2lib/1.0.6-d065ccd79984efc6d4660f410e4c81de/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/libuuid/2.34-27ce4c3579b5b1de2808ea9c4cd8ed29/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/protobuf/3.21.9-3b4dc3892e5c8da1829fdf0bfe570820/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/lcg/root/6.32.09-0a945d6e24dcaabe218b38fa8292785d/include -isystem/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/tbb/v2021.9.0-9b0f135342cfe979b6500c59f501774e/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/xz/5.2.5-6f3f49b07db84e10c9be594a1176c114/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/zlib/1.2.13-d217cdbdd8d586e845e05946de2796be/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/fmt/10.2.1-e35fd1db5eb3abc8ac0452e8ee427196/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/md5/1.0.0-5b594b264e04ae51e893b1d69a797ec6/include -I/cvmfs/cms-ib.cern.ch/sw/x86_64/nweek-02876/el8_amd64_gcc12/external/tinyxml2/6.2.0-f99ae2781d074227d47e8a3e7c8ec87e/include -O3 -pthread -pipe -Werror=main -Werror=pointer-arith -Werror=overlength-strings -Wno-vla -Werror=overflow -std=c++20 -ftree-vectorize -Werror=array-bounds -Werror=format-contains-nul -Werror=type-limits -fvisibility-inlines-hidden -fno-math-errno --param vect-max-version-for-alias-checks=50 -Xassembler --compress-debug-sections -Wno-error=array-bounds -Warray-bounds -fuse-ld=bfd -march=x86-64-v3 -felide-constructors -fmessage-length=0 -Wall -Wno-non-template-friend -Wno-long-long -Wreturn-type -Wextra -Wpessimizing-move -Wclass-memaccess -Wno-cast-function-type -Wno-unused-but-set-parameter -Wno-ignored-qualifiers -Wno-unused-parameter -Wunused -Wparentheses -Werror=return-type -Werror=missing-braces -Werror=unused-value -Werror=unused-label -Werror=address -Werror=format -Werror=sign-compare -Werror=write-strings -Werror=delete-non-virtual-dtor -Werror=strict-aliasing -Werror=narrowing -Werror=unused-but-set-variable -Werror=reorder -Werror=unused-variable -Werror=conversion-null -Werror=return-local-addr -Wnon-virtual-dtor -Werror=switch -fdiagnostics-show-option -Wno-unused-local-typedefs -Wno-attributes -Wno-psabi -Wno-error=unused-variable -DBOOST_DISABLE_ASSERTS -flto=auto -fipa-icf -flto-odr-type-merging -fno-fat-lto-objects -Wodr -fPIC -MMD -MF tmp/el8_amd64_gcc12/src/DQMServices/FwkIO/plugins/DQMServicesFwkIOPlugins/DQMRootSource.cc.d src/DQMServices/FwkIO/plugins/DQMRootSource.cc -o tmp/el8_amd64_gcc12/src/DQMServices/FwkIO/plugins/DQMServicesFwkIOPlugins/DQMRootSource.cc.o
src/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc: In member function 'void DQMRootOutputModule::startEndFile()':
src/DQMServices/FwkIO/plugins/DQMRootOutputModule.cc:545:22: error: 'const class edm::ProcessConfiguration' has no member named 'passID'; did you mean 'std::string edm::ProcessConfiguration::passID_'? (not accessible from this context)
  545 |       passID = itPC->passID();
      |                      ^~~~~~
In file included from src/DataFormats/Provenance/interface/ProcessHistory.h:4,
                 from src/DataFormats/Provenance/interface/StableProvenance.h:13,
                 from src/DataFormats/Provenance/interface/Provenance.h:13,

cmsbuild · 2025-02-14T23:20:05Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-47355/43703

There are other open Pull requests which might conflict with changes you have proposed:
- File DQMServices/FwkIO/plugins/DQMRootOutputModule.cc modified in PR(s): Implement TH2Poly in DQM Services for HGCal DQM #41932, [RFC] descriptions.addDefault -> descriptions.addWithDefaultLabel #44372
- File DQMServices/FwkIO/plugins/format.h modified in PR(s): Implement TH2Poly in DQM Services for HGCal DQM #41932

cmsbuild · 2025-02-14T23:20:23Z

Pull request #47355 was updated. @Dr15Jones, @antoniovagnerini, @cmsbuild, @makortel, @rseidita, @smuzaffar can you please check and sign again.

cmsbuild · 2025-02-19T22:13:10Z

+1

Size: This PR adds an extra 72KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3a0d8d/44516/summary.html
COMMIT: 020dcbb
CMSSW: CMSSW_15_1_X_2025-02-19-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/47355/44516/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 4 lines to the logs
Reco comparison results: 16 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3919988
DQMHistoTests: Total failures: 70
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3919898
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 214 log files, 184 edm output root files, 49 DQM output files
TriggerResults: no differences found

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 7
DQMHistoTests: Total histograms compared: 53071
DQMHistoTests: Total failures: 34
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 53037
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 6 files compared)
Checked 24 log files, 30 edm output root files, 7 DQM output files
TriggerResults: no differences found

makortel · 2025-02-19T23:35:06Z

CPU comparison differences are related to #47071

GPU comparison differences look compatible with the non-reproducibilities in the pixel code, except there is a new one (or a rediscovery of an old one) in ParticleFlow in 12834.42[23] workflows for which I opened a new issue #47406

makortel · 2025-02-20T14:28:16Z

+core

civanch · 2025-02-20T16:16:17Z

+1

makortel · 2025-02-24T14:34:40Z

@cms-sw/dqm-l2 @cms-sw/generators-l2 Could you review and sign, please? Thanks!

antoniovagnerini · 2025-02-24T17:07:29Z

+dqm

spurious differences from unstable WF 24834.911(open issue Non-reproducibility in TrackerPhase2OTL1Track #47071)

makortel · 2025-02-25T14:31:35Z

@cms-sw/generators-l2 Could you review and sign, please? I'd like to get the backport (#47416, which you signed already) to 15_0_0 if possible. Thanks!

lviliani · 2025-02-25T17:20:13Z

+1

cmsbuild · 2025-02-25T17:20:35Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @rappoccio, @antoniovilela, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2025-02-26T09:52:54Z

+1

cmsbuild added this to the CMSSW_15_1_X milestone Feb 14, 2025

cmsbuild added core-pending pending-signatures tests-pending orp-pending code-checks-pending labels Feb 14, 2025

makortel commented Feb 14, 2025

View reviewed changes

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 14, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 14, 2025

makortel mentioned this pull request Feb 14, 2025

Backport necessary developments to 15_0_X cms-sw/framework-team#1248

Open

2 tasks

cmsbuild added tests-rejected and removed tests-started labels Feb 14, 2025

makortel force-pushed the processConfigurationHardwareResourcesDescription branch from ec658cd to d168be9 Compare February 14, 2025 23:17

cmsbuild added dqm-pending tests-pending code-checks-pending and removed tests-rejected code-checks-approved labels Feb 14, 2025

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 14, 2025

cmsbuild added tests-started and removed tests-pending labels Feb 19, 2025

cmsbuild added tests-approved and removed tests-started labels Feb 19, 2025

makortel mentioned this pull request Feb 19, 2025

Non-reproducibility in ParticleFlow/PFClusterV in 12834.42[23] GPU workflows #47406

Open

cmsbuild added core-approved and removed core-pending labels Feb 20, 2025

makortel mentioned this pull request Feb 20, 2025

[15_0_X] Use HardwareResourcesDescription in ProcessConfiguration #47416

Open

cmsbuild added simulation-approved and removed simulation-pending labels Feb 20, 2025

cmsbuild added dqm-approved and removed dqm-pending labels Feb 24, 2025

cmsbuild added generators-approved fully-signed and removed generators-pending pending-signatures labels Feb 25, 2025

cmsbuild added orp-approved and removed orp-pending labels Feb 26, 2025

cmsbuild merged commit fdd03a5 into cms-sw:master Feb 26, 2025
14 checks passed

makortel deleted the processConfigurationHardwareResourcesDescription branch February 26, 2025 14:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use HardwareResourcesDescription in ProcessConfiguration #47355

Use HardwareResourcesDescription in ProcessConfiguration #47355

makortel commented Feb 14, 2025 •

edited

Loading

cmsbuild commented Feb 14, 2025 •

edited

Loading

makortel Feb 14, 2025

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

makortel commented Feb 14, 2025

makortel commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 19, 2025

makortel commented Feb 19, 2025

makortel commented Feb 20, 2025

civanch commented Feb 20, 2025

makortel commented Feb 24, 2025

antoniovagnerini commented Feb 24, 2025

makortel commented Feb 25, 2025

lviliani commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

mandrenguyen commented Feb 26, 2025

Use HardwareResourcesDescription in ProcessConfiguration #47355

Use HardwareResourcesDescription in ProcessConfiguration #47355

Conversation

makortel commented Feb 14, 2025 • edited Loading

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

cmsbuild commented Feb 14, 2025 • edited Loading

makortel Feb 14, 2025

Choose a reason for hiding this comment

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

makortel commented Feb 14, 2025

makortel commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

Build

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 14, 2025

cmsbuild commented Feb 19, 2025

Comparison Summary

GPU Comparison Summary

makortel commented Feb 19, 2025

makortel commented Feb 20, 2025

civanch commented Feb 20, 2025

makortel commented Feb 24, 2025

antoniovagnerini commented Feb 24, 2025

makortel commented Feb 25, 2025

lviliani commented Feb 25, 2025

cmsbuild commented Feb 25, 2025

mandrenguyen commented Feb 26, 2025

makortel commented Feb 14, 2025 •

edited

Loading

cmsbuild commented Feb 14, 2025 •

edited

Loading