python 3.9.6 and py2 cleanup

[smuzaffar]

• Update python to 3.9.6
• Cleanup build-with-pip to only support py3
• Cleanup python2 based macros

[cmsbuild]

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for branch IB/CMSSW_12_0_X/master.

@cmsbuild, @smuzaffar, @mrodozov, @iarspider can you please review it and eventually sign? Thanks.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.
cms-bot commands are listed here

[smuzaffar]

please test

[cmsbuild]

Pull request #7159 was updated.

[smuzaffar]

FYI @davidlange6 , along with python 3.9.6 updates, this Pr also cleans up build-with-pip.file to only build for py3. requirement.txt file alsocleaned up to not have python_version>3.0

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17094/summary.html
COMMIT: 169610d
CMSSW: CMSSW_12_0_X_2021-07-21-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7159/17094/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation warning when building: See details on the summary page.

[smuzaffar]

please test

[cmsbuild]

Pull request #7159 was updated.

[smuzaffar]

test parameters:

• full_cmssw = true

[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17096/summary.html
COMMIT: 75b4f82
CMSSW: CMSSW_12_0_X_2021-07-21-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7159/17096/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ tar -xzf /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/openloops-process/2.1.2-d8de6d1c641584c306538e0abf4291dc/process_src.tgz
tar (child): /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/openloops-process/2.1.2-d8de6d1c641584c306538e0abf4291dc/process_src.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.lSiFEJ (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.lSiFEJ (%build)


[cmsbuild]

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17108/summary.html
COMMIT: 75b4f82
CMSSW: CMSSW_12_0_X_2021-07-22-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7159/17108/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ tar -xzf /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/openloops-process/2.1.2-d8de6d1c641584c306538e0abf4291dc/process_src.tgz
tar (child): /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/slc7_amd64_gcc900/external/openloops-process/2.1.2-d8de6d1c641584c306538e0abf4291dc/process_src.tgz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.jzzGnY (%build)


RPM build errors:
Macro %rpmbuild_libdir defined but not used within scope
Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.jzzGnY (%build)


[cmsbuild]

Pull request #7159 was updated.

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17110/summary.html
COMMIT: 311b37e
CMSSW: CMSSW_12_0_X_2021-07-22-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7159/17110/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @jeongeun Migrate module configuration in RecoJets to use default cfipython cmssw#34543
• @davidlange6 Remove six python2->3 compatibility helper usage cmssw#34534
• @fabferro PPS geometry bug fix: missing file cmssw#34588

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17110/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17110/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test trackerMaterialAnalysisPlots had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
---> test materialBudgetHGCalPlots had ERRORS

Comparison Summary

There are some workflows for which there are errors in the baseline:
11634.911 step 1
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 38
• DQMHistoTests: Total histograms compared: 2788434
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2788411
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
• Checked 161 log files, 37 edm output root files, 38 DQM output files
• TriggerResults: no differences found

[smuzaffar]

please test

[cmsbuild]

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17129/summary.html
COMMIT: 311b37e
CMSSW: CMSSW_12_0_X_2021-07-22-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/7159/17129/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• @cms-sw Fix python toolfile to use python2 version #7163

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17129/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4a8f5d/17129/git-merge-result

Unit Tests

I found errors in the following unit tests:

---> test trackerMaterialAnalysisPlots had ERRORS
---> test materialBudgetTrackerPlots had ERRORS
---> test materialBudgetHGCalPlots had ERRORS

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2998535
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 38 files compared)
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: no differences found

[smuzaffar]

+externals
lets get python 3.9.6 in 12.0.X

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next IB/CMSSW_12_0_X/master IBs (but tests are reportedly failing). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)