Further cleanup of Condition access in AlCaDB, online DQM, Geometry, HLT, Simulation, and standard configuration; Further fixes for online DQM

[diguida]

This is the next step of cleanup and improvement for condition access:

• Revision of the configuration initialisation and fragments for Condition
  access:
  • Added CondCore/ESSources/python/CondDBESSources_condDBv2_cfi.py in order to keep compatibility with the explicit or implicit usage of the condition software version;
  • Added Configuration/StandardSequences/python/AdditionalConditions_cff.py: this fragment is used in order to load conditions not provided by Global Tag and/or Condition records;
  • Dropped Configuration/StandardSequences/python/FrontierConditions_GlobalTag_cfi.py and Configuration/StandardSequences/python/FrontierConditions_GlobalTag_condDBv2_cfi.py: they are not needed, as the initialisation of the ESSource for loading conditions is done in the proper package;
  • Dropped references to Configuration/StandardSequences/python/FrontierConditions_GlobalTag_cfi.pyin client code.
  • Improved deprecation message for modules CondCore/DBCommon/python/CondDBCommon_cfi.py and CondCore/DBCommon/python/CondDBSetup_cfi.py.
• Define the default connection string after checking the cluster where the job is run: if in GPN or WAN, use the usual one, if in the online network, fetch conditions from the squid running on localhost.
• Revision of Condition access in Online DQM:
  • Cleanup of DQM/Integration/python/config/FrontierCondition_GT_cfi.py and DQM/Integration/python/config/FrontierCondition_GT_autoExpress_cfi.py: the connection string is established centrally;
  • Dropped DQM/Integration/python/config/FrontierCondition_GT_Offline_cfi.py;
  • Correspondent changes in client applications, where the GT customisation is enabled.
• Clean-up some BuildFile from dependencies upon removed packages.
• Improved test configuration file.
• Cleaned up spurious includes of CondDBSetup_cfi and CondDBCommon_cfi in CondTools/DQM/python/DQMReferenceHistogramRootFileEventSetupAnalyzer_OrcoffOnly_cfi.py,
  CondTools/DQM/python/DQMReferenceHistogramRootFileEventSetupAnalyzer_SQLiteOnly_cfi.py, CondTools/DQM/python/DQMReferenceHistogramRootFileEventSetupAnalyzer_cfi.py, Configuration/StandardSequences/python/RawToDigi_cff.py, and SimCalorimetry/HcalSimProducers/python/hcalUnsuppressedDigis_cfi.py.

These are the fixes in order to run online DQM with 80X:

• Migration to consumes in DPGAnalysis/Skims/interface/PhysDecl.h, DQM/TrigXMonitor/interface/L1Scalers.h, DQM/TrigXMonitor/interface/L1TScalersSCAL.h, EventFilter/L1GlobalTriggerRawToDigi/interface/L1GlobalTriggerRecordProducer.h;
• Use latest HLT and express Global Tags;
• Load RECO geometry from GT in DQM/Integration/python/clients/ecal_dqm_sourceclient-live_cfg.py, DQM/Integration/python/clients/ecalcalib_dqm_sourceclient-live_cfg.py, DQM/Integration/python/clients/hcal_dqm_sourceclient-live_cfg.py
• Fixes in DQM/Integration/python/clients/l1tstage2_dqm_sourceclient-live_cfg.py
• Fixes in DQM/DTMonitorModule/python/dt_dqm_sourceclient_common_cff.py and DQM/Integration/python/clients/dt_dqm_sourceclient-live_cfg.py for loading correctly DT TwinMux sequence.

Originally in #13399

Unit tests ran OK on CMSSW_8_0_0

$ git cms-ckeckdeps -a
$ scram b -r -j 4 runtests

Short matrix tests passed, and, as expected, no CondDB python cfi deprecation warning is seen on CMSSW_8_0_0

4.22_RunCosmics2011A+RunCosmics2011A+RECOCOSD+ALCACOSD+SKIMCOSD+HARVESTDC Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Tue Feb  2 15:16:58 2016-date Tue Feb  2 15:12:56 2016; exit: 0 0 0 0 0
5.1_TTbar+TTbarFS+HARVESTFS Step0-PASSED Step1-PASSED  - time date Tue Feb  2 15:18:48 2016-date Tue Feb  2 15:12:57 2016; exit: 0 0
8.0_BeamHalo+BeamHaloINPUT+DIGICOS+RECOCOS+ALCABH+HARVESTCOS Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Tue Feb  2 15:17:01 2016-date Tue Feb  2 15:13:00 2016; exit: 0 0 0 0 0
25.0_TTbar+TTbarINPUT+DIGI+RECOAlCaCalo+HARVEST+ALCATT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Tue Feb  2 15:20:38 2016-date Tue Feb  2 15:13:04 2016; exit: 0 0 0 0 0
134.911_RunSinglePh2015D+RunSinglePh2015D+HLTDR2_25ns+RECODR2_25nsreHLT+HARVESTDR2_25nsreHLT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Tue Feb  2 15:26:21 2016-date Tue Feb  2 15:16:59 2016; exit: 0 0 0 0
135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS Step0-PASSED Step1-PASSED Step2-PASSED  - time date Tue Feb  2 15:29:02 2016-date Tue Feb  2 15:17:11 2016; exit: 0 0 0
140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI Step0-PASSED Step1-PASSED Step2-PASSED  - time date Tue Feb  2 15:22:55 2016-date Tue Feb  2 15:18:55 2016; exit: 0 0 0
1330.0_ZMM_13+ZMM_13INPUT+DIGIUP15+RECOUP15+HARVESTUP15 Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED  - time date Tue Feb  2 15:31:50 2016-date Tue Feb  2 15:20:46 2016; exit: 0 0 0 0
1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED  - time date Tue Feb  2 15:30:41 2016-date Tue Feb  2 15:22:57 2016; exit: 0 0 0 0 0
1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4 Step0-PASSED Step1-PASSED Step2-PASSED Step3-PASSED Step4-PASSED Step5-PASSED Step6-PASSED  - time date Tue Feb  2 15:33:40 2016-date Tue Feb  2 15:26:31 2016; exit: 0 0 0 0 0 0 0
10 10 9 7 5 1 1 tests passed, 0 0 0 0 0 0 0 failed

Full matrix test running on CMSSW_8_0_0.

No regressions expected.

[diguida]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10968/console

[cmsbuild]

A new Pull Request was created by @diguida (Salvatore Di Guida) for CMSSW_8_0_X.

It involves the following packages:

Alignment/HIPAlignmentAlgorithm
Alignment/OfflineValidation
CalibCalorimetry/CastorCalib
CalibCalorimetry/HcalAlgos
CondCore/DBCommon
CondCore/ESSources
CondTools/HLT
Configuration/AlCa
Configuration/StandardSequences
DQM/BeamMonitor
DQM/DTMonitorModule
DQM/Integration
Geometry/CSCGeometry
HLTrigger/HLTanalyzers
HLTrigger/special
SimMuon/GEMDigitizer
SimMuon/RPCDigitizer
Validation/Geometry

@perrotta, @cmsbuild, @civanch, @diguida, @ianna, @mdhildreth, @Martin-Grunewald, @franzoni, @Dr15Jones, @cerminar, @deguio, @fwyzard, @ggovi, @mmusich, @vanbesien, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @apfeiffer1, @batinkov, @makortel, @GiacomoSguazzoni, @tocheng, @VinInn, @Martin-Grunewald, @rovere, @tlampen, @mschrode, @cerati, @mmusich, @threus, @dgulhan, @jhgoh this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

[deguio]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10970/console

[Dr15Jones]

+1

[cmsbuild]

+1
Tested at: 367ee64
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-13170/10968/summary.html

[ianna]

+1

[civanch]

+1

[deguio]

+1

[mmusich]

+1

[mulhearn]

+1

[ggovi]

+1

[dmitrijus]

Sorry for resurrecting this PR.
Why isn't this merged? Let's make a decision: either to merge it or to close this PR.

[dmitrijus]

+1

[dmitrijus]

For some unknown reason it was dropped from our p5 tag collector page, so that is why there was no follow-up. I will ping @diguida .

[diguida]

@dmitrijus sorry for belated reply.
This PR was never deployed in online DQM: we discussed within DQM and we decided not to deploy it standalone, but wait for central validation, as it completely re-engineers the configuration for condition access in CMSSW.
OTOH, the 81X PR was integrated, the newer pre-releases are running fine with it. As a consequence, we have now two options:

1. we back-port the 81X PR in 80X: this implies closing this PR and making a new one on top of the latest commits in the 80X branch.
2. we decide not to implement this feature in 80X: the PR is closed, and the deployment of 81X for production will feature the revision of the condition access.

Hope this clarifies

[diguida]

I have no strong opinion, but I think option 1 is preferable. I let L2 and L1 conveners comment further.

[davidlange6]

complete re-engineerings are typically not back ported. I'll just close this.