add option for higher-stat relval tests

[kpedro88]

Request from (cms-sw/cmssw#21701 (comment)):

did you run any larger-scale samples for physics validation? It's hard to tell from the PR test plots (just some fluctuations)

Since your request is, of course, valid and has been raised few other times for other PRs, may I suggest to add a special trigger, like please histat_test to the bot, so that those kinds of checks can be done centrally on some handful samples, rather than relying on the good-will and machine availability of people submitting the PR?

This seems like a valid request to automate in-depth physics validation when needed for PRs. Most PRs won't need this.

One potential issue is that the high-stat samples would also be needed in the IB baselines. We could run them for all IB baselines, or just run them on-demand (somehow keeping track when they're already available so they don't get rerun unnecessarily).

We might want to limit this option to request high stats for a specific workflow, since most workflows may not need physics validation in any given case.

[smuzaffar]

@kpedro88 , #1716 now allows bot to generate baseline for extra workflows tested for PR and use it during PR comparison tests. What else is needed in order to to complete high-stat relval tests?

[cmsbuild]

A new Issue was created by @kpedro88 Kevin Pedro.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[kpedro88]

@smuzaffar to confirm, if I request (using cms-bot test parameters relvals_opt) more events for standard workflows (that already have 10-event samples generated in IBs and PR tests), will the bot recognize that this is a "different" workflow that needs to be rerun in the IB as well, to provide the comparison?

[smuzaffar]

@kpedro88 , no , bot does not change the event numbers. We can add a separate test e.g. enable high-stat ( just like enable gpu ) which can run some selected workflows with more event.

[kpedro88]

@smuzaffar okay, I think that would resolve this issue.

[smuzaffar]

we should come up with a list of workflows and default number of events for high-stats tests. Do you have any workflow in mind which I can use to test?

[kpedro88]

35034.0 is the current Phase 2 baseline workflow. 11634.0 might be useful to have a Run 3 test for changes proposed with a shorter timescale.

@cms-sw/upgrade-l2 should comment if they have other preferences.

[kpedro88]

Default number of events, maybe 1000? This could be a configurable parameter of the enable high-stat option.

[AdrianoDee]

@kpedro88 35034.0 is fine for Phase2.

[smuzaffar]

@kpedro88 , PR #1728 should allow to run high_stats relvals and comparison . Currently bot runs 35034.0 with 1000 events for high_stats tests. If you have any cmssw PR to test this then please use

test parameters:
- pull_request = cms-sw/cms-bot#1728
- enable_test = high_stats

Note that as number of events should remain same for both baseline and PR, so it is not possible to change the evnets via Pr comment.

[smuzaffar]

#1728 has been merged and one should be able to request high_stats PR tests . Note that currently it runs workflow 35034.0 with 500 events. 1000 event takes too much time (over 5 hours) and generate over 10GB of data.

[smuzaffar]

high stats relvals tests and comparisons are now supported