Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

[performanceautofiler]

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	95f7f7a026d74e0720d0dfdf2de799933b832df2
Compare	a3f7d295a93f2a25cd6616d7d209ba43fbeaf481
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Ctor - Duration of single invocation	28.63 μs	35.91 μs	1.25	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Ctor(Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.90503535932722 > 30.192944860854865.
IsChangePoint: Marked as a change because one of 1/19/2022 10:20:02 AM, 2/25/2022 8:42:58 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsRegressionStdDev: Marked as regression because -12.461424623881534 (T) = (0 -34419.26309278616) / Math.Sqrt((94699.13549592457 / (56)) + (2098647.1962386263 / (10))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (56) + (10) - 2, .025) and -0.19979555323834208 = (28687.60681758311 - 34419.26309278616) / 28687.60681758311 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[kunalspathak]

Two regressions as seen below:

#65734 and #65473

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[stephentoub]

@joperezr, this is the one you were expecting, right?

[ghost]

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Run Information

Architecture	x86
OS	Windows 10.0.18362
Baseline	95f7f7a026d74e0720d0dfdf2de799933b832df2
Compare	a3f7d295a93f2a25cd6616d7d209ba43fbeaf481
Diff	Diff

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Ctor - Duration of single invocation	28.63 μs	35.91 μs	1.25	0.01	False

Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Ctor(Options: Compiled)

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.90503535932722 > 30.192944860854865.
IsChangePoint: Marked as a change because one of 1/19/2022 10:20:02 AM, 2/25/2022 8:42:58 PM, 3/1/2022 5:49:50 AM falls between 2/20/2022 5:50:14 AM and 3/1/2022 5:49:50 AM.
IsRegressionStdDev: Marked as regression because -12.461424623881534 (T) = (0 -34419.26309278616) / Math.Sqrt((94699.13549592457 / (56)) + (2098647.1962386263 / (10))) is less than -1.997729654317006 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (56) + (10) - 2, .025) and -0.19979555323834208 = (28687.60681758311 - 34419.26309278616) / 28687.60681758311 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	-
Labels:	area-System.Text.RegularExpressions, tenet-performance, tenet-performance-benchmarks, untriaged, refs/heads/main, Regression, RunKind=micro, CoreClr, Windows 10.0.18362, x86
Milestone:	-

[joperezr]

mm not exactly, the one that I was expecting (and already commented on) was this one: #66010 which is for the CtorInvoke test when using Compiled engine. The difference between them is that this one is doing new Regex(WarningPattern, Options); while the one I was expecting was doing new Regex(WarningPattern, Options).IsMatch(@"(1");. When I tested locally this particular benchmark, I saw that the results before and after my change were very similar if not the same, since we aren't really touching anything with the constructor flow, and only really affecting the actual match time. I will take a closer look at this one to see why is there a delta.

[stephentoub]

This has recovered.