Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Regressions in ICU globalization microbenchmarks #77730

Closed
performanceautofiler bot opened this issue Nov 1, 2022 · 33 comments
Closed

Regressions in ICU globalization microbenchmarks #77730

performanceautofiler bot opened this issue Nov 1, 2022 · 33 comments
Assignees
@DrewScoggins
Labels
area-GC-coreclr
Milestone
8.0.0

Comments

@performanceautofiler
Copy link

Run Information

Architecture x64
OS alpine 3.15
Baseline f4b715f1cfadcbf85486f369724528c051be5733
Compare a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsSuffix_DifferentLastChar - Duration of single invocation 695.14 ns 3.48 μs 5.00 0.13 False
IsPrefix_FirstHalf - Duration of single invocation 2.12 μs 5.46 μs 2.57 0.08 False
IsSuffix_DifferentLastChar - Duration of single invocation 696.65 ns 781.47 ns 1.12 0.07 False
IsPrefix_DifferentFirstChar - Duration of single invocation 504.16 ns 533.12 ns 1.06 0.04 False
IsSuffix_SecondHalf - Duration of single invocation 3.79 μs 6.71 μs 1.77 0.04 False
IsPrefix_FirstHalf - Duration of single invocation 2.07 μs 4.70 μs 2.27 0.07 False
IsSuffix_DifferentLastChar - Duration of single invocation 640.10 ns 4.60 μs 7.19 0.12 False
IsSuffix_SecondHalf - Duration of single invocation 3.80 μs 7.70 μs 2.03 0.04 False
IsSuffix_DifferentLastChar - Duration of single invocation 669.14 ns 738.35 ns 1.10 0.07 False
IsPrefix_DifferentFirstChar - Duration of single invocation 496.41 ns 551.52 ns 1.11 0.02 False
IsPrefix_DifferentFirstChar - Duration of single invocation 494.44 ns 2.58 μs 5.22 0.08 False
IsPrefix_DifferentFirstChar - Duration of single invocation 497.24 ns 2.58 μs 5.18 0.11 False
IsSuffix_DifferentLastChar - Duration of single invocation 717.23 ns 762.39 ns 1.06 0.05 False
IsPrefix_DifferentFirstChar - Duration of single invocation 511.32 ns 557.37 ns 1.09 0.04 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, None, True))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 3.4778568875381324 > 737.7096710539914.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.4434562500556 (T) = (0 -3509.878318063499) / Math.Sqrt((75.73519679229132 / (22)) + (192312.28640823922 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -3.9835693731756994 = (704.2900490069602 - 3509.878318063499) / 704.2900490069602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.463388254122405 > 2.1683168435435682.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -35.92694617570154 (T) = (0 -4866.389786775149) / Math.Sqrt((1081.1735463180112 / (22)) + (177446.77020579352 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.3798745308588118 = (2044.8093896021664 - 4866.389786775149) / 2044.8093896021664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 781.4682706839588 > 741.5567027451641.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -8.87722642300704 (T) = (0 -774.2461878940562) / Math.Sqrt((368.9991682802959 / (22)) + (748.8526707185031 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08088299272502637 = (716.3089743341186 - 774.2461878940562) / 716.3089743341186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 533.1238339243107 > 529.0981993486763.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -25.158729650032445 (T) = (0 -541.8583535099809) / Math.Sqrt((41.9013980568892 / (22)) + (19.89834355128826 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08076992418216662 = (501.3632794417485 - 541.8583535099809) / 501.3632794417485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 6.714073450271101 > 3.9794821239877067.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -31.92475821744105 (T) = (0 -7190.940700508054) / Math.Sqrt((702.3535633842807 / (22)) + (327517.3024920029 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8956219622786153 = (3793.446606761323 - 7190.940700508054) / 3793.446606761323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.704891103454969 > 2.176003494929239.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.34595255971069 (T) = (0 -4948.7381858052795) / Math.Sqrt((609.0208225895935 / (22)) + (205537.70176964914 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.4121328292796187 = (2051.6026836230308 - 4948.7381858052795) / 2051.6026836230308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.604345788545753 > 685.9526031478639.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -32.97429061611585 (T) = (0 -3560.528560361033) / Math.Sqrt((84.24666816328991 / (22)) + (225224.56867890482 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.445169009547024 = (653.8876119581141 - 3560.528560361033) / 653.8876119581141 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.697149280985077 > 3.986216307087259.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -28.92035721425405 (T) = (0 -7102.175401152876) / Math.Sqrt((983.2222021904659 / (22)) + (374872.3601622246 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8648780206002814 = (3808.38602991673 - 7102.175401152876) / 3808.38602991673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (pl-PL, None, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 738.3453194242094 > 686.7923447017769.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.644301811770973 (T) = (0 -730.5555736984468) / Math.Sqrt((133.09015889314318 / (22)) + (286.94769643140063 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10007798112031158 = (664.0943517062801 - 730.5555736984468) / 664.0943517062801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 551.5152330302398 > 523.1353566564028.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -17.48624130281412 (T) = (0 -548.8303287199552) / Math.Sqrt((8.524478883162635 / (22)) + (226.46572909951897 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10037282183450977 = (498.7676156931622 - 548.8303287199552) / 498.7676156931622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.582708715125821 > 525.0820958060093.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -36.484324551519855 (T) = (0 -2963.0866452484565) / Math.Sqrt((28.342791575091777 / (22)) + (132096.2924956813 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.9217634702666535 = (500.3723401189866 - 2963.0866452484565) / 500.3723401189866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.576881128929599 > 523.2868817979111.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -26.485367207732722 (T) = (0 -2956.1671188974465) / Math.Sqrt((12.437656857148067 / (22)) + (249680.1531075944 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.929465634472556 = (498.5554013014205 - 2956.1671188974465) / 498.5554013014205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 762.3864551566398 > 752.768160745577.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.074856176912661 (T) = (0 -776.4127024266257) / Math.Sqrt((246.3782718320968 / (22)) + (378.96674779275503 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08296818049824481 = (716.9302998998724 - 776.4127024266257) / 716.9302998998724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (pl-PL, None, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 557.3742612611512 > 532.3676234159848.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.102683564556107 (T) = (0 -551.3130269614716) / Math.Sqrt((147.2814912162584 / (22)) + (294.85844611173354 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08137099633791905 = (509.82782858843296 - 551.3130269614716) / 509.82782858843296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository
@performanceautofiler performanceautofiler bot added CoreClr untriaged New issue has not been triaged by the area owner labels Nov 1, 2022
@performanceautofiler performanceautofiler bot mentioned this issue Nov 1, 2022
Closed
@AndyAyersMS AndyAyersMS removed the untriaged New issue has not been triaged by the area owner label Nov 1, 2022
@AndyAyersMS
Copy link
Member

Likely #77336

@AndyAyersMS AndyAyersMS transferred this issue from dotnet/perf-autofiling-issues Nov 1, 2022
@dotnet-issue-labeler
Copy link

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

@ghost ghost added the untriaged New issue has not been triaged by the area owner label Nov 1, 2022
This was referenced Nov 1, 2022
Closed
Closed
Closed
@AndyAyersMS AndyAyersMS changed the title [Perf] Alpine/x64: 14 Regressions on 10/25/2022 10:16:52 PM Regressions from Array.Initialize change Nov 1, 2022
This was referenced Nov 1, 2022
Closed
Closed
Closed
@AndyAyersMS
Copy link
Member

Might also be #76985.

@EgorBo
Copy link
Member

EgorBo commented Nov 3, 2022

Regressions on arm64 from the same commit range: dotnet/perf-autofiling-issues#9485

This was referenced Nov 3, 2022
Closed
Closed
Closed
@jkotas
Copy link
Member

jkotas commented Nov 4, 2022

I am not able to reproduce this regression. I am pretty sure that the regression is not caused by the Array.Initialize change (this method is not called by the microbenchmarks at all) nor the GC changes (the microbenchmarks do not allocate anything).

The regressed microbenchmarks typically spent 90+% time in ICU library that comes with the OS. Notice that all regressed cases have HighChars = true or non-US cultures that forces the globalization libraries take the slow path by calling ICU library.

The most likely explanation of these regressions is that the system ICU library or globalization settings were updated on the perf machines.

@AndyAyersMS
Copy link
Member

The issues from https://github.com/dotnet/perf-autofiling-issues/ repo resolved as duplicates look unrelated. They should be retrigged if they are real regressions.

Sorry about that -- the cross-linking the auto filing system is doing has changed recently and we struggling to make sense of things; seems like a fair number of unrelated things are getting lumped together.

@tarekgh
Copy link
Member

tarekgh commented Nov 8, 2022

@jkotas @EgorBo @AndyAyersMS this became clear it is not a globlaization/ICU issue. Any suggestion for who should look at this?

Side question, do we support building the runtime repo on Alpine? I am asking because I failed to build it. I am getting errors like:

 CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
  CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
  CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage

@jkotas
Copy link
Member

jkotas commented Nov 8, 2022

@sblom 's team should look at this and provide access to the environment that reproduces the worst (up to 7x) regressions. I expect that we are going to find that this regression was caused by some kind of change in the environment setup.

@tarekgh tarekgh removed the untriaged New issue has not been triaged by the area owner label Nov 9, 2022
@tarekgh tarekgh added this to the Future milestone Nov 9, 2022
@tarekgh tarekgh added the needs-author-action An issue or pull request that requires more info or actions from the author. label Nov 9, 2022
@caaavik-msft
Copy link
Contributor

caaavik-msft commented Nov 9, 2022 •
edited
Loading

I've created a branch internally forking off of commit 9b16818a which was the commit before the regression was introduced and ran the perf pipeline on it again just now using the current environment and I can't reproduce the regression, it is still showing a runtime of 656.2ns which is what it was before.

StringSearch.IsSuffix_DifferentLastChar: Job-ZFWQTY(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=CoreRun, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, WarmupCount=1) [Options=(, None, True)]
Runtime = .NET 8.0.0 (42.42.42.42424), X64 RyuJIT AVX2; GC = Concurrent Workstation
Mean = 656.157 ns, StdErr = 1.859 ns (0.28%), N = 14, StdDev = 6.954 ns
Min = 638.194 ns, Q1 = 653.613 ns, Median = 657.158 ns, Q3 = 661.107 ns, Max = 663.317 ns
IQR = 7.494 ns, LowerFence = 642.371 ns, UpperFence = 672.348 ns
ConfidenceInterval = [648.312 ns; 664.002 ns] (CI 99.9%), Margin = 7.845 ns (1.20% of Mean)
Skewness = -1.08, Kurtosis = 3.52, MValue = 2
-------------------- Histogram --------------------
[634.407 ns ; 667.104 ns) | @@@@@@@@@@@@@@

This to me indicates that it's not caused by a change in the environment setup, but caused by changes between commit 9b16818a and commit 54b12a83. I will try run the pipeline on commits between these two versions to see if I can isolate which one might have caused the issue.

@caaavik-msft
Copy link
Contributor

I ran the perf pipeline on commit c4341d45 which is a commit in between the two mentioned in the previous comment and I was able to reproduce the perf regression on it. The only other commits between 9b16818 and c4341d4 were definitely unrelated, so I think it is likely that the static linking GC PAL on linux commit is responsible for this regression somehow.

I've reached out to dnceng servies to see what I can do about getting access to a machine with the same/similar specs so I can collect some traces and we can better analyse the issue. Will update this issue when I have more information about that.

FYI @jkotas

@tarekgh tarekgh modified the milestones: Future, 8.0.0 Nov 9, 2022
@tarekgh
Copy link
Member

tarekgh commented Nov 9, 2022 •
edited
Loading

Thanks @caaavik-msft. I changed the area label to GC.

For future investigations, do you have a constructions how we can run the test against different commits? I was trying to build the source on the Alpine container to run there but I couldn't build it.

@ghost
Copy link

ghost commented Nov 9, 2022

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Architecture x64
OS alpine 3.15
Baseline f4b715f1cfadcbf85486f369724528c051be5733
Compare a8ea8e67dc20689aa2e4c738a2f84ea3483ec872
Diff Diff

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio Baseline ETL Compare ETL
IsSuffix_DifferentLastChar - Duration of single invocation 695.14 ns 3.48 μs 5.00 0.13 False
IsPrefix_FirstHalf - Duration of single invocation 2.12 μs 5.46 μs 2.57 0.08 False
IsSuffix_DifferentLastChar - Duration of single invocation 696.65 ns 781.47 ns 1.12 0.07 False
IsPrefix_DifferentFirstChar - Duration of single invocation 504.16 ns 533.12 ns 1.06 0.04 False
IsSuffix_SecondHalf - Duration of single invocation 3.79 μs 6.71 μs 1.77 0.04 False
IsPrefix_FirstHalf - Duration of single invocation 2.07 μs 4.70 μs 2.27 0.07 False
IsSuffix_DifferentLastChar - Duration of single invocation 640.10 ns 4.60 μs 7.19 0.12 False
IsSuffix_SecondHalf - Duration of single invocation 3.80 μs 7.70 μs 2.03 0.04 False
IsSuffix_DifferentLastChar - Duration of single invocation 669.14 ns 738.35 ns 1.10 0.07 False
IsPrefix_DifferentFirstChar - Duration of single invocation 496.41 ns 551.52 ns 1.11 0.02 False
IsPrefix_DifferentFirstChar - Duration of single invocation 494.44 ns 2.58 μs 5.22 0.08 False
IsPrefix_DifferentFirstChar - Duration of single invocation 497.24 ns 2.58 μs 5.18 0.11 False
IsSuffix_DifferentLastChar - Duration of single invocation 717.23 ns 762.39 ns 1.06 0.05 False
IsPrefix_DifferentFirstChar - Duration of single invocation 511.32 ns 557.37 ns 1.09 0.04 False

graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
graph
Test Report

Repro

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net6.0 --filter 'System.Globalization.Tests.StringSearch*'

Related Issues

Regressions

Improvements

Payloads

Baseline
Compare

Histogram

Edge Detector Info

Collection Data

System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, None, True))

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 3.4778568875381324 > 737.7096710539914.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.4434562500556 (T) = (0 -3509.878318063499) / Math.Sqrt((75.73519679229132 / (22)) + (192312.28640823922 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -3.9835693731756994 = (704.2900490069602 - 3509.878318063499) / 704.2900490069602 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 5.463388254122405 > 2.1683168435435682.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -35.92694617570154 (T) = (0 -4866.389786775149) / Math.Sqrt((1081.1735463180112 / (22)) + (177446.77020579352 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.3798745308588118 = (2044.8093896021664 - 4866.389786775149) / 2044.8093896021664 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 781.4682706839588 > 741.5567027451641.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -8.87722642300704 (T) = (0 -774.2461878940562) / Math.Sqrt((368.9991682802959 / (22)) + (748.8526707185031 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08088299272502637 = (716.3089743341186 - 774.2461878940562) / 716.3089743341186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 533.1238339243107 > 529.0981993486763.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -25.158729650032445 (T) = (0 -541.8583535099809) / Math.Sqrt((41.9013980568892 / (22)) + (19.89834355128826 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08076992418216662 = (501.3632794417485 - 541.8583535099809) / 501.3632794417485 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 6.714073450271101 > 3.9794821239877067.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -31.92475821744105 (T) = (0 -7190.940700508054) / Math.Sqrt((702.3535633842807 / (22)) + (327517.3024920029 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8956219622786153 = (3793.446606761323 - 7190.940700508054) / 3793.446606761323 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_FirstHalf(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.704891103454969 > 2.176003494929239.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -34.34595255971069 (T) = (0 -4948.7381858052795) / Math.Sqrt((609.0208225895935 / (22)) + (205537.70176964914 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -1.4121328292796187 = (2051.6026836230308 - 4948.7381858052795) / 2051.6026836230308 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 4.604345788545753 > 685.9526031478639.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -32.97429061611585 (T) = (0 -3560.528560361033) / Math.Sqrt((84.24666816328991 / (22)) + (225224.56867890482 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.445169009547024 = (653.8876119581141 - 3560.528560361033) / 653.8876119581141 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 7.697149280985077 > 3.986216307087259.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -28.92035721425405 (T) = (0 -7102.175401152876) / Math.Sqrt((983.2222021904659 / (22)) + (374872.3601622246 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.8648780206002814 = (3808.38602991673 - 7102.175401152876) / 3808.38602991673 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (pl-PL, None, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 738.3453194242094 > 686.7923447017769.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -16.644301811770973 (T) = (0 -730.5555736984468) / Math.Sqrt((133.09015889314318 / (22)) + (286.94769643140063 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10007798112031158 = (664.0943517062801 - 730.5555736984468) / 664.0943517062801 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 551.5152330302398 > 523.1353566564028.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -17.48624130281412 (T) = (0 -548.8303287199552) / Math.Sqrt((8.524478883162635 / (22)) + (226.46572909951897 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.10037282183450977 = (498.7676156931622 - 548.8303287199552) / 498.7676156931622 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (en-US, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.582708715125821 > 525.0820958060093.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -36.484324551519855 (T) = (0 -2963.0866452484565) / Math.Sqrt((28.342791575091777 / (22)) + (132096.2924956813 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.9217634702666535 = (500.3723401189866 - 2963.0866452484565) / 500.3723401189866 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (, None, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 2.576881128929599 > 523.2868817979111.
IsChangePoint: Marked as a change because one of 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -26.485367207732722 (T) = (0 -2956.1671188974465) / Math.Sqrt((12.437656857148067 / (22)) + (249680.1531075944 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -4.929465634472556 = (498.5554013014205 - 2956.1671188974465) / 498.5554013014205 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (en-US, IgnoreCase, True))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 762.3864551566398 > 752.768160745577.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -12.074856176912661 (T) = (0 -776.4127024266257) / Math.Sqrt((246.3782718320968 / (22)) + (378.96674779275503 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08296818049824481 = (716.9302998998724 - 776.4127024266257) / 716.9302998998724 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

```#### System.Globalization.Tests.StringSearch.IsPrefix_DifferentFirstChar(Options: (pl-PL, None, False))

```log

Description of detection logic

IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionWindowed: Marked as regression because 557.3742612611512 > 532.3676234159848.
IsChangePoint: Marked as a change because one of 10/11/2022 8:42:28 PM, 10/25/2022 7:09:53 PM, 11/1/2022 6:58:35 AM falls between 10/23/2022 1:14:54 PM and 11/1/2022 6:58:35 AM.
IsRegressionStdDev: Marked as regression because -10.102683564556107 (T) = (0 -551.3130269614716) / Math.Sqrt((147.2814912162584 / (22)) + (294.85844611173354 / (29))) is less than -2.0095752371279447 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (22) + (29) - 2, .025) and -0.08137099633791905 = (509.82782858843296 - 551.3130269614716) / 509.82782858843296 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author: performanceautofiler[bot]
Assignees: -
Labels:

area-GC-coreclr, needs-author-action, Regression
Milestone: Future

@caaavik-msft
Copy link
Contributor

Thanks @caaavik-msft. I changed the area label to GC.

For future investigations, do you have a constructions how we can run the test against different commits? I was trying to build the source on the Alpine container to run there but I couldn't build it.

The way I ran it is going to the internal version of the runtime repo in ADO, branching it at the commit I want to run the pipeline on, then editing the yaml files to make it only run the specific pipelines I care about, then you can run the dotnet-runtime-perf pipeline pointing to your branch.

I've gotten access to one of the alpine build machines we use for these benchmarks and am trying to get it built so I can run it and reproduce the regression but am running into some errors which I am trying to work out, once I have gotten it building though, will run PerfCollect to get the info we need to debug it.

@caaavik-msft
Copy link
Contributor

Just an update on this issue, after getting access to a machine with the exact same specs as the build agent, I have been unable to reproduce this regression by running the benchmarks myself, even on the latest version of the runtime branch. I guess the way that the helix job runs things must be different than the way I am running them, but I am not sure why that might be. I'm going to get Drew to take a look at this when he returns from his leave to help with this further as he has a lot more experience in this area.

@ghost ghost added the no-recent-activity label Dec 1, 2022
@ghost
Copy link

ghost commented Dec 1, 2022

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

@DrewScoggins DrewScoggins self-assigned this Dec 13, 2022
@ghost ghost removed the no-recent-activity label Dec 13, 2022
@ghost ghost added the no-recent-activity label Jan 11, 2023
@ghost
Copy link

ghost commented Jan 11, 2023

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

@tarekgh tarekgh removed the needs-author-action An issue or pull request that requires more info or actions from the author. label Jan 11, 2023
@ghost ghost removed the no-recent-activity label Jan 11, 2023
@mrsharm
Copy link
Member

mrsharm commented Jun 22, 2023

CC: @mangod9

@mangod9
Copy link
Member

mangod9 commented Aug 3, 2023

Doesnt feel like this issue is actionable, @DrewScoggins ?

@DrewScoggins
Copy link
Member

This got resolved in January.

@ghost ghost locked as resolved and limited conversation to collaborators Sep 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@DrewScoggins DrewScoggins

Labels
area-GC-coreclr
Projects
None yet
Milestone
8.0.0
Development

No branches or pull requests