[Perf] Windows/x64: 59 Regressions due to PGO update #87179

performanceautofiler · 2023-06-06T08:31:32Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ICollection - Duration of single invocation	13.10 μs	17.22 μs	1.32	0.04	False
List - Duration of single invocation	12.83 μs	17.39 μs	1.35	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.221965465701697 > 13.807469303835454.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -12.59767594136789 (T) = (0 -17214.049295440363) / Math.Sqrt((2013151.2654274935 / (25)) + (98.19219233095977 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.2621305908275654 = (13638.881285773525 - 17214.049295440363) / 13638.881285773525 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.ContainsTrue<Int32>.List(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.388277422049 > 13.959162170242886.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -12.904286364505177 (T) = (0 -17409.06331095206) / Math.Sqrt((2056912.7223882207 / (25)) + (11076.39780713844 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.27331935161706755 = (13672.189375621447 - 17409.06331095206) / 13672.189375621447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Remove_IntInt - Duration of single invocation	11.45 ns	16.29 ns	1.42	0.06	False
Replace_String - Duration of single invocation	38.91 ns	46.95 ns	1.21	0.03	False
TrimStart - Duration of single invocation	11.08 ns	16.17 ns	1.46	0.07	False
ToLowerInvariant - Duration of single invocation	18.06 ns	22.96 ns	1.27	0.03	False
TrimEnd - Duration of single invocation	11.57 ns	14.91 ns	1.29	0.09	False
Trim_CharArr - Duration of single invocation	16.40 ns	19.69 ns	1.20	0.10	False
TrimStart_CharArr - Duration of single invocation	12.41 ns	14.03 ns	1.13	0.05	False
ToUpperInvariant - Duration of single invocation	18.13 ns	27.33 ns	1.51	0.05	False
Trim_CharArr - Duration of single invocation	13.18 ns	16.50 ns	1.25	0.05	False
Replace_String - Duration of single invocation	35.69 ns	50.18 ns	1.41	0.03	False
Remove_IntInt - Duration of single invocation	11.93 ns	15.23 ns	1.28	0.05	False
ToLowerInvariant - Duration of single invocation	18.59 ns	23.14 ns	1.24	0.04	False
Trim_CharArr - Duration of single invocation	14.50 ns	16.73 ns	1.15	0.08	False
ToLowerInvariant - Duration of single invocation	31.45 ns	35.73 ns	1.14	0.03	False
Remove_IntInt - Duration of single invocation	11.28 ns	15.25 ns	1.35	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.286023356741666 > 12.91177764480207.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -13.550925850566289 (T) = (0 -16.143226907608938) / Math.Sqrt((1.2931853427437008 / (24)) + (0.28293719304310155 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.34729928512542557 = (11.981916034421483 - 16.143226907608938) / 11.981916034421483 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.948632398824834 > 45.132741645284355.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/17/2023 10:06:19 AM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -9.467008971783427 (T) = (0 -46.62551707133518) / Math.Sqrt((8.53770693900059 / (25)) + (0.573687182384257 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15224457929309637 = (40.46494807546849 - 46.62551707133518) / 40.46494807546849 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimStart(s: " Test")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.165391021408933 > 11.30719128424789.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -48.63622296556723 (T) = (0 -16.35964596795198) / Math.Sqrt((0.17202241041931612 / (24)) + (0.030943304456261533 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.47065447176134534 = (11.124058221751213 - 16.35964596795198) / 11.124058221751213 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.962254251870444 > 19.454100154148634.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -35.500729363171196 (T) = (0 -23.347707883502615) / Math.Sqrt((0.37532936199380573 / (24)) + (0.04925291202905812 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.29695458966437155 = (18.001947076300166 - 23.347707883502615) / 18.001947076300166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimEnd(s: "Test ")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.913651208547238 > 12.72516506002495.
IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 4/19/2023 8:38:27 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.274444649292139 (T) = (0 -15.648369454951709) / Math.Sqrt((0.14975776255986786 / (24)) + (1.2964555813686132 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3010052373317946 = (12.02790658017998 - 15.648369454951709) / 12.02790658017998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.68856913041714 > 17.4880858480232.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -19.497367065447687 (T) = (0 -19.924239585484628) / Math.Sqrt((0.31606642421014763 / (24)) + (0.12677602164963053 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20929676797304597 = (16.475889221866108 - 19.924239585484628) / 16.475889221866108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.02842604331755 > 13.737467144653055.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -7.813505745388528 (T) = (0 -14.142283264510453) / Math.Sqrt((0.7491159584776212 / (24)) + (0.11758300126126087 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1377322553843227 = (12.43023848324774 - 14.142283264510453) / 12.43023848324774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToUpperInvariant(s: "test")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.33346013334883 > 18.687162821963323.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -3.161744711592882 (T) = (0 -22.108599390788534) / Math.Sqrt((0.21718701830331613 / (24)) + (14.017759538709216 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2544395865715673 = (17.62428388537403 - 22.108599390788534) / 17.62428388537403 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.50055467177426 > 15.102117059623264.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -14.258750931932699 (T) = (0 -16.487820179957204) / Math.Sqrt((0.9421479652021646 / (24)) + (0.002043369491451209 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2077020133146914 = (13.652225464710694 - 16.487820179957204) / 13.652225464710694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.183800855262206 > 42.44772404786201.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -14.937590072600948 (T) = (0 -50.36156535756361) / Math.Sqrt((19.54562471648434 / (25)) + (0.4426933315480499 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.3748792685114141 = (36.62980925742682 - 50.36156535756361) / 36.62980925742682 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.234881995647653 > 12.548569291285613.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -7.291263627626198 (T) = (0 -14.559666237283732) / Math.Sqrt((0.11878048749863417 / (24)) + (0.8419137856409523 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.21537965450867097 = (11.979521117761033 - 14.559666237283732) / 11.979521117761033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.135248297997048 > 19.596599138145027.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -34.88193701168203 (T) = (0 -23.20114159891524) / Math.Sqrt((0.31180615878346046 / (24)) + (0.06536239609790322 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2897602741505068 = (17.98872400081987 - 23.20114159891524) / 17.98872400081987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.733750673324913 > 15.749647040257416.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -17.592966750486983 (T) = (0 -16.873858487586865) / Math.Sqrt((0.32950062302134553 / (24)) + (0.0043234044063571436 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1427040498573515 = (14.766604257413194 - 16.873858487586865) / 14.766604257413194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.72884961714696 > 34.557854647388986.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -11.669014811508955 (T) = (0 -35.66501082277905) / Math.Sqrt((2.398002263904492 / (24)) + (0.061138336249441684 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.12088405020055965 = (31.818644235679432 - 35.66501082277905) / 31.818644235679432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.253034117136021 > 12.635653580588492.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.303553574006205 (T) = (0 -15.15861929406102) / Math.Sqrt((0.3863057974234546 / (24)) + (1.1840634869368947 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.30849969383118686 = (11.584732778712194 - 15.15861929406102) / 11.584732778712194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Array_ComparerStruct - Duration of single invocation	240.68 μs	282.80 μs	1.18	0.15	False
LinqOrderByExtension - Duration of single invocation	213.08 μs	253.26 μs	1.19	0.07	False
LinqQuery - Duration of single invocation	214.01 μs	249.26 μs	1.16	0.05	False
Array_Comparison - Duration of single invocation	233.63 μs	281.74 μs	1.21	0.13	False
List - Duration of single invocation	230.80 μs	269.02 μs	1.17	0.14	False
Array - Duration of single invocation	224.32 μs	271.32 μs	1.21	0.17	False
Array_ComparerClass - Duration of single invocation	235.83 μs	284.99 μs	1.21	0.17	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<String>.Array_ComparerStruct(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 282.80378666666655 > 262.63437830000004.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -27.5218931740511 (T) = (0 -285219.21895238094) / Math.Sqrt((35488010.78479053 / (24)) + (3308863.3287484846 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15579322644767585 = (246773.5685119047 - 285219.21895238094) / 246773.5685119047 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.LinqOrderByExtension(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 253.26065866666667 > 234.69183755000003.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -21.665681347818406 (T) = (0 -250614.69681946628) / Math.Sqrt((49150568.40816188 / (24)) + (2214237.6471708487 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15330552502037817 = (217301.21930616614 - 250614.69681946628) / 217301.21930616614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.LinqQuery(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 249.26142933333333 > 232.7877699.
IsChangePoint: Marked as a change because one of 4/25/2023 9:59:52 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -18.766436696394923 (T) = (0 -251000.35188278387) / Math.Sqrt((58408438.660455175 / (24)) + (5685731.849366357 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15567254107877324 = (217189.85522359583 - 251000.35188278387) / 217189.85522359583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array_Comparison(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 281.73827066666666 > 253.89817395000003.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -34.11144897021536 (T) = (0 -280901.71866666665) / Math.Sqrt((32505894.62266493 / (24)) + (1933736.6810228333 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18352986804596233 = (237342.31492648553 - 280901.71866666665) / 237342.31492648553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.List(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.024396 > 246.05755494230766.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 8:24:48 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -32.467794440344896 (T) = (0 -269152.1215238095) / Math.Sqrt((34983085.52278327 / (24)) + (272014.575374633 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17311381076405472 = (229433.93816880343 - 269152.1215238095) / 229433.93816880343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.31517199999996 > 245.20233394999997.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 3:57:43 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -29.192584994534666 (T) = (0 -269557.5234034537) / Math.Sqrt((39448744.262925684 / (24)) + (1897646.3343733747 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17627491207936258 = (229162.0102029914 - 269557.5234034537) / 229162.0102029914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array_ComparerClass(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 284.9897853333334 > 253.62559705000004.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -32.70111773399092 (T) = (0 -283583.14293877553) / Math.Sqrt((33488824.27026515 / (24)) + (3985248.79086351 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1927945385846994 = (237746.848904304 - 283583.14293877553) / 237746.848904304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

The text was updated successfully, but these errors were encountered:

performanceautofiler · 2023-06-06T08:31:35Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Int32

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ToString - Duration of single invocation	9.97 ns	12.19 ns	1.22	0.06	False
ToString - Duration of single invocation	12.18 ns	15.37 ns	1.26	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Int32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Int32.ToString(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 12.194173766352954 > 10.42855260869867.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 8:05:26 PM, 4/25/2023 6:46:56 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -24.017672147410114 (T) = (0 -12.13573175075449) / Math.Sqrt((0.16339912761450404 / (24)) + (0.0051315193111998525 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20753412309473165 = (10.05001143955452 - 12.13573175075449) / 10.05001143955452 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_Int32.ToString(value: 2147483647)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.366940449177877 > 13.379995387957416.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 4:22:12 PM, 4/9/2023 11:04:55 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -58.56776692202065 (T) = (0 -15.382709670426832) / Math.Sqrt((0.06363330067880928 / (24)) + (0.00037871781104181795 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.24694206139038322 = (12.33634676921122 - 15.382709670426832) / 12.33634676921122 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ForEach_Array - Duration of single invocation	3.55 ns	4.85 ns	1.36	0.17	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark*'

Payloads

Baseline
Compare

Histogram

Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.845897894674565 > 3.72914748446663.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 4/9/2023 11:04:55 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -39.80117946438802 (T) = (0 -4.8432498896594165) / Math.Sqrt((0.024104688773064764 / (25)) + (8.835308936469366E-06 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.34290009337618577 = (3.606560095980781 - 4.8432498896594165) / 3.606560095980781 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringEquality

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Compare_DifferentFirstChar - Duration of single invocation	29.84 ns	35.78 ns	1.20	0.07	False
Compare_DifferentFirstChar - Duration of single invocation	30.59 ns	35.03 ns	1.14	0.06	False
Compare_DifferentFirstChar - Duration of single invocation	31.05 ns	34.90 ns	1.12	0.06	False
Compare_DifferentFirstChar - Duration of single invocation	30.57 ns	34.85 ns	1.14	0.07	False
Compare_DifferentFirstChar - Duration of single invocation	29.24 ns	35.40 ns	1.21	0.05	False
Compare_Same_Upper - Duration of single invocation	1.06 μs	1.22 μs	1.15	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringEquality*'

Payloads

Baseline
Compare

Histogram

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, None))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.777936054757255 > 31.8225536374239.
IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 3/22/2023 9:02:35 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -25.9944892803942 (T) = (0 -35.12692066255957) / Math.Sqrt((0.825578600028122 / (25)) + (0.10713989745344599 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.19429200836063254 = (29.412338370058425 - 35.12692066255957) / 29.412338370058425 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, None))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.027398082722215 > 32.24310684667808.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -24.9215516446299 (T) = (0 -35.1391844099929) / Math.Sqrt((1.0049650611505123 / (25)) + (0.05394692354217035 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.18375393369134166 = (29.684534437336264 - 35.1391844099929) / 29.684534437336264 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, IgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.89905207112815 > 32.55239759268578.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/21/2023 4:09:07 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -18.599874696735537 (T) = (0 -34.93385628453139) / Math.Sqrt((1.461578163747821 / (25)) + (0.008834045992968482 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.14958305788440154 = (30.388283860777136 - 34.93385628453139) / 30.388283860777136 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 34.84770300810809 > 32.36832380707033.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/25/2023 10:38:38 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -16.282833598294843 (T) = (0 -35.02266932753604) / Math.Sqrt((1.7929051406585863 / (25)) + (0.06097231859316187 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15187479328425865 = (30.404927281791103 - 35.02266932753604) / 30.404927281791103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (pl-PL, None))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.399874882569996 > 31.431500723585202.
IsChangePoint: Marked as a change because one of 3/16/2023 8:14:39 PM, 3/17/2023 10:06:19 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -21.82108344371345 (T) = (0 -35.185459162233364) / Math.Sqrt((1.1945592332826311 / (24)) + (0.11179093079518872 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18908246512094118 = (29.59042807737869 - 35.185459162233364) / 29.59042807737869 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.216070719470538 > 1.2139448913586368.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -6.711893969776785 (T) = (0 -1289.006398188405) / Math.Sqrt((11506.51789963786 / (24)) + (1765.8431492360558 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1639411709935116 = (1107.4497838135073 - 1289.006398188405) / 1107.4497838135073 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Reader

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ReadReturnBytes - Duration of single invocation	6.62 μs	9.25 μs	1.40	0.52	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Reader*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: DeepTree)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 9.247130770735524 > 7.127696773203326.
IsChangePoint: Marked as a change because one of 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.994528545319696 (T) = (0 -8866.647675742965) / Math.Sqrt((168433.7183788124 / (24)) + (270592.0478579049 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2768086567869769 = (6944.3824872361265 - 8866.647675742965) / 6944.3824872361265 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_UInt32

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ToString - Duration of single invocation	11.36 ns	13.81 ns	1.22	0.07	False
ToString - Duration of single invocation	11.82 ns	16.83 ns	1.42	0.10	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_UInt32*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_UInt32.ToString(value: 12345)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 13.805227304194526 > 11.907051987488424.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -13.15791615034516 (T) = (0 -13.783123848207003) / Math.Sqrt((1.0947929629492876 / (24)) + (0.0017769354453144226 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2570052281164397 = (10.965048943241337 - 13.783123848207003) / 10.965048943241337 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_UInt32.ToString(value: 4294967295)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.828897958496626 > 12.518416905356082.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 3/31/2023 8:05:26 PM, 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -19.308874539302327 (T) = (0 -17.241301055312608) / Math.Sqrt((0.0725864063081279 / (24)) + (0.5436750537920058 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.46655259782795455 = (11.756346878283075 - 17.241301055312608) / 11.756346878283075 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Linq.Tests.Perf_OrderBy

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
OrderByString - Duration of single invocation	229.73 μs	267.57 μs	1.16	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Linq.Tests.Perf_OrderBy*'

Payloads

Baseline
Compare

Histogram

System.Linq.Tests.Perf_OrderBy.OrderByString(NumberOfPeople: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 267.56547366828084 > 249.77837025201143.
IsChangePoint: Marked as a change because one of 3/10/2023 12:53:36 AM, 4/25/2023 6:46:56 PM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -15.730206654781929 (T) = (0 -260848.48772384392) / Math.Sqrt((33086084.44142996 / (24)) + (10881201.970933482 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.11517210717257442 = (233908.7267751015 - 260848.48772384392) / 233908.7267751015 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CtorFromCollectionNonGeneric<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedList - Duration of single invocation	239.24 μs	261.47 μs	1.09	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CtorFromCollectionNonGeneric&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CtorFromCollectionNonGeneric<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 261.4689342948718 > 252.28922403846158.
IsChangePoint: Marked as a change because one of 5/10/2023 12:32:34 AM, 5/13/2023 6:35:24 AM, 5/23/2023 8:43:31 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -9.460652268457213 (T) = (0 -266712.15221685596) / Math.Sqrt((22962302.274779834 / (24)) + (72462445.52526964 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.13543957513952068 = (234897.70662968382 - 266712.15221685596) / 234897.70662968382 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Perf_Convert

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ToHexString - Duration of single invocation	35.61 ns	41.11 ns	1.15	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Perf_Convert*'

Payloads

Baseline
Compare

Histogram

System.Perf_Convert.ToHexString

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 41.108947363361395 > 37.8864965239414.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -67.86532446338427 (T) = (0 -41.16012128464307) / Math.Sqrt((0.13543207802879936 / (25)) + (0.0049278608384559055 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.14810644868855471 = (35.8504399410516 - 41.16012128464307) / 35.8504399410516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Document.Tests.Perf_EnumerateArray

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EnumerateUsingIndexer - Duration of single invocation	1.35 μs	1.43 μs	1.06	0.03	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Document.Tests.Perf_EnumerateArray*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 1.4280076392795011 > 1.4129753762867647.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/25/2023 12:18:49 AM, 4/6/2023 1:55:21 PM, 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 3:46:39 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -12.634445529896837 (T) = (0 -1428.070834754199) / Math.Sqrt((779.3334737013729 / (24)) + (0.19981601918857617 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.05311653235372219 = (1356.0425564324316 - 1428.070834754199) / 1356.0425564324316 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-06-06T08:31:37Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Common

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Email_IsMatch - Duration of single invocation	104.09 ns	120.91 ns	1.16	0.12	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Common*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 120.9061087231152 > 108.18101259099014.
IsChangePoint: Marked as a change because one of 3/25/2023 12:18:49 AM, 3/28/2023 8:13:28 AM, 4/19/2023 8:38:27 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -24.2530373590888 (T) = (0 -121.52991415282663) / Math.Sqrt((4.961668062545958 / (24)) + (2.3169952998706913 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17142852719271842 = (103.74505258470033 - 121.52991415282663) / 103.74505258470033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
DeserializeFromUtf8Bytes - Duration of single invocation	218.31 μs	296.62 μs	1.36	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Serialization.Tests.ReadJson&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Serialization.Tests.ReadJson<MyEventsListerViewModel>.DeserializeFromUtf8Bytes(Mode: Reflection)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 296.6232204861111 > 235.87053683035717.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -6.143205658751805 (T) = (0 -280767.39393674134) / Math.Sqrt((21901424.551616486 / (24)) + (549701872.2299329 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.24226181005016648 = (226013.0607455469 - 280767.39393674134) / 226013.0607455469 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsKeyTrue<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedList - Duration of single invocation	178.83 μs	214.91 μs	1.20	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsKeyTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsKeyTrue<String, String>.SortedList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 214.90989155251143 > 196.96154535520128.
IsChangePoint: Marked as a change because one of 4/4/2023 4:23:54 PM, 5/13/2023 1:49:08 AM, 5/23/2023 8:43:31 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -25.043793111382673 (T) = (0 -216468.02785893442) / Math.Sqrt((22470899.648432195 / (24)) + (7303704.540361785 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.19443031901101895 = (181231.18980952245 - 216468.02785893442) / 181231.18980952245 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrueComparer<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedSet - Duration of single invocation	217.29 μs	258.24 μs	1.19	0.08	False
ImmutableSortedSet - Duration of single invocation	218.41 μs	258.39 μs	1.18	0.09	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrueComparer&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrueComparer<String>.SortedSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.24418715846997 > 232.50905164930558.
IsChangePoint: Marked as a change because one of 4/4/2023 4:23:54 PM, 5/13/2023 2:42:33 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -48.366713596759695 (T) = (0 -258794.92201104906) / Math.Sqrt((14381339.742568674 / (25)) + (686664.9440006126 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.18113892478834256 = (219106.25124595274 - 258794.92201104906) / 219106.25124595274 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.ContainsTrueComparer<String>.ImmutableSortedSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 258.3912773224044 > 241.90896277573538.
IsChangePoint: Marked as a change because one of 3/10/2023 12:53:36 AM, 4/4/2023 4:23:54 PM, 4/9/2023 11:04:55 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -27.511509267728616 (T) = (0 -258727.78465484554) / Math.Sqrt((36405436.53794992 / (25)) + (1207056.8860890076 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.1570081518638032 = (223617.94447002443 - 258727.78465484554) / 223617.94447002443 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueTrue<String, String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedDictionary - Duration of single invocation	198.46 μs	241.93 μs	1.22	0.09	False
SortedList - Duration of single invocation	177.71 μs	215.00 μs	1.21	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueTrue&lt;String, String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.TryGetValueTrue<String, String>.SortedDictionary(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 241.9265576923077 > 213.25930154896741.
IsChangePoint: Marked as a change because one of 5/8/2023 9:08:59 PM, 5/13/2023 1:49:08 AM, 5/21/2023 1:46:51 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.947397728207946 (T) = (0 -236757.29316406048) / Math.Sqrt((15501942.451120304 / (24)) + (117139536.39338888 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1870148464190742 = (199456.0505104867 - 236757.29316406048) / 199456.0505104867 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.TryGetValueTrue<String, String>.SortedList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 215.00434360730597 > 195.3036030159884.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 3:32:16 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -33.163794837002506 (T) = (0 -214951.60721137514) / Math.Sqrt((17237805.88271307 / (24)) + (1721084.4632037359 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1785379970854021 = (182388.35552435633 - 214951.60721137514) / 182388.35552435633 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndRemove<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedList - Duration of single invocation	443.62 μs	516.32 μs	1.16	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndRemove&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.CreateAddAndRemove<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 516.3165477667494 > 493.0252450284091.
IsChangePoint: Marked as a change because one of 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -18.77546180360394 (T) = (0 -516995.3329417478) / Math.Sqrt((159313991.32457975 / (24)) + (26255949.173059206 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.13257320563818242 = (456478.51314867655 - 516995.3329417478) / 456478.51314867655 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedList - Duration of single invocation	178.61 μs	215.41 μs	1.21	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.IndexerSet<String>.SortedList(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 215.40935502283102 > 194.33377322036304.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 5/13/2023 1:49:08 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -38.14898230442877 (T) = (0 -214752.4881846482) / Math.Sqrt((16151506.236441173 / (25)) + (491334.4019260547 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.17694270276452084 = (182466.3916775354 - 214752.4881846482) / 182466.3916775354 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SortedSet - Duration of single invocation	175.73 μs	216.13 μs	1.23	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<String>.SortedSet(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 216.13018835616438 > 191.13948797686692.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -18.20021333022759 (T) = (0 -217710.74406468167) / Math.Sqrt((50278012.85131415 / (24)) + (13664011.670386543 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20217449963520906 = (181097.4564264542 - 217710.74406468167) / 181097.4564264542 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
SystemTextJson_Reflection_ - Duration of single invocation	229.98 μs	247.97 μs	1.08	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'MicroBenchmarks.Serializers.Json_FromString&lt;MyEventsListerViewModel&gt;*'

Payloads

Baseline
Compare

Histogram

MicroBenchmarks.Serializers.Json_FromString<MyEventsListerViewModel>.SystemTextJson_Reflection_

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 247.9710018382353 > 242.90568001383429.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -3.023498789448895 (T) = (0 -288314.05399125116) / Math.Sqrt((69841315.17357163 / (25)) + (2421190311.3286247 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.24350214234159073 = (231856.49961835862 - 288314.05399125116) / 231856.49961835862 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Enum

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ToString_Format_Flags_Large - Duration of single invocation	16.11 ns	23.19 ns	1.44	0.12	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Enum*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.189677808061752 > 17.991915259990463.
IsChangePoint: Marked as a change because one of 4/11/2023 3:40:00 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -17.5478289766965 (T) = (0 -22.07517190443852) / Math.Sqrt((0.3551127890941325 / (24)) + (0.5883041066815936 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3331750046348609 = (16.558345174259113 - 22.07517190443852) / 16.558345174259113 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Json.Tests.Perf_Get

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
GetDecimal - Duration of single invocation	3.61 μs	4.29 μs	1.19	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Json.Tests.Perf_Get*'

Payloads

Baseline
Compare

Histogram

System.Text.Json.Tests.Perf_Get.GetDecimal

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.287676877528575 > 4.158444303203006.
IsChangePoint: Marked as a change because one of 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -9.925445162083971 (T) = (0 -4311.1949156006995) / Math.Sqrt((74811.6585864201 / (24)) + (864.9813857869368 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15082864716180078 = (3746.1657964746223 - 4311.1949156006995) / 3746.1657964746223 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Tests.Perf_StringBuilder

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ToString_MultipleSegments - Duration of single invocation	23.87 ns	31.37 ns	1.31	0.07	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Tests.Perf_StringBuilder*'

Payloads

Baseline
Compare

Histogram

System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 31.365683172667303 > 25.52096050313922.
IsChangePoint: Marked as a change because one of 3/17/2023 10:06:19 AM, 5/10/2023 6:18:38 PM, 5/11/2023 5:02:01 PM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.67313164952705 (T) = (0 -29.365274089750752) / Math.Sqrt((0.9370196294302815 / (25)) + (2.578172603950685 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.23174736909741914 = (23.84033838957463 - 29.365274089750752) / 23.84033838957463 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

performanceautofiler · 2023-06-06T08:31:39Z

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.MathBenchmarks.Single

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Asinh - Duration of single invocation	89.01 μs	94.61 μs	1.06	0.01	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.MathBenchmarks.Single*'

Payloads

Baseline
Compare

Histogram

System.MathBenchmarks.Single.Asinh

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 94.61360943775098 > 94.44397185304918.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -32.28112493260026 (T) = (0 -94597.97863746982) / Math.Sqrt((630456.5711809898 / (24)) + (1055.9468095413195 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.05872389948211555 = (89350.94285086346 - 94597.97863746982) / 89350.94285086346 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Add_Remove_SteadyState<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ConcurrentStack - Duration of single invocation	14.60 ns	16.98 ns	1.16	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Add_Remove_SteadyState&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Tests.Add_Remove_SteadyState<String>.ConcurrentStack(Count: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.98158993663234 > 15.703474968011061.
IsChangePoint: Marked as a change because one of 3/8/2023 9:36:36 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -28.532097498797846 (T) = (0 -17.041236485949234) / Math.Sqrt((0.07120845961853098 / (24)) + (0.02820371434351972 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.16284877589279093 = (14.65473141412186 - 17.041236485949234) / 14.65473141412186 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Text.Tests.Utf8ParserTests

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
TryParseDecimal - Duration of single invocation	34.24 ns	39.61 ns	1.16	0.02	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Text.Tests.Utf8ParserTests*'

Payloads

Baseline
Compare

Histogram

System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value: 123456.789)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 39.60591479240837 > 39.12622222666717.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -7.705811960885725 (T) = (0 -39.21589997080073) / Math.Sqrt((5.469400193594025 / (24)) + (0.30444971130660026 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.11403775020011185 = (35.20158985973004 - 39.21589997080073) / 35.20158985973004 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
Count - Duration of single invocation	3.10 ms	4.35 ms	1.40	0.05	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas*'

Payloads

Baseline
Compare

Histogram

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: NonBacktracking)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 4.354508076923076 > 3.066873263653233.
IsChangePoint: Marked as a change because one of 5/23/2023 8:43:31 PM, 5/25/2023 6:27:07 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -10.63618348495044 (T) = (0 -4332207.307692307) / Math.Sqrt((170888892553.52792 / (25)) + (29159411450.345142 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.34682189849605943 = (3216614.8415985103 - 4332207.307692307) / 3216614.8415985103 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
EncodeUtf8 - Duration of single invocation	13.05 ns	15.54 ns	1.19	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

Payloads

Baseline
Compare

Histogram

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,no <escaping /> required,16)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.538433351486187 > 13.968079092640599.
IsChangePoint: Marked as a change because one of 5/1/2023 4:40:42 AM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -10.7443895509019 (T) = (0 -15.314357393532148) / Math.Sqrt((0.18450410126129144 / (25)) + (0.22707380932608648 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.16279140889120783 = (13.170339302846516 - 15.314357393532148) / 13.170339302846516 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

kunalspathak · 2023-06-06T16:22:46Z

Similar regression: dotnet/perf-autofiling-issues#18510

kunalspathak · 2023-06-06T16:23:26Z

#86698

kunalspathak · 2023-06-06T16:26:34Z

Similar issue dotnet/perf-autofiling-issues#18511

AndyAyersMS · 2023-06-07T15:27:38Z

With Dynamic PGO enabled we would not expect a static PGO update to have much impact on benchmarks. Will have to investigate.

AndyAyersMS · 2023-06-07T16:03:00Z

Using the autofiling collator, there is very little overlap between the reports. Given this is all x64 and so using identical PGO data it seems hard to understand why we'd see this kind of pattern.

Notes	Recent Score	Orig Score	x64-win-intel	x64-win-amd	Benchmark
noise	2.05	1.15		2.05 1.15	System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))
still analyzing	1.46	1.19		1.46 1.19	System.Text.Json.Tests.Perf_Get.GetDecimal
noise	1.43	1.44	1.43 1.44		System.Collections.CtorFromCollection(Int32).SortedSet(Size: 512)
noise	1.41	1.36		1.41 1.36	Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array
newly noisy #87324	1.28	1.32		1.28 1.32	System.Collections.ContainsTrue(Int32).ICollection(Size: 512)
	1.16	1.16		1.16 1.16	System.Buffers.Text.Tests.Utf8ParserTests.TryParseDecimal(value: 123456.789)
	1.13	1.21		1.13 1.21	System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")
	1.12	1.15	1.12 1.15		System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, IgnoreNonSpace, False))
	1.12	1.15	1.12 1.15		System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (en-US, None, False))
	1.12	1.15	1.12 1.15		System.Globalization.Tests.StringSearch.IsSuffix_SecondHalf(Options: (, None, False))
	1.10	1.10	1.10 1.10		Span.IndexerBench.WriteViaIndexer1(length: 1024)
	1.09	1.42		1.09 1.42	System.Tests.Perf_UInt32.ToString(value: 4294967295)
	1.09	1.41		1.09 1.41	System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")
	1.06	1.40		1.06 1.40	System.Text.Json.Tests.Perf_Reader.ReadReturnBytes(IsDataCompact: False, TestCase: DeepTree)
	1.05	1.26		1.05 1.26	System.Tests.Perf_Int32.ToString(value: 2147483647)
	1.03	1.11	1.03 1.11		System.Collections.ContainsTrue(Int32).Stack(Size: 512)
	1.02	1.20	1.06 1.27	0.99 1.14	System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreCase))
	1.02	1.36		1.02 1.36	System.Text.Json.Serialization.Tests.ReadJson(MyEventsListerViewModel).DeserializeFromUtf8Bytes(Mode: Reflection)
	1.02	1.19		1.02 1.19	System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf8(arguments: UnsafeRelaxed,no (escaping /) required,16)
	1.02	1.17	1.02 1.17		System.Reflection.Invoke.StaticMethod4_arrayNotCached_int_string_struct_class
	1.01	1.35		1.01 1.35	System.Collections.ContainsTrue(Int32).List(Size: 512)
	1.01	1.14	0.99 1.13	1.03 1.16	System.Linq.Tests.Perf_OrderBy.OrderByString(NumberOfPeople: 512)
	1.00	1.46	1.00 1.46		MicroBenchmarks.Serializers.Json_FromStream(MyEventsListerViewModel).SystemTextJson_SourceGen_
	1.00	1.34	1.00 1.34		System.Text.Json.Serialization.Tests.ReadJson(MyEventsListerViewModel).DeserializeFromString(Mode: SourceGen)
	1.00	1.06		1.00 1.06	System.Text.Json.Document.Tests.Perf_EnumerateArray.EnumerateUsingIndexer(TestCase: ArrayOfNumbers)
	1.00	1.06		1.00 1.06	System.MathBenchmarks.Single.Asinh
	0.99	1.10	0.99 1.10		System.Collections.AddGivenSize(String).HashSet(Size: 512)
	0.99	1.15		0.99 1.15	System.Perf_Convert.ToHexString
	0.99	1.21	1.01 1.23	0.98 1.20	System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, None))
	0.99	1.17		0.99 1.17	System.Collections.Sort(String).List(Size: 512)
	0.99	1.16		0.99 1.16	System.Collections.Tests.Add_Remove_SteadyState(String).ConcurrentStack(Count: 512)
	0.99	1.26	0.99 1.31	0.99 1.21	System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (pl-PL, None))
	0.99	1.13	1.00 1.17	0.98 1.09	System.Collections.CtorFromCollectionNonGeneric(String).SortedList(Size: 512)
	0.99	1.31	0.99 1.31		System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, IgnoreNonSpace))
	0.98	1.20	0.94 1.20	1.02 1.21	System.Collections.Sort(String).Array_ComparerClass(Size: 512)
	0.98	1.17	0.99 1.19	0.97 1.16	System.Collections.Sort(String).LinqQuery(Size: 512)
	0.98	1.17	0.98 1.16	0.97 1.19	System.Collections.Sort(String).LinqOrderByExtension(Size: 512)
	0.98	1.13	0.97 1.19	0.99 1.08	MicroBenchmarks.Serializers.Json_FromString(MyEventsListerViewModel).SystemTextJson_Reflection_
	0.98	1.16		0.98 1.16	System.Collections.CreateAddAndRemove(String).SortedList(Size: 512)
	0.97	1.16		0.97 1.16	System.Text.RegularExpressions.Tests.Perf_Regex_Common.Email_IsMatch(Options: Compiled)
	0.97	1.21	0.99 1.21	0.95 1.21	System.Collections.TryGetValueTrue(String, String).SortedList(Size: 512)
	0.96	1.18		0.96 1.18	System.Collections.ContainsTrueComparer(String).ImmutableSortedSet(Size: 512)
	0.96	1.18		0.96 1.18	System.Collections.Sort(String).Array_ComparerStruct(Size: 512)
	0.96	1.17	1.00 1.21	0.92 1.14	System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (en-US, None))
	0.96	1.21	0.96 1.22	0.96 1.21	System.Collections.Sort(String).Array(Size: 512)
	0.96	1.23	0.96 1.23		System.Collections.ContainsKeyTrue(String, String).ImmutableSortedDictionary(Size: 512)
	0.96	1.21	0.96 1.21	0.96 1.21	System.Collections.IndexerSet(String).SortedList(Size: 512)
	0.95	1.20	0.95 1.20		System.Text.Tests.Perf_StringBuilder.Insert_Strings
	0.95	1.19		0.95 1.19	System.Collections.ContainsTrueComparer(String).SortedSet(Size: 512)
	0.95	1.31		0.95 1.31	System.Text.Tests.Perf_StringBuilder.ToString_MultipleSegments(length: 100)
	0.94	1.16	0.99 1.21	0.90 1.12	System.Globalization.Tests.StringEquality.Compare_DifferentFirstChar(Count: 1024, Options: (, IgnoreCase))
	0.94	1.14		0.94 1.14	System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")
	0.94	1.20	0.91 1.21	0.96 1.20	System.Collections.ContainsKeyTrue(String, String).SortedList(Size: 512)
	0.93	1.22		0.93 1.22	System.Tests.Perf_Int32.ToString(value: 12345)
	0.93	1.26	0.93 1.26		System.Collections.TryGetValueTrue(String, String).ImmutableSortedDictionary(Size: 512)
	0.92	1.23		0.92 1.23	System.Collections.ContainsTrue(String).SortedSet(Size: 512)
	0.91	1.22	0.91 1.22		System.Collections.ContainsKeyTrue(String, String).SortedDictionary(Size: 512)
	0.91	1.22		0.91 1.22	System.Collections.TryGetValueTrue(String, String).SortedDictionary(Size: 512)
	0.90	1.40		0.90 1.40	System.Text.RegularExpressions.Tests.Perf_Regex_Industry_Mariomkas.Count(Pattern: "[\w]+://[^/\\s?#]+[^\\s?#]+(?:\?[^\\s#])?(?:#[^\\s])?", Options: NonBacktracking)
	0.90	1.16	0.87 1.12	0.93 1.21	System.Collections.Sort(String).Array_Comparison(Size: 512)
	0.90	1.44		0.90 1.44	System.Tests.Perf_Enum.ToString_Format_Flags_Large(value: All, format: "d")
	0.90	1.15	0.90 1.15		System.Memory.ReadOnlySpan.IndexOfString(input: "foobardzsdzs", value: "rddzs", comparisonType: InvariantCulture)
	0.89	1.18	0.98 1.24	0.81 1.13	System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])
	0.86	1.29		0.86 1.29	System.Tests.Perf_String.TrimEnd(s: "Test ")
	0.85	1.14	0.85 1.14		System.Memory.ReadOnlySpan.IndexOfString(input: "string1", value: "string2", comparisonType: InvariantCulture)
	0.84	1.27		0.84 1.27	System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")
	0.83	1.44	1.01 1.42	0.69 1.46	System.Tests.Perf_String.TrimStart(s: " Test")
	0.82	1.51		0.82 1.51	System.Tests.Perf_String.ToUpperInvariant(s: "test")
	0.81	1.24	0.81 1.24		System.Globalization.Tests.StringSearch.IsSuffix_DifferentLastChar(Options: (, None, False))
	0.80	1.18	0.80 1.18		System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAA5AAAA", value: "5", comparisonType: InvariantCulture)
	0.80	1.42		0.80 1.42	System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)
	0.79	1.35		0.79 1.35	System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)
	0.79	1.24		0.79 1.24	System.Tests.Perf_String.ToLowerInvariant(s: "TEST")
	0.77	1.22		0.77 1.22	System.Tests.Perf_UInt32.ToString(value: 12345)
	0.75	1.20		0.75 1.20	System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])
	0.74	1.28		0.74 1.28	System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)
	0.73	1.25		0.73 1.25	System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])
	0.70	1.15		0.70 1.15	System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' '])

Also most of these are already mitigated, eg

It looks like we took another Static PGO update on June 5, so perhaps it was static PGO both times?

Looking at the worst-case for intel, this one has persisted. Note the benchmark became noisier once dynamic PGO was introduced -- I had hopes that we'd see an overall reduction in noise with dynamic PGO, but am now starting to wonder if that was just wishful thinking.

The jit is still fairly sensitive to small details in the PGO data, and there is an inherent degree of randomness in live PGO data, both from nondeterminisim in the test, and deliberate nondeterminism in the collection.

So, two tasks for follow-up:

look at a benchmark that regressed / improved and see what role (if any) static PGO played. Perhaps this one:
look at a benchmark like the above which did not improve.

AndyAyersMS · 2023-06-07T19:40:10Z

Looking at the example just above, it spends ~15% of benchmark intervals in corelib (byref write barrier, memcpy ,etc)

00.89%   4.6E+05     ?        Unknown
18.36%   9.53E+06    Tier-1   [System.Text.Json]System.Text.Json.Serialization.Converters.ObjectDefaultConverter`1[System.__Canon].OnTryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&)
14.81%   7.69E+06    native   coreclr.dll
09.24%   4.8E+06     Tier-1   [System.Text.Json]System.Text.Json.Serialization.JsonConverter`1[System.__Canon].TryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&,bool&)
08.13%   4.22E+06    Tier-1   [System.Text.Json]JsonHelpers.TryParseDateTimeOffset(value class System.ReadOnlySpan`1<unsigned int8>,value class DateTimeParseData&)
07.24%   3.76E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ReadSingleSegment()
06.97%   3.62E+06    Tier-1   [System.Private.CoreLib]IndexOfAnyAsciiSearcher.IndexOfAnyVectorized(unsigned int8&,int32,value class System.Runtime.Intrinsics.Vector256`1<unsigned int8>&)
05.60%   2.91E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeNextToken(unsigned int8)
04.89%   2.54E+06    Tier-1   [System.Text.Json]Utf8JsonReader.TryGetDateTimeOffsetCore(value class System.DateTimeOffset&)
03.51%   1.82E+06    Tier-1   [System.Text.Json]Utf8JsonReader.ConsumeValue(unsigned int8)
02.70%   1.4E+06     Tier-1   [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[System.Nullable`1[System.DateTimeOffset]].ReadJsonAndSetMember(class System.Object,value class System.Text.Json.ReadStack&,value class System.Text.Json.Utf8JsonReader&)
02.35%   1.22E+06    Tier-1   [System.Text.Json]JsonHelpers.TryCreateDateTimeOffset(value class System.DateTime,value class DateTimeParseData&,value class System.DateTimeOffset&)
02.10%   1.09E+06    Tier-1   [System.Text.Json]System.Text.Json.Serialization.Metadata.JsonPropertyInfo`1[System.__Canon].ReadJsonAndSetMember(class System.Object,value class System.Text.Json.ReadStack&,value class System.Text.Json.Utf8JsonReader&)
01.56%   8.1E+05     Tier-1   [System.Text.Json]System.Text.Json.Serialization.JsonCollectionConverter`2[System.__Canon,System.__Canon].OnTryRead(value class System.Text.Json.Utf8JsonReader&,class System.Type,class System.Text.Json.JsonSerializerOptions,value class System.Text.Json.ReadStack&,!0&)
01.17%   6.1E+05     Tier-1   [System.Text.Json]System.Text.Json.Serialization.Converters.ListOfTConverter`2[System.__Canon,System.__Canon].Add(required_modifier System.Runtime.InteropServices.InAttribute !1&,value class System.Text.Json.ReadStack&)
01.10%   5.7E+05     Tier-1   [System.Private.CoreLib]Utf8Utility.TranscodeToUtf16(unsigned int8*,int32,wchar*,int32,unsigned int8*&,wchar*&)
01.04%   5.4E+05     Tier-1   [System.Text.Json]JsonReaderHelper.TranscodeHelper(value class System.ReadOnlySpan`1<unsigned int8>)

My guess is that #87076 invalidated native PGO for important parts of corelib, and this destablized perf for a time, and then #87090 produced matching PGO data and things went back to "normal."

danmoseley · 2023-06-07T19:56:47Z

Naive q, could we imagine collecting data with and without new PGO on otherwise identical bits every time PGO data updates? That would immediately show how much of any change was the PGO update.

AndyAyersMS · 2023-06-07T20:46:17Z

Locally I consistently see SortedSet as faster with the "diff" compiler (which is at 0071d92)

Method	Job	Toolchain	Size	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Gen0	Gen1	Allocated	Alloc Ratio
SortedSet	Job-VCLHWJ	\base-rel\corerun.exe	512	7.859 us	0.5589 us	0.6437 us	7.936 us	7.071 us	8.937 us	1.00	0.00	3.5734	0.4810	22.07 KB	1.00
SortedSet	Job-QWFOGA	\diff-rel\corerun.exe	512	7.316 us	0.1808 us	0.2082 us	7.280 us	7.074 us	7.779 us	0.94	0.09	3.5734	0.4810	22.07 KB	1.00

Since this benchmark is pretty noisy I think we can write this off (for now) as noise.

AndyAyersMS · 2023-06-07T20:48:33Z

As for the (implied) hypothesis that the benchmarks above all spend significant time in native code: System.Text.Json.Tests.Perf_Get.GetDecimal does not, and is still regressed, at least for x86 and amd64 (but not intel):

38.51%   1.858E+07   Tier-1   [System.Private.CoreLib]Utf8Parser.TryParseNumber(value class System.ReadOnlySpan`1<unsigned int8>,value class NumberBuffer&,int32&,value class ParseNumberOptions,bool&)
22.26%   1.074E+07   Tier-1   [System.Private.CoreLib]Number.TryNumberToDecimal(value class NumberBuffer&,value class System.Decimal&)
22.22%   1.072E+07   Tier-1   [System.Text.Json]Utf8JsonReader.TryGetDecimalCore(value class System.Decimal&,value class System.ReadOnlySpan`1<unsigned int8>)
07.73%   3.73E+06    Tier-1   [System.Private.CoreLib]Decimal+DecCalc.DecAddSub(value class DecCalc&,value class DecCalc&,bool)
07.46%   3.6E+06     Tier-1   [MicroBenchmarks]Perf_Get.GetDecimal()
00.41%   2E+05       Tier-1   [System.Text.Json]Utf8JsonReader.TryGetNumber(value class System.ReadOnlySpan`1<unsigned int8>,int32&)
00.35%   1.7E+05     native   clrjit.dll
00.35%   1.7E+05     native   coreclr.dll
00.29%   1.4E+05     Tier-1   [System.Text.Json]Utf8JsonReader..ctor(value class System.ReadOnlySpan`1<unsigned int8>,bool,value class System.Text.Json.JsonReaderState)
00.25%   1.2E+05     native   ntoskrnl.exe
00.06%   3E+04       Tier-1   [System.Private.CoreLib]SpanHelpers.NonPackedIndexOfValueType(!!0&,!!0,int32)

Finally had time to drill in. Looking now with base f4ef5c8 and diff 0071d92.

I can repro a small regression locally:

Method	Job	Toolchain	Mean	Error	StdDev	Median	Min	Max	Ratio	RatioSD	Allocated	Alloc Ratio
GetDecimal	Job-TEKZDW	\base-rel\corerun.exe	3.681 us	0.0491 us	0.0459 us	3.698 us	3.606 us	3.735 us	1.00	0.00	-	NA
GetDecimal	Job-NKOPKC	\diff-rel\corerun.exe	3.878 us	0.0607 us	0.0568 us	3.895 us	3.748 us	3.947 us	1.05	0.02	-	NA

There are a few surprises:

first, static PGO data can impact benchmark methods. I have yet to confirm the exact sequencing, but it is something like the following: a root method is being rejitted at Tier1, and one of its callees has not yet been rejitted with instrumentation. If this is an R2R method we then look for static PGO data for the method
second, for some reason in this case, the static PGO data does not match and is rejected, so we then end up with no PGO data for the method. This seems odd because the regression here happened after new PGO data was propagated, and it presumably is already stale? I need to dig in and find out why.

For the stale PGO, looks like the optimization repo is running ~2 week old builds? EG the update in #86698 on June 2nd brought in data from the May 23 build 20230523.4, and that build used the runtime bits from dotnet-sdk-8.0.100-preview.5.23273.15 (also May 23).

So the recent PGO updates are:

02-Jun 20230523.4 8.0.100-preview.5.23273.15
06-Jun 20230603.6 8.0.100-preview.6.23302.6
13-Jun 20230604.4 8.0.100-preview.6.23303.5
15-Jun 20230614.6 8.0.100-preview.6.23313.16
16-Jun 20230615.5 8.0.100-preview.6.23314.19

The diff sync point is 02-Jun so PGO data should have been compatible once more by 06-Jun.

Thus if the regression was related to static PGO we'd expect it to be fixed by now. But that does not seem to be the case.

Looking at profiles it appears there is significant time spent in Memmove that does not show up in the baseline run. But I am not sure this makes sense as the apparent caller is TryParseNumber and the codegen for that method in base/diff is quite similar (save for some block ordering diffs caused by the stale PGO). Perhaps the call comes from a callee that has tail called into Memmove so will check for that next.

07.10%   3.84E+06    Tier-1   [System.Private.CoreLib]Buffer.Memmove(unsigned int8&,unsigned int8&,unsigned int)

Poked at this for quite a while and still have no idea why the base/diff profiles diverge. Did a second profile run with similar results; both show the diff run is invoking Memmove while the base run doesn't seem to be doing this. This is with --apples mode in BDN so the two sets of runs should be doing the same amount of work. Memmove tails calls to something but don't see unexpected samples piling up anywhere else or attributed to nothing.

Here's what PerfView shows for the first BDN interval of each run

BASE -- 236.986ms

DIFF -- 260.216ms

and PerfView's regression report

(not sure what it does when there is a novel entry on one side, perhaps it attributes it to the common parent).

At any rate the only difference in codegen for TryParseNumber is layout, and the layout difference comes from static PGO, so perhaps that is the culprit.

ghost · 2023-06-08T16:27:16Z

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

Issue Details

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsTrue<Int32>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio	Baseline ETL	Compare ETL
ICollection - Duration of single invocation	13.10 μs	17.22 μs	1.32	0.04	False
List - Duration of single invocation	12.83 μs	17.39 μs	1.35	0.08	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsTrue&lt;Int32&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.ContainsTrue<Int32>.ICollection(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.221965465701697 > 13.807469303835454.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -12.59767594136789 (T) = (0 -17214.049295440363) / Math.Sqrt((2013151.2654274935 / (25)) + (98.19219233095977 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.2621305908275654 = (13638.881285773525 - 17214.049295440363) / 13638.881285773525 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.ContainsTrue<Int32>.List(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 17.388277422049 > 13.959162170242886.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -12.904286364505177 (T) = (0 -17409.06331095206) / Math.Sqrt((2056912.7223882207 / (25)) + (11076.39780713844 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.27331935161706755 = (13672.189375621447 - 17409.06331095206) / 13672.189375621447 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Remove_IntInt - Duration of single invocation	11.45 ns	16.29 ns	1.42	0.06	False
Replace_String - Duration of single invocation	38.91 ns	46.95 ns	1.21	0.03	False
TrimStart - Duration of single invocation	11.08 ns	16.17 ns	1.46	0.07	False
ToLowerInvariant - Duration of single invocation	18.06 ns	22.96 ns	1.27	0.03	False
TrimEnd - Duration of single invocation	11.57 ns	14.91 ns	1.29	0.09	False
Trim_CharArr - Duration of single invocation	16.40 ns	19.69 ns	1.20	0.10	False
TrimStart_CharArr - Duration of single invocation	12.41 ns	14.03 ns	1.13	0.05	False
ToUpperInvariant - Duration of single invocation	18.13 ns	27.33 ns	1.51	0.05	False
Trim_CharArr - Duration of single invocation	13.18 ns	16.50 ns	1.25	0.05	False
Replace_String - Duration of single invocation	35.69 ns	50.18 ns	1.41	0.03	False
Remove_IntInt - Duration of single invocation	11.93 ns	15.23 ns	1.28	0.05	False
ToLowerInvariant - Duration of single invocation	18.59 ns	23.14 ns	1.24	0.04	False
Trim_CharArr - Duration of single invocation	14.50 ns	16.73 ns	1.15	0.08	False
ToLowerInvariant - Duration of single invocation	31.45 ns	35.73 ns	1.14	0.03	False
Remove_IntInt - Duration of single invocation	11.28 ns	15.25 ns	1.35	0.06	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'

Payloads

Baseline
Compare

Histogram

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.286023356741666 > 12.91177764480207.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -13.550925850566289 (T) = (0 -16.143226907608938) / Math.Sqrt((1.2931853427437008 / (24)) + (0.28293719304310155 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.34729928512542557 = (11.981916034421483 - 16.143226907608938) / 11.981916034421483 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 46.948632398824834 > 45.132741645284355.
IsChangePoint: Marked as a change because one of 3/14/2023 8:42:09 PM, 3/17/2023 10:06:19 AM, 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -9.467008971783427 (T) = (0 -46.62551707133518) / Math.Sqrt((8.53770693900059 / (25)) + (0.573687182384257 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.15224457929309637 = (40.46494807546849 - 46.62551707133518) / 40.46494807546849 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimStart(s: " Test")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.165391021408933 > 11.30719128424789.
IsChangePoint: Marked as a change because one of 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -48.63622296556723 (T) = (0 -16.35964596795198) / Math.Sqrt((0.17202241041931612 / (24)) + (0.030943304456261533 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.47065447176134534 = (11.124058221751213 - 16.35964596795198) / 11.124058221751213 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 22.962254251870444 > 19.454100154148634.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -35.500729363171196 (T) = (0 -23.347707883502615) / Math.Sqrt((0.37532936199380573 / (24)) + (0.04925291202905812 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.29695458966437155 = (18.001947076300166 - 23.347707883502615) / 18.001947076300166 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimEnd(s: "Test ")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.913651208547238 > 12.72516506002495.
IsChangePoint: Marked as a change because one of 3/21/2023 10:36:00 PM, 4/19/2023 8:38:27 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.274444649292139 (T) = (0 -15.648369454951709) / Math.Sqrt((0.14975776255986786 / (24)) + (1.2964555813686132 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.3010052373317946 = (12.02790658017998 - 15.648369454951709) / 12.02790658017998 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 19.68856913041714 > 17.4880858480232.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -19.497367065447687 (T) = (0 -19.924239585484628) / Math.Sqrt((0.31606642421014763 / (24)) + (0.12677602164963053 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.20929676797304597 = (16.475889221866108 - 19.924239585484628) / 16.475889221866108 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 14.02842604331755 > 13.737467144653055.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 4/25/2023 6:46:56 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -7.813505745388528 (T) = (0 -14.142283264510453) / Math.Sqrt((0.7491159584776212 / (24)) + (0.11758300126126087 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1377322553843227 = (12.43023848324774 - 14.142283264510453) / 12.43023848324774 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToUpperInvariant(s: "test")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 27.33346013334883 > 18.687162821963323.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 5/22/2023 7:40:07 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -3.161744711592882 (T) = (0 -22.108599390788534) / Math.Sqrt((0.21718701830331613 / (24)) + (14.017759538709216 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2544395865715673 = (17.62428388537403 - 22.108599390788534) / 17.62428388537403 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.50055467177426 > 15.102117059623264.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/19/2023 1:39:34 AM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -14.258750931932699 (T) = (0 -16.487820179957204) / Math.Sqrt((0.9421479652021646 / (24)) + (0.002043369491451209 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2077020133146914 = (13.652225464710694 - 16.487820179957204) / 13.652225464710694 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 50.183800855262206 > 42.44772404786201.
IsChangePoint: Marked as a change because one of 3/20/2023 8:31:51 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -14.937590072600948 (T) = (0 -50.36156535756361) / Math.Sqrt((19.54562471648434 / (25)) + (0.4426933315480499 / (7))) is less than -2.0422724562973107 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (25) + (7) - 2, .025) and -0.3748792685114141 = (36.62980925742682 - 50.36156535756361) / 36.62980925742682 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.234881995647653 > 12.548569291285613.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 6:22:05 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -7.291263627626198 (T) = (0 -14.559666237283732) / Math.Sqrt((0.11878048749863417 / (24)) + (0.8419137856409523 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.21537965450867097 = (11.979521117761033 - 14.559666237283732) / 11.979521117761033 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 23.135248297997048 > 19.596599138145027.
IsChangePoint: Marked as a change because one of 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -34.88193701168203 (T) = (0 -23.20114159891524) / Math.Sqrt((0.31180615878346046 / (24)) + (0.06536239609790322 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.2897602741505068 = (17.98872400081987 - 23.20114159891524) / 17.98872400081987 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' '])

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 16.733750673324913 > 15.749647040257416.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/17/2023 6:40:18 PM, 5/19/2023 10:30:38 AM, 5/22/2023 7:40:07 AM, 5/23/2023 1:41:12 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -17.592966750486983 (T) = (0 -16.873858487586865) / Math.Sqrt((0.32950062302134553 / (24)) + (0.0043234044063571436 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1427040498573515 = (14.766604257413194 - 16.873858487586865) / 14.766604257413194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 35.72884961714696 > 34.557854647388986.
IsChangePoint: Marked as a change because one of 5/2/2023 3:18:41 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -11.669014811508955 (T) = (0 -35.66501082277905) / Math.Sqrt((2.398002263904492 / (24)) + (0.061138336249441684 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.12088405020055965 = (31.818644235679432 - 35.66501082277905) / 31.818644235679432 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 15.253034117136021 > 12.635653580588492.
IsChangePoint: Marked as a change because one of 4/9/2023 11:04:55 PM, 4/12/2023 8:39:35 PM, 5/19/2023 10:30:38 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -8.303553574006205 (T) = (0 -15.15861929406102) / Math.Sqrt((0.3863057974234546 / (24)) + (1.1840634869368947 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.30849969383118686 = (11.584732778712194 - 15.15861929406102) / 11.584732778712194 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name	Value
Architecture	x64
OS	Windows 10.0.19042
Queue	OwlWindows
Baseline	0214a047483067a6a813a34be5b6b1f757600961
Compare	2f18320e731cc3b06bc43cdb4d82dffc4f8f81fc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Sort<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector
Array_ComparerStruct - Duration of single invocation	240.68 μs	282.80 μs	1.18	0.15	False
LinqOrderByExtension - Duration of single invocation	213.08 μs	253.26 μs	1.19	0.07	False
LinqQuery - Duration of single invocation	214.01 μs	249.26 μs	1.16	0.05	False
Array_Comparison - Duration of single invocation	233.63 μs	281.74 μs	1.21	0.13	False
List - Duration of single invocation	230.80 μs	269.02 μs	1.17	0.14	False
Array - Duration of single invocation	224.32 μs	271.32 μs	1.21	0.17	False
Array_ComparerClass - Duration of single invocation	235.83 μs	284.99 μs	1.21	0.17	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

Payloads

Baseline
Compare

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Sort&lt;String&gt;*'

Payloads

Baseline
Compare

Histogram

System.Collections.Sort<String>.Array_ComparerStruct(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 282.80378666666655 > 262.63437830000004.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 7:45:57 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -27.5218931740511 (T) = (0 -285219.21895238094) / Math.Sqrt((35488010.78479053 / (24)) + (3308863.3287484846 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15579322644767585 = (246773.5685119047 - 285219.21895238094) / 246773.5685119047 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.LinqOrderByExtension(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 253.26065866666667 > 234.69183755000003.
IsChangePoint: Marked as a change because one of 4/25/2023 3:46:39 AM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -21.665681347818406 (T) = (0 -250614.69681946628) / Math.Sqrt((49150568.40816188 / (24)) + (2214237.6471708487 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15330552502037817 = (217301.21930616614 - 250614.69681946628) / 217301.21930616614 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.LinqQuery(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 249.26142933333333 > 232.7877699.
IsChangePoint: Marked as a change because one of 4/25/2023 9:59:52 PM, 5/26/2023 9:55:14 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -18.766436696394923 (T) = (0 -251000.35188278387) / Math.Sqrt((58408438.660455175 / (24)) + (5685731.849366357 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.15567254107877324 = (217189.85522359583 - 251000.35188278387) / 217189.85522359583 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array_Comparison(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 281.73827066666666 > 253.89817395000003.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -34.11144897021536 (T) = (0 -280901.71866666665) / Math.Sqrt((32505894.62266493 / (24)) + (1933736.6810228333 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.18352986804596233 = (237342.31492648553 - 280901.71866666665) / 237342.31492648553 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.List(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 269.024396 > 246.05755494230766.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/19/2023 8:24:48 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -32.467794440344896 (T) = (0 -269152.1215238095) / Math.Sqrt((34983085.52278327 / (24)) + (272014.575374633 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17311381076405472 = (229433.93816880343 - 269152.1215238095) / 229433.93816880343 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 271.31517199999996 > 245.20233394999997.
IsChangePoint: Marked as a change because one of 5/13/2023 1:49:08 AM, 5/20/2023 3:57:43 PM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -29.192584994534666 (T) = (0 -269557.5234034537) / Math.Sqrt((39448744.262925684 / (24)) + (1897646.3343733747 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.17627491207936258 = (229162.0102029914 - 269557.5234034537) / 229162.0102029914 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

System.Collections.Sort<String>.Array_ComparerClass(Size: 512)

Description of detection logic

IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsRegressionBase: Marked as regression because the compare was 5% greater than the baseline, and the value was not too small.
IsRegressionChecked: Marked as regression because the three check build points were 0.05 greater than the baseline.
IsRegressionWindowed: Marked as regression because 284.9897853333334 > 253.62559705000004.
IsChangePoint: Marked as a change because one of 5/19/2023 10:30:38 AM, 6/2/2023 6:53:36 PM, 6/5/2023 10:43:22 AM falls between 5/27/2023 6:38:16 PM and 6/5/2023 10:43:22 AM.
IsRegressionStdDev: Marked as regression because -32.70111773399092 (T) = (0 -283583.14293877553) / Math.Sqrt((33488824.27026515 / (24)) + (3985248.79086351 / (7))) is less than -2.0452296421282288 = MathNet.Numerics.Distributions.StudentT.InvCDF(0, 1, (24) + (7) - 2, .025) and -0.1927945385846994 = (237746.848904304 - 283583.14293877553) / 237746.848904304 is less than -0.05.
IsImprovementBase: Marked as not an improvement because the compare was not 5% less than the baseline, or the value was too small.
IsChangeEdgeDetector: Marked not as a regression because Edge Detector said so.

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Author:	performanceautofiler[bot]
Assignees:	AndyAyersMS
Labels:	`os-windows`, `arch-x64`, `area-CodeGen-coreclr`, `runtime-coreclr`, `PGO`, `needs-area-label`
Milestone:	8.0.0

AndyAyersMS · 2023-06-09T15:37:28Z

Naive q, could we imagine collecting data with and without new PGO on otherwise identical bits every time PGO data updates? That would immediately show how much of any change was the PGO update.

In general it would be nice if the benchmark system could anticipate impactful changes and run them in isolation. I don't know how challenging that would be to arrange as the whole system is capacity limited. As is the benchmark system seems to end up batching 10 or so commits per run, though sometimes it is less and sometimes much more.

We expect to see less impact from managed static PGO updates going forward as most benchmarks should now be using dynamic PGO data instead. PGO updates as a whole may still be impactful because they also update native PGO, and there are a number of benchmarks (like some of the ones here) that spend significant time in native code.

AndyAyersMS · 2023-06-21T15:31:03Z

System.Globalization.Tests.StringEquality.Compare_Same_Upper(Count: 1024, Options: (en-US, OrdinalIgnoreCase))

Seems to have calmed down recently

AndyAyersMS · 2023-06-21T15:40:56Z

Microsoft.Extensions.Primitives.Performance.StringValuesBenchmark.ForEach_Array

Also looks like noise

AndyAyersMS · 2023-08-01T18:35:03Z

Not seeing anything actionable here, and most tests are either noisy or are still improved over the long run.

performanceautofiler bot added arch-x64 os-windows PGO runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Jun 6, 2023

kunalspathak pinned this issue Jun 6, 2023

dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 6, 2023

kunalspathak transferred this issue from dotnet/perf-autofiling-issues Jun 6, 2023

kunalspathak changed the title ~~[Perf] Windows/x64: 59 Regressions on 6/3/2023 12:54:16 AM~~ [Perf] Windows/x64: 59 Regressions due to PGO update Jun 6, 2023

kunalspathak mentioned this issue Jun 6, 2023

[Perf] Windows/x64: 36 Regressions on 6/3/2023 12:54:16 AM dotnet/perf-autofiling-issues#18510

Closed

kunalspathak mentioned this issue Jun 6, 2023

[Perf] Windows/x64: 1 Regression on 6/3/2023 11:04:56 AM dotnet/perf-autofiling-issues#18511

Closed

AndyAyersMS self-assigned this Jun 7, 2023

AndyAyersMS added this to the 8.0.0 milestone Jun 7, 2023

ghost removed the untriaged New issue has not been triaged by the area owner label Jun 7, 2023

jeffschwMSFT added the area-CodeGen-coreclr CLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI label Jun 8, 2023

AndyAyersMS mentioned this issue Jun 9, 2023

Increased variance in some benchmark tests #87324

Closed

AndyAyersMS added the Priority:2 Work that is important, but not critical for the release label Jun 9, 2023

JulieLeeMSFT removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Jun 19, 2023

AndyAyersMS closed this as completed Aug 1, 2023

ghost locked as resolved and limited conversation to collaborators Aug 31, 2023

[Perf] Windows/x64: 59 Regressions due to PGO update #87179

[Perf] Windows/x64: 59 Regressions due to PGO update #87179

Comments

performanceautofiler bot commented Jun 6, 2023

Run Information

Regressions in System.Collections.ContainsTrue<Int32>

Repro

Payloads

Payloads

Histogram

System.Collections.ContainsTrue<Int32>.ICollection(Size: 512)

Description of detection logic

JIT Disasms

System.Collections.ContainsTrue<Int32>.List(Size: 512)

Description of detection logic

JIT Disasms

Docs

Run Information

Regressions in System.Tests.Perf_String

Repro

Payloads

Payloads

Histogram

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 10, i2: 1)

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence. This is another very nice sentence.", oldValue: "a", newValue: "")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.TrimStart(s: " Test")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TeSt")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.TrimEnd(s: "Test ")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Te st ", c: [' ', ' '])

Description of detection logic

JIT Disasms

System.Tests.Perf_String.TrimStart_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

JIT Disasms

System.Tests.Perf_String.ToUpperInvariant(s: "test")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: " Test", c: [' ', ' '])

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Replace_String(text: "This is a very nice sentence", oldValue: "nice", newValue: "bad")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 0, i2: 8)

Description of detection logic

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "TEST")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Trim_CharArr(s: "Test ", c: [' ', ' '])

Description of detection logic

JIT Disasms

System.Tests.Perf_String.ToLowerInvariant(s: "This is a much longer piece of text that might benefit more from vectorization.")

Description of detection logic

JIT Disasms

System.Tests.Perf_String.Remove_IntInt(s: "dzsdzsDDZSDZSDZSddsz", i1: 7, i2: 4)

Description of detection logic

JIT Disasms

Docs

Run Information

Regressions in System.Collections.Sort<String>

Repro

Payloads

Payloads

Histogram

System.Collections.Sort<String>.Array_ComparerStruct(Size: 512)

Description of detection logic

JIT Disasms

System.Collections.Sort<String>.LinqOrderByExtension(Size: 512)

Description of detection logic