InProcess run crashes on certain benchmarks

[filipnavara]

Running the microbenchmarks in-process with dotnet run -f netcoreapp5.0 -c Release --anyCategories coreclr --iterationCount 1 --warmupCount 0 --invocationCount 1 --unrollFactor 1 --strategy ColdStart --stopOnFirstError true -i causes unexpected failures in comparison to out-of-process runs.

Specifically, the benchmarks using [IterationCleanup] logic seem to be affected.

Example of crash:

2019-08-22T13:11:12.2769300Z // Found 4 benchmarks:
2019-08-22T13:11:12.2769780Z //   Sorting.QuickSortSpan: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [Size=512]
2019-08-22T13:11:12.2770520Z //   Sorting.BubbleSortSpan: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [Size=512]
2019-08-22T13:11:12.2771100Z //   Sorting.QuickSortArray: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [Size=512]
2019-08-22T13:11:12.2771840Z //   Sorting.BubbleSortArray: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [Size=512]
2019-08-22T13:11:12.2772000Z 
2019-08-22T13:11:12.2772100Z // **************************
2019-08-22T13:11:12.2772610Z // Benchmark: Sorting.QuickSortSpan: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0) [Size=512]
2019-08-22T13:11:12.2772780Z // *** Execute ***
2019-08-22T13:11:12.2772870Z // Launch: 1 / 1
2019-08-22T13:11:12.2772990Z // ! Failed to set up priority High for process System.Diagnostics.Process (MicroBenchmarks). Make sure you have the right permissions. Message: Permission denied
2019-08-22T13:11:12.2773090Z 
2019-08-22T13:11:12.2773180Z // Benchmark Process Environment Information:
2019-08-22T13:11:12.2773530Z // Runtime=.NET Core 5.0.0-alpha1.19415.3 (CoreCLR 5.0.19.41307, CoreFX 5.0.19.40909), X64 RyuJIT
2019-08-22T13:11:12.2773670Z // GC=Concurrent Workstation
2019-08-22T13:11:12.2774150Z // Job: Job-UPXTRJ(PowerPlanMode=00000000-0000-0000-0000-000000000000, Toolchain=InProcessEmitToolchain, InvocationCount=1000, IterationCount=1, IterationTime=250.0000 ms, MaxIterationCount=20, MinIterationCount=15, RunStrategy=ColdStart, UnrollFactor=1, WarmupCount=0)
2019-08-22T13:11:12.2774290Z 
2019-08-22T13:11:12.3437140Z WorkloadActual   1: 1000 op, 12884827.00 ns, 12.8848 us/op
2019-08-22T13:11:12.3437510Z 
2019-08-22T13:11:12.3566370Z 
2019-08-22T13:11:12.3629270Z System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
2019-08-22T13:11:12.3630200Z  ---> System.IndexOutOfRangeException: Index was outside the bounds of the array.
2019-08-22T13:11:12.3630380Z    at Span.Sorting.QuickSortSpan() in /Users/filipnavara/agent/_work/1/s/work/performance/src/benchmarks/micro/coreclr/Span/Sorting.cs:line 39
2019-08-22T13:11:12.3630510Z    at BenchmarkDotNet.Autogenerated.Runnable_352.WorkloadActionUnroll(Int64 invokeCount)
2019-08-22T13:11:12.3630630Z    at BenchmarkDotNet.Engines.Engine.MeasureGcStats(IterationData data)
2019-08-22T13:11:12.3630740Z    at BenchmarkDotNet.Engines.Engine.Run()
2019-08-22T13:11:12.3630840Z    at BenchmarkDotNet.Autogenerated.Runnable_352.Run(BenchmarkCase benchmarkCase, IHost host)
2019-08-22T13:11:12.3631180Z    --- End of inner exception stack trace ---
2019-08-22T13:11:12.3631320Z    at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
2019-08-22T13:11:12.3631450Z    at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
2019-08-22T13:11:12.3631590Z    at BenchmarkDotNet.Toolchains.InProcess.Emit.Implementation.RunnableProgram.Run(BenchmarkId benchmarkId, Assembly partitionAssembly, BenchmarkCase benchmarkCase, IHost host)
2019-08-22T13:11:12.3631700Z ExitCode != 0
2019-08-22T13:11:12.3631860Z No more Benchmark runs will be launched as NO measurements were obtained from the previous run!

/cc @adamsitnik

[filipnavara]

For context, I am doing automated runs under different runtimes and configurations. The tooling for Mono/netcore is currently quite incomplete so certain runs (Mono LLVM JIT and Mono LLVM AOT) can only be done in-process for the moment.

[adamsitnik]

I've sent a fix to BDN dotnet/BenchmarkDotNet#1231

I am doing automated runs under different runtimes and configurations

Is this blocking you?

[filipnavara]

It is one of the blockers, not a critical at the moment as I can skip over the affected tests. Thanks!