Jitted Methods Counting Test Fails When Hardware Intrinsics are Disabled

[ivdiazsa]

Tracking issue following up #93163.

The above-mentioned issue exposed a bug with the Jitted Methods Counting Test, regarding Hardware Intrinsics. When DOTNET_EnableHWIntrinsic=0, there is a much higher JIT activity, which in turn returns a much higher amount of jitted methods than the test was designed for. As per @jkotas' feedback, DOTNET_EnableHWIntrinsic=0 has a similar effect to DOTNET_ReadyToRun=0, so this test ought not be run in those cases.

For general information and context, here's how to build and run the test locally.

# Build the runtime and libraries.

./build.sh --subset clr+libs --runtimeConfiguration Checked --librariesConfiguration Release

# Build the tests and core root.

src/tests/build.sh -x64 -checked -generatelayoutonly
src/tests/build.sh -x64 -checked -test:readytorun/JittedMethodsCountingTest/JittedMethodsCountingTest.csproj

# Setup and run the tests.

cd artifacts/tests/coreclr/linux.x64.Checked
export CORE_ROOT=$(pwd)/Tests/Core_Root
./readytorun/JittedMethodsCountingTest/JittedMethodsCountingTest.sh

[ghost]

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Tracking issue following up #93163.

The above-mentioned issue exposed a bug with the Jitted Methods Counting Test, regarding Hardware Intrinsics. When DOTNET_EnableHWIntrinsic=0, there is a much higher JIT activity, which in turn returns a much higher amount of jitted methods than the test was designed for. The solution to this would be to enable two running modes:

- Normal Mode
- No Hardware Intrinsics Mode

The reason to have these two modes is that if we simply set the acceptance level high enough for the no-intrinsics version, then the normal version will lose its purpose, as we expect many less methods there.

Additionally, as part of the migration to the new test system, the Jitted Methods Counting Test ended up in some sort of weird floating-around and is always skipped in normal test runs. Will have to fix that as well here.

For general information and context, here's how to build and run the test locally.

# Build the runtime and libraries.

./build.sh --subset clr+libs --runtimeConfiguration Checked --librariesConfiguration Release

# Build the tests and core root.

src/tests/build.sh -x64 -checked -generatelayoutonly
src/tests/build.sh -x64 -checked -priority1 -test:readytorun/JittedMethodsCountingTest/JittedMethodsCountingTest.csproj

# Setup and run the tests.

cd artifacts/tests/coreclr/linux.x64.Checked
export CORE_ROOT=$(pwd)/Tests/Core_Root
./readytorun/JittedMethodsCountingTest/JittedMethodsCountingTest.sh

Author:	ivdiazsa
Assignees:	ivdiazsa
Labels:	test-bug, area-Infrastructure-coreclr
Milestone:	9.0.0

[ivdiazsa]

@trylek FYI

[jkotas]

DOTNET_EnableHWIntrinsic=0 is very similar to DOTNET_ReadyToRun=0. Both prevent some or all precompiled code from being used. I think that a solution that follows the current test design would be to check for DOTNET_EnableHWIntrinsic=0 and skip the test when it is set.

There are other env. variables that have same effect. For example, I expect the test to fail on x64 if DOTNET_SSE=0 is set. We may end up with checks for number of the env variables over time, based on what we hit in the outer loop test runs.

[ivdiazsa]

DOTNET_EnableHWIntrinsic=0 is very similar to DOTNET_ReadyToRun=0. Both prevent some or all precompiled code from being used. I think that a solution that follows the current test design would be to check for DOTNET_EnableHWIntrinsic=0 and skip the test when it is set.

There are other env. variables that have same effect. For example, I expect the test to fail on x64 if DOTNET_SSE=0 is set. We may end up with checks for number of the env variables over time, based on what we hit in the outer loop test runs.

Oh okok got it. Then let me change it so that it's disabled as well when DOTNET_EnableHWIntrinsic=0. Thanks for the observation Jan.