Windows arm64 Tests failing on DDARM64-132

[jkotas]

Example of the log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-67774-merge-bbcfa5ff54a3482cb8/profiler.transitions/1/console.cc2581a0.log?sv=2019-07-07&se=2022-04-30T07%3A34%3A10Z&sr=c&sp=rl&sig=%2FEI53WaNPE73mV2L3H0ORI4rqfH87tVi0DfxRVx97fw%3D

The log does not have any details about the failure.

Console log: 'profiler.rejit' from job 1bec3f43-7d47-4f00-9299-6cefc1cd6299 workitem 365fa792-6588-4765-bb67-642543d541b7 (windows.10.arm64v8.open) executed on machine DDARM64-132

Failed in #67817, #67774 and many other jobs. It is a different test each time. The only common pattern is DDARM64-132 machine name.

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[jkotas]

@dotnet/dnceng Could you please remove DDARM64-132 from the pool and check its health?

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Example of the log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-67774-merge-bbcfa5ff54a3482cb8/profiler.transitions/1/console.cc2581a0.log?sv=2019-07-07&se=2022-04-30T07%3A34%3A10Z&sr=c&sp=rl&sig=%2FEI53WaNPE73mV2L3H0ORI4rqfH87tVi0DfxRVx97fw%3D

The log does not have any details about the failure.

Console log: 'profiler.rejit' from job 1bec3f43-7d47-4f00-9299-6cefc1cd6299 workitem 365fa792-6588-4765-bb67-642543d541b7 (windows.10.arm64v8.open) executed on machine DDARM64-132

Failed in #67817, #67774 and many other jobs. It is a different test each time. The only common pattern is DDARM64-132 machine name.

Author:	jkotas
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[danmoseley]

Cc @dotnet/dncenghot

[jakobbotsch]

Similar looking failure in https://dev.azure.com/dnceng/public/_build/results?buildId=1709921&view=ms.vss-test-web.build-test-results-tab&runId=46511082&resultId=108903&paneView=debug on a different machine: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-67238-merge-9e174ab97ff8449ab2/JIT.HardwareIntrinsics.X86.Avx1/1/console.310ed48e.log?sv=2019-07-07&se=2022-04-30T23%3A51%3A23Z&sr=c&sp=rl&sig=NCIi6AFYy6maoBk1tHEkmJxAq%2FznXMWAbfiHy4IpcIA%3D

[dkurepa]

Hello, I'll take the machine offline, you can follow the issue on our side at dotnet/arcade#9016

[MattGal]

for -046 (more recent log) I filed dotnet/arcade#9024 (@agocke FYI)

[jakobbotsch]

Same problem now seems to be occurring on DDARM64-125, e.g. run 1, run 2.

[lpatalas]

I took that machine offline and created dotnet/arcade#9088 to follow up

[agocke]

Dup against dotnet/arcade#9088

[markwilkie]

I wonder if it makes sense to have a "tracker" known issue to see how often we get hit with machines going offline over time? https://github.com/dotnet/arcade/blob/main/Documentation/Projects/Build%20Analysis/KnownIssues.md

cc/ @ulisesh @AlitzelMendez

[ulisesh]

Unfortunately, the output generated by machines in a bad state is empty so we can't generate a known issue for it. I'm not sure how @ilyas1974 is tracking how often machines have to be reimaged