Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

Open
pavelsavara opened this issue Jul 24, 2023 · 9 comments
Open

[wasm][AOT] emcc : error - received SIGKILL (-9) #89402

pavelsavara opened this issue Jul 24, 2023 · 9 comments
Assignees
@radekdoulik
Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Milestone
10.0.0

Comments

@pavelsavara
Copy link
Member

pavelsavara commented Jul 24, 2023 •
edited by build-analysis bot
Loading

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": ["emcc", "SIGKILL"],
  "ExcludeConsoleLog": false
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=653707
Error message validated: [emcc SIGKILL]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 4/25/2024 7:29:32 AM UTC

Report

Build Definition Test Pull Request
966316 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
966300 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
966122 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113028
966083 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113027
965844 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112938
966049 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112933
965974 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
965981 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112997
965960 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113000
965946 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
965878 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112933
965870 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112876
965853 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112653
965670 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
965631 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112955
965592 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112876
965571 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113005
965514 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112404
965521 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
965497 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111706
965468 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
965458 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
965451 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
965447 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #106374
965441 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112986
965394 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112936
965359 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113005
965332 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112938
965210 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113003
965142 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112999
965092 dotnet/runtime WasmTestOnChrome-ST-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
963907 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112909
965106 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
965075 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #113000
965037 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112997
965018 dotnet/runtime normal-Microsoft.Extensions.Options.SourceGeneration.Unit.Tests.WorkItemExecution
965003 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112996
964926 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112938
964891 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112993
964128 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112970
964861 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
964806 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #107819
964809 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #107798
964797 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112616
963850 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111759
964719 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112986
964713 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112933
964668 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
964617 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112983
964581 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
964465 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112977
964485 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112979
964439 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112976
964441 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112936
964442 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112833
964448 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112937
964335 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112973
964307 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112938
962591 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112878
964089 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
964066 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112930
964063 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
964029 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112921
963997 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112965
963983 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112964
963958 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112787
963830 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #110818
963808 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
963793 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112951
963175 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112849
963755 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112730
963751 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112171
963705 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112947
963683 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112921
963655 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112909
963611 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112851
963494 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112815
963468 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112940
963460 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
963341 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112938
963308 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112936
963315 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112937
963307 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112935
963270 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112833
963187 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112736
963121 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112933
962370 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112050
963070 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112921
962972 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112930
962946 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
962404 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112826
962825 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
962819 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112891
962828 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #111759
962796 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112730
962744 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112796
962687 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112922
962706 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112923
962684 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution #112921
962679 dotnet/runtime WasmTestOnChrome-ST-System.Runtime.Tests.WorkItemExecution
Displaying 100 of 430 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
13 153 424
@pavelsavara pavelsavara added arch-wasm WebAssembly architecture blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' area-Codegen-AOT-mono wasm-aot-test WebAssembly AOT Test Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm labels Jul 24, 2023
@pavelsavara pavelsavara added this to the 8.0.0 milestone Jul 24, 2023
@ghost
Copy link

ghost commented Jul 24, 2023

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

Error Blob

{
  "BuildRetry": false,
  "ErrorPattern": "emcc.*SIGKILL",
  "ExcludeConsoleLog": true
}

Reproduction Steps

AOT wasm Microsoft.Extensions.Options.SourceGeneration.Unit.Tests

Log

Author: pavelsavara
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-Codegen-AOT-mono, wasm-aot-test, Known Build Error, os-browser
Milestone: 8.0.0

@pavelsavara pavelsavara mentioned this issue Jul 24, 2023
Merged
@lewing
Copy link
Member

lewing commented Jul 24, 2023

Is this it hitting the ulimit of the container?

@lewing lewing self-assigned this Jul 24, 2023
@radical
Copy link
Member

radical commented Jul 24, 2023 

[ 7455.672962] [ 2825] 1000 2825 158937 41558 684032 0 0 python3 
[ 7455.672964] [ 4160] 102 4160 67335 453 180224 0 0 rsyslogd 
[ 7455.672966] [ 4181] 0 4181 455864 16951 634880 0 0 amacoreagent 
[ 7455.672968] [ 4453] 0 4453 298556 3977 249856 0 0 auoms 
[ 7455.672969] [ 4495] 0 4495 215902 6240 204800 0 0 auomscollect 
[ 7455.672971] [ 4750] 102 4750 191707 14176 622592 0 0 mdsd 
[ 7455.672973] [ 4967] 0 4967 334103 3030 253952 0 0 azsecd 
[ 7455.672975] [ 5046] 0 5046 352600 4193 270336 0 0 azsecmond 
[ 7455.672977] [ 32457] 0 32457 317321 3577 258048 0 0 azsecd 
[ 7455.672979] [ 2226] 0 2226 376486 334278 2924544 0 0 clamscan 
[ 7455.672981] [ 22067] 0 22067 180189 824 114688 0 -998 containerd-shim 
[ 7455.672983] [ 22089] 1000 22089 1159 16 57344 0 0 helix_docker_wo 
[ 7455.672985] [ 22156] 1000 22156 1159 17 53248 0 0 execute.sh 
[ 7455.672987] [ 22212] 1000 22212 5047 87 81920 0 0 bash 
[ 7455.672989] [ 22215] 1000 22215 68517493 67031 1163264 0 0 dotnet 
[ 7455.672990] [ 23320] 1000 23320 1159 17 61440 0 0 sh 
[ 7455.672992] [ 23321] 1000 23321 34332 22818 335872 0 0 python3 
[ 7455.672994] [ 23345] 1000 23345 1420573 1386396 11264000 0 0 wasm-opt 
[ 7455.672995] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/docker/a2a4080691301bb74e3b282eab0b6f638e9d6f1227e62967b7fd46654706e554,task=wasm-opt,pid=23345,uid=1000 
[ 7455.673018] Out of memory: Killed process 23345 (wasm-opt) total-vm:5682292kB, anon-rss:5545584kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:11000kB oom_score_adj:0

@lewing
Copy link
Member

lewing commented Jul 25, 2023

lets disable these tests in aot for now

@lewing lewing mentioned this issue Aug 13, 2023
Open
3 tasks
@lewing
Copy link
Member

lewing commented Aug 13, 2023

closting as a duplicate of dotnet/dnceng#450

@lewing lewing closed this as completed Aug 13, 2023
@ghost ghost locked as resolved and limited conversation to collaborators Sep 12, 2023
@pavelsavara
Copy link
Member Author

I see this again https://helixre107v0xdcypoyl9e7f.blob.core.windows.net/dotnet-runtime-refs-pull-99833-merge-af0760c542d34f95a8/normal-System.Runtime.Tests/1/console.f849dad3.log?helixlogtype=result

@pavelsavara pavelsavara reopened this Apr 2, 2024
@lewing
Copy link
Member

lewing commented Apr 24, 2024

Analysis claims there are recent failures but the links are not updated?

@pavelsavara
Copy link
Member Author

we run most of those problematic cases only in runtime-wasm pipeline. Which needs to be triggered manually.
I updated the filter

@lewing lewing removed the blocking-clean-ci Blocking PR or rolling runs of 'runtime' or 'runtime-extra-platforms' label May 2, 2024
@dotnet dotnet unlocked this conversation May 27, 2024
@akoeplinger akoeplinger modified the milestones: 8.0.0, 9.0.0 May 27, 2024
@lewing lewing modified the milestones: 9.0.0, 10.0.0 Aug 15, 2024
@lewing
Copy link
Member

lewing commented Feb 12, 2025

all the recent failures I looked at are in wasm-opt which is known to have pretty degenerate memory behavior when linking with debug symbols. I think we should investigate what we can do here. @radekdoulik lets discuss offline

@lewing lewing assigned radekdoulik and unassigned lewing Feb 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@radekdoulik radekdoulik

Labels
arch-wasm WebAssembly architecture area-Codegen-AOT-mono Known Build Error Use this to report build issues in the .NET Helix tab os-browser Browser variant of arch-wasm wasm-aot-test WebAssembly AOT Test
Projects
None yet
Milestone
10.0.0
Development

No branches or pull requests
5 participants
@radical @lewing @pavelsavara @akoeplinger @radekdoulik