External runners suddenly failing with System.Globalization.CultureNotFoundException

[ramosbugs]

Describe the bug

We've been using external GitHub Actions runners (located on Amazon EC2) without any issue for months. Beginning this afternoon, all our GitHub Actions jobs are failing with the following error returned by the runner:

System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
en-US is an invalid culture identifier.
   at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
   at System.Globalization.CultureInfo..ctor(String name)
   at GitHub.Runner.Common.HostContext.SetDefaultCulture(String name)
   at GitHub.Runner.Worker.Worker.SetCulture(AgentJobRequestMessage message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

To Reproduce

Run a GitHub Actions workflow with jobs specifying runs-on with an external runner.

Expected behavior

Actions job runs on external runner.

Runner Version and Platform

Version of your runner?

2.286.0 (also fails on 2.280.2)

OS of the machine running the runner? OSX/Windows/Linux/...

Linux ARM64

What's not working?

See stack trace above.

Job Log Output

No output for job since it fails to start. However, the workflow run shows the following annotation:

System.Globalization.CultureNotFoundException: Only the invariant culture is supported in globalization-invariant mode. See https://aka.ms/GlobalizationInvariantMode for more information. (Parameter 'name')
en-US is an invalid culture identifier.
   at System.Globalization.CultureInfo..ctor(String name, Boolean useUserOverride)
   at System.Globalization.CultureInfo..ctor(String name)
   at GitHub.Runner.Common.HostContext.SetDefaultCulture(String name)
   at GitHub.Runner.Worker.Worker.SetCulture(AgentJobRequestMessage message)
   at GitHub.Runner.Worker.Worker.RunAsync(String pipeIn, String pipeOut)
   at GitHub.Runner.Worker.Program.MainAsync(IHostContext context, String[] args)

Runner and Worker's Diagnostic Logs

Runner_20220113-014923-utc.log
Runner_20220113-014927-utc.log
Worker_20220113-014943-utc.log

[TingluoHuang]

@ramosbugs do you have a log before things broken?
also can you run printenv|sort and paste the result here on those machine?

[TingluoHuang]

you might want to check whether you have DOTNET_SYSTEM_GLOBALIZATION_INVARIANT set.
Also, to unblock, you can try export DOTNET_SYSTEM_GLOBALIZATION_PREDEFINED_CULTURES_ONLY=false and then starts the runner.
Please let me know if these help.

[ramosbugs]

@ramosbugs do you have a log before things broken?

Unfortunately I don't have any old logs since I use a fresh VM for each workflow run.

also can you run printenv|sort and paste the result here on those machine?

DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
HOME=/home/ubuntu
LANG=C.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LOGNAME=ubuntu
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.ar
c=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.l
zo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31
:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.p
bm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=0
1;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*
.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;
36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
MOTD_SHOWN=pam
PATH=/home/ubuntu/.pyenv/shims:/home/ubuntu/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
PWD=/home/ubuntu
PYENV_ROOT=/home/ubuntu/.pyenv
SHELL=/bin/bash
SHLVL=0
SSH_CLIENT=18.237.140.162 21040 22
SSH_CONNECTION=18.237.140.162 21040 10.0.0.25 22
SSH_TTY=/dev/pts/0
TERM=linux
USER=ubuntu
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
XDG_RUNTIME_DIR=/run/user/1000
XDG_SESSION_CLASS=user
XDG_SESSION_ID=1
XDG_SESSION_TYPE=tty
_=/usr/bin/printenv

Interestingly, my VM init script is setting DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1, which explains how globalization-invariant mode is getting enabled.

I was copying that from https://github.com/machulav/ec2-github-runner/pull/41/files#diff-3d2b59189eeedc2d428ddd632e97658fe310f587f7cb63b01f9b98ffc11c0197R55100, though I'm not sure the original reason why that was needed to fix ARM64 support in machulav/ec2-github-runner#41 (@machulav?).

Removing DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1 appears to fix the issue! It seems like there must have been a change deployed at GitHub today which broke compatibility with that option, but I don't see any reason I need to keep setting it :-)

[TingluoHuang]

A new runner released to service today, the runner bump it's dotnet runtime from dotnet 3.1 to dotnet 6...
Apparently there is a breaking change introduced in dotnet 6.
https://docs.microsoft.com/en-us/dotnet/core/compatibility/globalization/6.0/culture-creation-invariant-mode

I will change the runner to follow the old behavior in the new release to avoid any interruptions.

[ramosbugs]

Thank you for the quick response and helpful suggestions!!