system.net.security.tests.negotiateauthenticationkerberostest.loopback_success #73343
Comments
|
Tagging subscribers to this area: @dotnet/ncl Issue DetailsRunfo Creating Tracking Issue (data being generated)
|
noahfalk
added
the
blocking-clean-ci
|
cc: @filipnavara |
wfurt
added
area-System.Net.Security
and removed
untriaged
|
Tagging subscribers to this area: @dotnet/ncl, @vcsjones Issue DetailsThis test failed in CI rolling build 1920255.
|
|
There are multiple unrelated errors in the log:
|
|
I submitted fix for the NativeAOT problem. For the other one I will try to open a test PR to redirect KRB5_TRACE logs into xUnit output to get more information. |
|
So, there's no Kerberos trace in the logs even in the latest builds. This matches the behavior when the environment variables are not correctly passed to the second process. It's a wild guess and I am not sure why would that happen specifically in the ARM32/Linux environment. I think I can verify the theory easily. |
|
The environment variable theory didn't pan out but to my surprise there are some dumps in the artifacts of the test: https://dev.azure.com/dnceng/public/_build/results?buildId=1925111&view=ms.vss-test-web.build-test-results-tab&runId=49813094&resultId=202895&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab |
|
Uff, looks like Azure Pipelines UI just doesn't show the test output but it's actually there: The interesting part is likely |
|
how about we disable the tests on Arm32 @filipnavara? I'll try to get hands on repro machine and I can try to collect more info next week. |
|
I am fine with disabling them and continuing to investigate. I'm just not quite sure how to disable the tests on particular architecture (aside from manual check here). |
|
as it is transient, I think updating |
|
Submitted as #73445, needs a |
|
I managed to reproduce the issue with the Helix payload on Debian machine running in AWS on ARM64 machine (with ARM32 binaries). That's a good start for debugging it. |
|
The krb5 error comes from here and indeed the input date is wrong. The value is |
|
is it possible the system time is far off? Do you know if it uses |
|
The system time is correct, Kerberos.NET gets it right (printed it to console). It's wrong only inside the encrypted ASN.1 payload. I'll narrow it down. |
ghost
added
the
in-pr
|
So, it is not really corrupted. It's just that the expiration time was so far in the future that GSSAPI could not handle it. The interesting part is that it behaves differently on ARM32 but it succeeds on all other platforms, including x86. |
ghost
removed
the
in-pr
|
This new test is failing on s390x, see https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-9f276d5b7fb54d338b/System.Net.Security.Tests/3/console.08289088.log?helixlogtype=result Looking into this a bit more closely, we get the following error from GSSAPI: This in turn happens because the keytab file is in fact incorrect, specifically it contains length fields in little-endian byte order even though specifying a file version number of "2" to indicate big-endian byte order: According to https://web.mit.edu/kerberos/krb5-1.12/doc/formats/keytab_file_format.html:
So the first couple of bytes "05 02" should be followed by a big-endian length indication, but "4b 00 00 00" is clearly little-endian. I understand in the context of this test case, the keytab file was written by Kerberos.NET, and this does indeed apparently attempt to write a big-endian length field: https://github.com/dotnet/Kerberos.NET/blob/559bec87d6eb1fb8698418c3aacafbbc7c32bdec/Kerberos.NET/Crypto/KeyEntry.cs#L264 However, it does so by unconditionally inverting bytes from native endianess, which works only if the host is little-endian. On a big-endian host this is incorrect. How can we best proceed to fix this? I guess first of all this needs to be fixed in Kerberos.NET, but will the test case then pull in an updated version? Or should we just disable the test on big-endian hosts for now? FYI @steveisok @directhex |
|
Oops. PR to fix. Will go into a new nuget package once merged. dotnet/Kerberos.NET#307 |
|
Fixed in nuget v4.5.176 https://www.nuget.org/packages/Kerberos.NET/4.5.176. Uploading now, should be available for general distribution in a few minutes. |
Thanks! It looks like this is now on nuget.org, but it doesn't seem to be available yet in "dotnet-public" (https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-public/nuget/v3/index.json), where the runtime build is looking for it. But the good news is that when I force it to use the 4.5.176 package from nuget.org, all |
|
I think we may need to manual sync once again for updated Kerberos.NET package. |
|
We consume |
|
Any update on the package sync? The test is still failing on mainline on s390x. |
|
The NuGet has to be copied from the public nuget.org feed to one of those that are consumed by this repository (it is currently in dotnet-public iirc). It was a manual one off process but unfortunately I don't have the access to the resources to do it. |
|
Thanks, @filipnavara; any ideas on who we might tap for help? |
|
cc @dotnet/dnceng |
|
Chatted with @wfurt , pushed Kerberos.NET @ 4.5.178 to dotnet-public. |
ghost
added
in-pr
|
@filipnavara this is still happening in main in latest Rolling runs - I will file a new issue |
|
@karelz It must be happening on some obscure platform combination that is not part of any normal/extra PR pipelines. I need a link to the failed runs to figure out what is going on. The other issue showed some obscure build that looked like "Android" but was not the standard Android pipeline (where the tests are intentionally disabled). |
|
We have a Linux-bionic configuration that will run on android, but without needing the dalvik vm. I just recently moved it over to runtime-extra-platforms, so that's probably what you're seeing fail. |
ghost
locked as resolved and limited conversation to collaborators
This test failed in CI rolling build 1920255.
https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-d6e349c2b9554af286/System.Net.Security.Tests/3/console.eb09806e.log?%3Fhelixlogtype%3Dresult
Runfo Tracking Issue: system.net.security.tests.negotiateauthenticationkerberostest.loopback_success
Build Result Summary
The text was updated successfully, but these errors were encountered: