Sockets tests SendFileGetsCanceledByDispose and AcceptGetsCanceledByDispose - throw NotSupportedException - on AppleTV/iOS

[karelz]

Regression as of 8/4 - happens in each Rolling run and some PRs with 7, 14, or 21 failures (based on which legs ran):

Affected tests:

• SendFileGetsCanceledByDispose
  • System.Net.Sockets.Tests.SendFile_Apm
  • System.Net.Sockets.Tests.SendFile_CancellableTask
  • System.Net.Sockets.Tests.SendFile_Task
• AcceptGetsCanceledByDispose
  • System.Net.Sockets.Tests.AcceptApm
  • System.Net.Sockets.Tests.AcceptCancellableTask
  • System.Net.Sockets.Tests.AcceptEap
  • System.Net.Sockets.Tests.AcceptTask

Failure:

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendPacketsAsync(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, ReadOnlyMemory`1 preBuffer, ReadOnlyMemory`1 postBuffer, TransmitFileOptions flags, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, CancellationToken cancellationToken)
   at System.Net.Sockets.Tests.SocketHelperCancellableTask.SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SocketTestHelperBase`1[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_1.<<SendFileGetsCanceledByDispose>b__2>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_0.<<SendFileGetsCanceledByDispose>b__0>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
   at System.Net.Sockets.Tests.SendFile`1.<SendFileGetsCanceledByDispose>d__10[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Net.Sockets.Tests', //testNameSubstring
    '', //methodName
    'NotSupportedException', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Regression as of 8/4 - happens in each Rolling run and some PRs with 7, 14, or 21 failures (based on which legs ran):

Affected tests:

• SendFileGetsCanceledByDispose
  • System.Net.Sockets.Tests.SendFile_Apm
  • System.Net.Sockets.Tests.SendFile_CancellableTask
  • System.Net.Sockets.Tests.SendFile_Task
• AcceptGetsCanceledByDispose
  • System.Net.Sockets.Tests.AcceptApm
  • System.Net.Sockets.Tests.AcceptCancellableTask
  • System.Net.Sockets.Tests.AcceptEap
  • System.Net.Sockets.Tests.AcceptTask

Failure:

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.SendPacketsAsync(Socket socket, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, ReadOnlyMemory`1 preBuffer, ReadOnlyMemory`1 postBuffer, TransmitFileOptions flags, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.SendFileAsync(String fileName, CancellationToken cancellationToken)
   at System.Net.Sockets.Tests.SocketHelperCancellableTask.SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SocketTestHelperBase`1[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].SendFileAsync(Socket s, String fileName)
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_1.<<SendFileGetsCanceledByDispose>b__2>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.Net.Sockets.Tests.SendFile`1.<>c__DisplayClass10_0.<<SendFileGetsCanceledByDispose>b__0>d[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---
   at System.RetryHelper.ExecuteAsync(Func`1 test, Int32 maxAttempts, Func`2 backoffFunc, Predicate`1 retryWhen, String testName)
   at System.Net.Sockets.Tests.SendFile`1.<SendFileGetsCanceledByDispose>d__10[[System.Net.Sockets.Tests.SocketHelperCancellableTask, System.Net.Sockets.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]].MoveNext()
--- End of stack trace from previous location ---

let failedTests = (testNameSubstring : string, methodName : string, messageSubstr: string, includePR : bool, includePassedOnRerun : bool) {
cluster('engsrvprod.kusto.windows.net').database('engineeringdata').AzureDevOpsTests
    | where TestName contains testNameSubstring
    | where includePassedOnRerun or (Outcome == 'Failed')
    | extend startOfTestName = indexof_regex(TestName, @"[^.]+$")
    | extend Method = substring(TestName, startOfTestName)
    | extend Type = substring(TestName, 0, startOfTestName - 1)
    | project-away startOfTestName
    | where (methodName == '') or (Method == methodName)
    | where Message contains messageSubstr
    | distinct JobId, WorkItemId, Message, StackTrace, Method, Type, Arguments, Outcome
    | join kind=inner (cluster('engsrvprod.kusto.windows.net').database('engineeringdata').Jobs
        | where ((Branch == 'refs/heads/main') or (Branch == 'refs/heads/master') or (includePR and (Source startswith "pr/")))
        | where Type startswith "test/functional/cli/"
            and not(Properties contains "runtime-staging")
        | where Branch <> 'refs/pull/73374/merge'
        | summarize arg_max(Finished, Properties, Type, Branch, Source, Started, QueueName) by JobId
        | project-rename JobType = Type) on JobId
    | extend PropertiesJson = parse_json(Properties)
    | extend OS = replace_regex(tostring(PropertiesJson.operatingSystem), @'\((.*)\).*|([^\(].*)', @'\1\2')
    | extend Runtime = iif(PropertiesJson.runtimeFlavor == "mono", "Mono", iif(PropertiesJson.DefinitionName contains "coreclr", "CoreCLR", ""))
    | extend TargetBranch = extractjson("$.['System.PullRequest.TargetBranch']", Properties)
    | extend Architecture = PropertiesJson.architecture
    | extend Scenario = iif(isempty(PropertiesJson.scenario), "--", PropertiesJson.scenario)
    //| extend DefinitionName = PropertiesJson.DefinitionName
    | project-away PropertiesJson
};
failedTests(
    'System.Net.Sockets.Tests', //testNameSubstring
    '', //methodName
    'NotSupportedException', //messageSubstr
    true,  //includePR
    true); //includePassedOnRerun

Author:	karelz
Assignees:	-
Labels:	area-System.Net.Sockets, blocking-clean-ci, os-ios, os-tvos
Milestone:	-

[karelz]

These tests are significantly impacting Rolling runs and CI overall. Please disable the tests ASAP.

Note that similar errors first appeared in unmerged PRs: #73258 and #73374 which both seem related -- @directhex perhaps you might know what's going on?

cc @steveisok

[MihaZupan]

Potentially related: #72000
cc: @tmds

[tmds]

   at System.Net.Sockets.Socket.SendPacketsAsync(SocketAsyncEventArgs e, CancellationToken cancellationToken)

I guess this is at:

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs

Lines 2981 to 2984 in 9929753

	if (!Connected)
	{
	throw new NotSupportedException(SR.net_notconnected);
	}

So on these platforms the new cases added in #72000 throw because we can't determine the Socket is connected at:

runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/Socket.cs

Lines 177 to 178 in 9929753

	// Try to determine if we're connected, based on querying for a peer, just as we would in RemoteEndPoint,
	// but ignoring any failures; this is best-effort (RemoteEndPoint also does a catch-all around the Create call).

I'm not sure what happens in the AcceptGetsCanceledByDispose case (may be similar). A stacktrace would help.

@antonfirsov can you take a look? I'm still on leave for a week.

[directhex]

There are plenty of indications that UNIX Domain Sockets are not actually supported on tvOS/iOS, and only work in the simulator by accident. The PRs you mention are part of the investigation in that regard. Waiting on some test results from Alex to determine whether the whole of UDS should be PNSE on one or both of those OSes.

[akoeplinger]

@directhex this is about normal sockets not Unix Domain Sockets.

I think @tmds's investigation makes sense and this is due to new cases added in #72000. @antonfirsov @wfurt would you mind taking a look?

[wfurt]

I think we should disable the tests on affected platforms. I don't know why the UDS would not be connected. The second link make sense only when Socket is created from handle and I don't think this is the case.

[directhex]

OK, I've caught up with what's going on now. I'll ActiveIssue those with this bug.

[karelz]

@directhex thanks! Looking forward for the tests to be disabled.
Please also triage this issue into appropriate milestone. Thanks!

[simonrozsival]

The tests were disabled in #73374

[jeffschwMSFT]

With the test being disabled, I am removing the blocking-clean-ci label, thanks