Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug Report] Unexpected ObjectDisposedException when IoT Hub is throttling #2351

Closed
danigian opened this issue Apr 19, 2022 · 2 comments
Closed

[Bug Report] Unexpected ObjectDisposedException when IoT Hub is throttling #2351

danigian opened this issue Apr 19, 2022 · 2 comments
Assignees
@abhipsaMisra
Labels
bug Something isn't working. IoTSDK Tracks all IoT SDK issues across the board

Comments

@danigian
Copy link
Contributor

Context

  • OS, version, SKU and CPU architecture used: arm32/x86 debian bullseye based docker image
  • Application's .NET Target Framework : net6
  • Device: laptop/rpi
  • SDK version used: 1.40.0 (not sure about IoT Hub C# SDK version used underneath by IoT Edge)

Description of the issue

Cross-posting from Azure/iotedge#6042

Assuming a IoT Hub with 100 devices, one edge gateway and one module with the following code

// Creating registry manager in order to get 50 connection strings
var registry = RegistryManager.CreateFromConnectionString(iothubownerstring);

var deviceTwins = await registry.CreateQuery("select * from devices", 50).GetNextAsTwinAsync();

// Getting 50 devices
var listDeviceTasks = new List<Task<Device>>();
foreach (var element in deviceTwins)
{
	listDeviceTasks.Add(registry.GetDeviceAsync(element.DeviceId));
}
var devices = await Task.WhenAll(listDeviceTasks);

// Building up the connection string manually
var connectionStrings = new List<string>();
foreach (var d in devices)
{
	connectionStrings.Add($"HostName={iothubname}.azure-devices.net;DeviceId={d.Id};SharedAccessKey={d.Authentication.SymmetricKey.PrimaryKey};GatewayHostName={gwhostname}");
}

// Setting AMQP Transport Settings in such a way that Operation Timeout is 10 seconds instead of default 60secs
var amqp = new AmqpTransportSettings(Microsoft.Azure.Devices.Client.TransportType.Amqp_Tcp_Only)
{
	AmqpConnectionPoolSettings = new AmqpConnectionPoolSettings()
	{
		Pooling = true,
		MaxPoolSize = 1
	},
	OperationTimeout = TimeSpan.FromSeconds(3)
};
amqp.RemoteCertificateValidationCallback += (_, _, _, _) => true;

// Constructing list of GetTwin Task
var tasks = new List<Task<Twin>>();
foreach (var element in connectionStrings)
{
	var client = DeviceClient.CreateFromConnectionString(element, new[] { amqp });

	client.SetRetryPolicy(new ExponentialBackoff(int.MaxValue,
				minBackoff: TimeSpan.FromMilliseconds(100),
				maxBackoff: TimeSpan.FromSeconds(10),
				deltaBackoff: TimeSpan.FromMilliseconds(100)));
	tasks.Add(client.GetTwinAsync());
}

await Task.WhenAll(tasks);

I would expect that, for a S1 IoT Hub, the first time I am awaiting the last "tasks", I should be throttled by the amount of registry operations allowed in one single minute.

Therefore I would either expect something to be thrown concerning the Throttling or, given the OperationTimeout set to 3 seconds for AMQP transport setting, a TimeoutException.

Instead, what really happens is that a ObjectDisposedException is thrown. See details below

Exception stacktrace 
System.ObjectDisposedException:
   at Microsoft.Azure.Devices.Client.AuthenticationWithTokenRefresh+<GetTokenAsync>d__19.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsTokenProvider+<GetTokenAsync>d__3.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.TaskHelpers.EndAsyncResult (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.IteratorAsyncResult`1.StepCallback (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Amqp.AsyncResult.End (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at Microsoft.Azure.Amqp.AmqpCbsLink+<>c__DisplayClass4_0.<SendTokenAsync>b__1 (Microsoft.Azure.Amqp, Version=2.4.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotCbsLink+<SendTokenAsync>d__2.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpAuthenticationRefresher+<InitLoopAsync>d__10.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpIotConnection+<CreateRefresherAsync>d__9.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpConnectionHolder+<CreateRefresherAsync>d__14.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<EnsureSessionIsOpenAsync>d__28.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.AmqpIot.AmqpUnit+<OpenAsync>d__27.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler+<OpenAsync>d__13.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler+<OpenAsync>d__7.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<>c__DisplayClass27_0+<<ExecuteWithErrorHandlingAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler+<ExecuteWithErrorHandlingAsync>d__28`1.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<OpenInternalAsync>d__38.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<EnsureOpenedAsync>d__36.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<>c__DisplayClass29_0+<<SendTwinGetAsync>b__0>d.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler+<SendTwinGetAsync>d__29.MoveNext (Microsoft.Azure.Devices.Client, Version=1.39.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at TestCode.DeviceClient+<GetTwinAsync>d__8.MoveNext (TestCode.DeviceClient, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null: /TestCode/DeviceClient.cs:74)

Console log of the issue

Additional IoT Edge logs are to be found in the support bundle file

FYI @ancaantochi (if you want to provide more details about what you found out on "authentication taking longer and the SDK not handling it properly")
@danigian danigian added the bug Something isn't working. label Apr 19, 2022
@github-actions github-actions bot added the IoTSDK Tracks all IoT SDK issues across the board label Apr 19, 2022
@abhipsaMisra
Copy link
Member

Synced up offline with @ancaantochi -

I was able to reproduce as described by customer with getting a list of devices and trying to get twin via edge, it reproduces when leaf devices are not set as child of edge device.

I will try to reproduce at my end and investigate why you're hitting this exception.

@and-rewsmith and-rewsmith mentioned this issue May 3, 2022
Closed
@abhipsaMisra abhipsaMisra mentioned this issue Nov 4, 2022
Merged
@abhipsaMisra
Copy link
Member

abhipsaMisra commented Mar 29, 2023 •
edited
Loading

The fix has been released on both GA and LTS.
GA - https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/2022-11-16
LTS - https://github.com/Azure/azure-iot-sdk-csharp/releases/tag/lts_2021-3-18_patch7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@abhipsaMisra abhipsaMisra

Labels
bug Something isn't working. IoTSDK Tracks all IoT SDK issues across the board
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@danigian @abhipsaMisra