Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move local thread pool work items to global by default when blocking on a task #112796

Merged
merged 3 commits into from
Feb 26, 2025

Conversation

kouvel
Copy link
Member

@kouvel kouvel commented Feb 21, 2025

  • Enables the change in Transfer ThreadPool local queue to high-pri queue on Task blocking #109841 by default
  • A worker thread blocking on a task while having pending local work items can lead to priority inversion and deadlock-like issues, as local work items are stolen with the lowest priority by other threads. In a situation where all worker threads are blocked on tasks and new ones continue to block on tasks similarly, the global queues may not deplete for a long time and any local work items of blocked threads may not run meanwhile. There have been some reports of issues like this occurring while coinciding with other external issues that aggravate the problem.
  • When a worker thread blocks on a task, the change moves any local work items to the high-priority global queue. Using the normal-priority queues can lead to other priority inversion issues in worst-case scenarios like above, where the normal-priority queues may consist of a number of new requests, and local work items typically represent already in-flight work.

@kouvel kouvel added this to the 10.0.0 milestone Feb 21, 2025
@kouvel kouvel requested a review from stephentoub February 21, 2025 19:16
@kouvel kouvel self-assigned this Feb 21, 2025
@Copilot Copilot bot review requested due to automatic review settings February 21, 2025 19:16

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (1)

src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:723

  • The new behavior of moving local work items to a high-priority global queue when blocking on a task should be covered by tests.
internal static void TransferAllLocalWorkItemsToHighPriorityGlobalQueue()
Copy link
Member

@stephentoub stephentoub left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

…on a task

- Enables the change in dotnet#109841 by default
- A worker thread blocking on a task while having pending local work items can lead to priority inversion and deadlock-like issues, as local work items are stolen with the lowest priority by other threads. In a situation where all worker threads are blocked on tasks and new ones continue to block on tasks similarly, the global queues may not deplete for a long time and any local work items of blocked threads may not run meanwhile. There have been some reports of issues like this occurring while coinciding with other external issues that aggravate the problem.
- When a worker thread blocks on a task, the change moves any local work items to the high-priority global queue. Using the normal-priority queues can lead to other priority inversion issues in worst-case scenarios like above, where the normal-priority queues may consist of a number of new requests, and local work items typically represent already in-flight work.
@kouvel
Copy link
Member Author

kouvel commented Feb 25, 2025

Rebased to try tests in latest

@kouvel kouvel merged commit f5c7344 into dotnet:main Feb 26, 2025
136 of 139 checks passed
@kouvel kouvel deleted the MoveLocalWi branch February 26, 2025 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants