Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance testing Fixes and Enhancements #259

Merged
merged 23 commits into from
Jun 29, 2023
Merged

Performance testing Fixes and Enhancements #259

merged 23 commits into from
Jun 29, 2023

Conversation

giventocode
Copy link
Contributor

@giventocode giventocode commented Jun 14, 2023 •
edited
Loading

Closes: #258 , #264

This PR includes:

  • new logic in PipelineOptionsOptimizer to calculate the number of threads for the read and write operations using the CPU core count. The new logic sets the number threads at 10 per core with a maximum of 90.
  • Cancellation of channel read/write operations when worker thread fail.
  • Added timeout when reading from a stream to work around this issue:

@giventocode giventocode requested a review from BMurri June 14, 2023 00:16
@giventocode giventocode changed the title [258] Optimizer sets numbers of threads using the number of cores Performance testing Fixes and Enhancements Jun 19, 2023
BMurri
BMurri requested changes Jun 19, 2023
View reviewed changes
Copy link
Collaborator

@BMurri BMurri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from this view. If it's doing well in the stress test, I'll approve it then.

BMurri
BMurri requested changes Jun 23, 2023
View reviewed changes
@giventocode giventocode merged commit 3c2d8b2 into main Jun 29, 2023
@giventocode giventocode deleted the ja-258 branch June 29, 2023 17:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BMurri BMurri BMurri approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Consider CPU core count when optimizing the number of threads for the readers and writers
2 participants
@giventocode @BMurri