Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid bursting object store reads caused by too many SSTs in compaction task input #8559

Closed
zwang28 opened this issue Mar 15, 2023 · 1 comment
Closed

Avoid bursting object store reads caused by too many SSTs in compaction task input #8559

zwang28 opened this issue Mar 15, 2023 · 1 comment
Labels
component/storage Storage type/bug Something isn't working
Milestone
release-0.19

Comments

@zwang28
Copy link
Contributor

zwang28 commented Mar 15, 2023 •
edited
Loading

Describe the bug

When a compaction task contains a lots of SSTs as input, it incurs bursting object store reads due to 1) meta block prefetch and 2) opening streaming read for SSTs. It can result in compaction task failure due to some resource limits (which I still cannot figure out #7002):
risingwave_storage::hummock::compactor: Compaction task 2198903 failed with error: ObjectStore failed with IO error timeout: error trying to connect: HTTP connect timeout occurred after 3.1s."

To Reproduce

Let compactor executes a large compaction task, which contains 1000+ small SSTs.

Expected behavior

No response

Additional context

https://risingwave-labs.slack.com/archives/C03DLHWHEN4/p1678857503219889
@zwang28 zwang28 added type/bug Something isn't working component/storage Storage labels Mar 15, 2023
@github-actions github-actions bot added this to the release-0.1.18 milestone Mar 15, 2023
@zwang28 zwang28 changed the title Avoid large concurrent object store reads caused by too many SSTs in compaction task input Avoid bursting object store reads caused by too many SSTs in compaction task input Mar 15, 2023
@zwang28
Copy link
Contributor Author

zwang28 commented Mar 17, 2023

Fixed by #8563. Thanks @Little-Wallace

@zwang28 zwang28 closed this as completed Mar 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/storage Storage type/bug Something isn't working
Projects
None yet
Milestone
release-0.19
Development

No branches or pull requests
1 participant
@zwang28