Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[237] Optimizer sets the block size dynamically. #247

Merged
merged 21 commits into from
Jun 7, 2023
Merged

[237] Optimizer sets the block size dynamically. #247

merged 21 commits into from
Jun 7, 2023

Conversation

giventocode
Copy link
Contributor

Closes: #237

This PR includes:

  • New functionality in PipelineOptionsOptimizer to adjust the block size if a large file is provided.
    • The increment unit is 4MiB.
  • Decreased default block size to 8MiB from 10MiB
  • Removed check of max 50,000 blocks allowed, a more nuance validation will be implemented as follow up, for cases where defaults are not used.

@giventocode giventocode marked this pull request as ready for review June 6, 2023 15:45
@giventocode giventocode requested a review from MattMcL4475 June 6, 2023 15:45
@giventocode giventocode force-pushed the ja-dynamic-blocksize branch from 022008e to a155627 Compare June 7, 2023 19:58
@giventocode giventocode requested a review from BMurri June 7, 2023 20:00
@giventocode giventocode merged commit 3df896d into main Jun 7, 2023
@giventocode giventocode deleted the ja-dynamic-blocksize branch June 7, 2023 21:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@BMurri BMurri BMurri approved these changes

@MattMcL4475 MattMcL4475 Awaiting requested review from MattMcL4475

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Dynamically set the block size depending on the source file for uploads.
2 participants
@giventocode @BMurri