Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[246] 4MiB hash list for uploads #252

Merged
merged 29 commits into from
Jun 8, 2023
Merged

[246] 4MiB hash list for uploads #252

merged 29 commits into from
Jun 8, 2023

Conversation

giventocode
Copy link
Contributor

Closes: #246

This PR includes the following:

  • The block/chunk size of each item in the hash list is 4MiB.
  • Introduced validation to only allow multiples of 4MiB as valid block sizes.
  • The root hash is set in the blob property: md5_4mib_hashlist_root_hash
  • Added bash script to generate the root hash locally /scripts/rootHash.sh

@giventocode giventocode requested review from BMurri and MattMcL4475 June 7, 2023 23:44
@giventocode giventocode marked this pull request as ready for review June 7, 2023 23:46
Copy link
Collaborator

@BMurri BMurri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For some reason I thought we were doing a hash of hashes but I see we are doing a concatenation of hashes. Not necessarily a problem, I was surprised

@giventocode
Copy link
Contributor Author

For some reason I thought we were doing a hash of hashes but I see we are doing a concatenation of hashes. Not necessarily a problem, I was surprised

It is a hash of hashes. The source of the final hash (root hash) is a list of hashes (concatenated, without a separator) of chunks of 4MiB.

@giventocode giventocode mentioned this pull request Jun 8, 2023
16 tasks
@BMurri
Copy link
Collaborator

BMurri commented Jun 8, 2023

For some reason I thought we were doing a hash of hashes but I see we are doing a concatenation of hashes. Not necessarily a problem, I was surprised

It is a hash of hashes. The source of the final hash (root hash) is a list of hashes (concatenated, without a separator) of chunks of 4MiB.

I missed that somehow. I see it now.

@giventocode giventocode merged commit 5425a16 into main Jun 8, 2023
@giventocode giventocode deleted the ja-4mib-hashlist branch June 8, 2023 22:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[#148 Implement TES Task Runner] Use 4MiB block size for md5 hash list.
3 participants