fix: improve bitswap message merging #522
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When we send messages repeatedly to the same peer we spend most of our time `base64` encoding data to dedupe messages. This can happen when sending data to Kubo since it only sends us max 32x wants in one message, so for a large DAG we can receive many messages in quick succession. Instead, encode the keys for blocks/wants/presences in base64 at message creation time, then when we need to merge messages it's a simple string comparison instead. We need strings for comparison because JavaScript's `Map` type uses `===` to compare key values, if there was a `Comparator`-style interface we'd be able to use `Uint8Array`s for keys, but there isn't, so we can't.
3 tasks
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When we send messages repeatedly to the same peer we spend most of our time
base64encoding data to dedupe messages.This can happen when sending data to Kubo since it only sends us max 32x wants in one message, so for a large DAG we can receive many messages in quick succession.
Instead, encode the keys for blocks/wants/presences in base64 at message creation time, then when we need to merge messages it's a simple string comparison instead.
We need strings for comparison because JavaScript's
Maptype uses===to compare key values, if there was aComparator-style interface we'd be able to useUint8Arrays for keys, but there isn't, so we can't.Change checklist