feat(static-sharding)!: allow multiple pubSubTopics #1586

danisharora099 · 2023-09-20T13:29:36Z

This PR, part of the #1310 milestone:

allows consumers to set multiple PubSubTopic during node creation
- keeps our DefaultPubSubTopic still as default
introduced PubSubTopic as an argument for the encoder/decoder creation API
updates all protocols (lightpush, filter, store and relay) to confirm that the request can only be sent to one of the PubSubTopic that were used during initialisation
updates existing tests for the new API

Notes

Resolves bug: pubSubTopic not respected on createLightNode #1446
Related to Static Sharding #1310
best reviewed per commit
while there is work to be done on filtering & connecting with the right relevant peers, this in itself should work on the assumption that js-waku connects to all nodes discovered

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku core	27.81 KB (+0.46% 🔺)	557 ms (+0.46% 🔺)	1.4 s (-0.39% 🔽)	2 s
Waku Simple Light Node	262.97 KB (+0.12% 🔺)	5.3 s (+0.12% 🔺)	3.8 s (-8.73% 🔽)	9.1 s
ECIES encryption	29.56 KB (+3% 🔺)	592 ms (+3% 🔺)	1.6 s (+25.04% 🔺)	2.2 s
Symmetric encryption	29.56 KB (+3% 🔺)	592 ms (+3% 🔺)	1.7 s (-7.59% 🔽)	2.3 s
DNS discovery	110.58 KB (+0.03% 🔺)	2.3 s (+0.03% 🔺)	2.4 s (-25.86% 🔽)	4.6 s
Privacy preserving protocols	117.31 KB (+0.05% 🔺)	2.4 s (+0.05% 🔺)	3.4 s (+0.92% 🔺)	5.7 s
Light protocols	25.96 KB (+0.5% 🔺)	520 ms (+0.5% 🔺)	1.4 s (+16.18% 🔺)	1.9 s
History retrieval protocols	25.13 KB (+0.91% 🔺)	503 ms (+0.91% 🔺)	1.1 s (+16.28% 🔺)	1.6 s
Deterministic Message Hashing	5.65 KB (0%)	113 ms (0%)	583 ms (+22.66% 🔺)	696 ms

fryorcraken

Preliminary review. blocking on the multiple store requests

packages/core/src/lib/base_protocol.ts

packages/core/src/lib/message/version_0.ts

packages/core/src/lib/store/index.ts

packages/core/src/lib/wait_for_remote_peer.ts

packages/interfaces/src/protocols.ts

fryorcraken

Missing various tests:

initiate with multiple pubsub topic
try to use any protocol with a pubsub topic that is not configured.

packages/relay/src/index.ts

…topics

…y & modularity

…sharding

packages/core/src/lib/base_protocol.ts

packages/core/src/lib/filter/index.ts

packages/core/src/lib/keep_alive_manager.ts

…sharding

fryorcraken

I think relay could have been done with a 2nd PR to keep reviewing easy.

packages/utils/src/common/sharding.ts

packages/utils/package.json

packages/tests/tests/wait_for_remote_peer.node.spec.ts

packages/relay/src/index.ts

packages/interfaces/src/relay.ts

packages/message-encryption/src/ecies.ts

packages/message-encryption/src/symmetric.ts

packages/relay/src/index.ts

fryorcraken · 2023-09-26T06:42:28Z

packages/core/src/lib/light_push/index.ts

@@ -41,12 +42,12 @@ type PreparePushMessageResult =
 * Implements the [Waku v2 Light Push protocol](https://rfc.vac.dev/spec/19/).
 */
 class LightPush extends BaseProtocol implements ILightPush {
-  options: ProtocolCreateOptions;
+  private readonly pubsubTopics: PubSubTopic[];


Suggested change

private readonly pubsubTopics: PubSubTopic[];

private readonly pubSubTopics: PubSubTopic[];

refactored this, thanks for the spot.

perhaps this change should be part of a larger code-wide refactor? tracking here: https://github.com/waku-org/js-waku/compare/chore/pubSubTopic-typo?expand=1

packages/relay/src/index.ts

Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>

* refactor: simplify handling of observers * refactor: Remove redundant PubSubTopic from Observer

…o feat/staticsharding

danisharora099 · 2023-09-26T16:00:42Z

I think relay could have been done with a 2nd PR to keep reviewing easy.

looking back in hindsight, would definitely make things simpler! 😅

fryorcraken

Nice.

packages/interfaces/src/relay.ts

packages/relay/src/index.ts

…sharding

danisharora099 added 6 commits September 20, 2023 18:40

ProtocolCreateOptions now has pubSubTopic as pubSubTopic[]

90e5b77

chore: update encoder & decoder to support PubSubTopic

7217497

feat(protocols): allow multiple PubSubTopic[]

c232e9f

feat(relay): allow multiple PubSubTopic[]

0e1e1a1

chore(tests): update for new API

74c8800

chore: minor fixes

bad5d5e

danisharora099 requested a review from a team as a code owner September 20, 2023 13:29

danisharora099 mentioned this pull request Sep 20, 2023

Static Sharding #1310

Closed

9 tasks

danisharora099 mentioned this pull request Sep 20, 2023

bug: pubSubTopic not respected on createLightNode #1446

Closed

danisharora099 added 2 commits September 20, 2023 19:20

chore: make store more robust

24c9ad0

fix(relay): correctly set types

1bac330

fryorcraken requested changes Sep 21, 2023

View reviewed changes

fryorcraken reviewed Sep 21, 2023

View reviewed changes

packages/relay/src/index.ts Outdated Show resolved Hide resolved

packages/relay/src/index.ts Show resolved Hide resolved

packages/relay/src/index.ts Outdated Show resolved Hide resolved

danisharora099 added 6 commits September 21, 2023 13:12

chore(address comments): update terminology around configured pubsub …

cbc514a

…topics

chore(address comments): minor refactoring

697072c

chore(relay): split subscribe into smaller functions for readabilit…

3285849

…y & modularity

chore(address comments): refactor waitForGossipSubPeerInMesh

5392b31

chore(store): only allow to query one pubSubTopic

7f517ad

fix: store bug

76208e2

danisharora099 force-pushed the feat/staticsharding branch from 7d5e2a1 to 76208e2 Compare September 21, 2023 09:28

Merge branch 'master' of github.com:waku-org/js-waku into feat/static…

2036d1f

…sharding

danisharora099 force-pushed the feat/staticsharding branch from cda719b to 2036d1f Compare September 21, 2023 09:39

feat(tests): add some basic tests

356548e

danisharora099 requested review from fryorcraken and weboko September 22, 2023 09:08

weboko reviewed Sep 25, 2023

View reviewed changes

packages/core/src/lib/base_protocol.ts Outdated Show resolved Hide resolved

weboko reviewed Sep 25, 2023

View reviewed changes

packages/core/src/lib/filter/index.ts Show resolved Hide resolved

weboko reviewed Sep 25, 2023

View reviewed changes

packages/core/src/lib/keep_alive_manager.ts Outdated Show resolved Hide resolved

weboko reviewed Sep 25, 2023

View reviewed changes

packages/core/src/lib/keep_alive_manager.ts Outdated Show resolved Hide resolved

danisharora099 added 3 commits September 26, 2023 09:02

sharding utils

fd85ba0

address comments

8e8da03

Merge branch 'master' of github.com:waku-org/js-waku into feat/static…

ea7eef7

…sharding

fryorcraken reviewed Sep 26, 2023

View reviewed changes

danisharora099 and others added 6 commits September 26, 2023 13:49

feat(relay): re-add API for getMeshPeers

a84726b

update error message

a626ceb

Co-authored-by: fryorcraken <110212804+fryorcraken@users.noreply.github.com>

refactor for new API

8054eb4

feat: simplify handling of observers (#1614)

8b9f558

* refactor: simplify handling of observers * refactor: Remove redundant PubSubTopic from Observer

Merge branch 'feat/staticsharding' of github.com:waku-org/js-waku int…

5a01805

…o feat/staticsharding

use ?? instead of ||

8c01b74

danisharora099 mentioned this pull request Sep 26, 2023

chore: refactor all instances of pubsubTopic to pubSubTopic #1618

Closed

2 tasks

danisharora099 added 2 commits September 26, 2023 21:16

update pubsubTopic to pubSubTopic

08b05ea

update interval typo

d49a298

danisharora099 requested review from fryorcraken and weboko September 26, 2023 16:01

change occurence of pubsubTopic to pubSubTopic

11e8094

fryorcraken approved these changes Sep 27, 2023

View reviewed changes

packages/interfaces/src/relay.ts Outdated Show resolved Hide resolved

fryorcraken reviewed Sep 27, 2023

View reviewed changes

packages/relay/src/index.ts Outdated Show resolved Hide resolved

danisharora099 and others added 6 commits September 27, 2023 11:50

relay: rm getAllMeshPeers and make pubSubTopics public

1977bd7

relay: use push_or_init_map and move to utils

3fd68e5

Merge branch 'master' of github.com:waku-org/js-waku into feat/static…

cbf588e

…sharding

fix: update API for tests

03ae72a

fix: relay waitForRemotePeer

6f0cbb7

Merge branch 'master' into feat/staticsharding

7739014

danisharora099 merged commit a3c45b6 into master Sep 27, 2023

danisharora099 deleted the feat/staticsharding branch September 27, 2023 09:58

fryorcraken mentioned this pull request Sep 27, 2023

chore: release master #1568

Merged

danisharora099 mentioned this pull request Oct 11, 2023

chore: use pubsubTopic/pubsubTopics across the codebase #1653

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(static-sharding)!: allow multiple pubSubTopics #1586

feat(static-sharding)!: allow multiple pubSubTopics #1586

danisharora099 commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

fryorcraken left a comment

fryorcraken left a comment

fryorcraken left a comment

fryorcraken Sep 26, 2023

danisharora099 Sep 26, 2023 •

edited

Loading

danisharora099 commented Sep 26, 2023

fryorcraken left a comment

	private readonly pubsubTopics: PubSubTopic[];
	private readonly pubSubTopics: PubSubTopic[];

feat(static-sharding)!: allow multiple pubSubTopics #1586

feat(static-sharding)!: allow multiple pubSubTopics #1586

Conversation

danisharora099 commented Sep 20, 2023 • edited Loading

Notes

Next

github-actions bot commented Sep 20, 2023 • edited Loading

size-limit report 📦

fryorcraken left a comment

Choose a reason for hiding this comment

fryorcraken left a comment

Choose a reason for hiding this comment

fryorcraken left a comment

Choose a reason for hiding this comment

fryorcraken Sep 26, 2023

Choose a reason for hiding this comment

danisharora099 Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

danisharora099 commented Sep 26, 2023

fryorcraken left a comment

Choose a reason for hiding this comment

danisharora099 commented Sep 20, 2023 •

edited

Loading

github-actions bot commented Sep 20, 2023 •

edited

Loading

danisharora099 Sep 26, 2023 •

edited

Loading