feat!: support for cursors on store API

[danisharora099]

Problem

The most efficient way to query Waku Store is using a cursor.

The current WakuStore API does not enable a cursor to be passed, the cursor is only used for pagination after the first page is received.

Solution

The cursor is constructed using data from a given Waku message. When using the cursor, the store returns messages starting from the cursor, which is the message used to build the cursor.

If the library consumer knows from which message they want to query, then a cursor could be used from the first page.

This could be useful in the following scenarios:

• last seen message is stored in local storage, when re-opening web page, a forward query is done from this message
• connection dropout: last received message could be used as a cursor

Notes

• Resolves Waku Store API that uses message to build cursor #964

[github-actions]

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
Waku core	132.86 KB (+1.41% 🔺)	2.7 s (+1.41% 🔺)	3.7 s (+86% 🔺)	6.4 s
Waku default setup	300.97 KB (+2.65% 🔺)	6.1 s (+2.65% 🔺)	6.4 s (+13.18% 🔺)	12.4 s
Asymmetric, symmetric encryption and signature	42.56 KB (0%)	852 ms (0%)	1.5 s (-31.05% 🔽)	2.3 s
DNS discovery	107.65 KB (0%)	2.2 s (0%)	3.2 s (-16.02% 🔽)	5.4 s
Privacy preserving protocols	132.03 KB (+1.4% 🔺)	2.7 s (+1.4% 🔺)	2.4 s (-5.09% 🔽)	5 s
Light protocols	132.04 KB (+1.4% 🔺)	2.7 s (+1.4% 🔺)	3 s (+20.61% 🔺)	5.7 s
History retrieval protocols	132.03 KB (+1.4% 🔺)	2.7 s (+1.4% 🔺)	3.2 s (+15.13% 🔺)	5.8 s

[fryorcraken]

When adding help wanted can you also add a comment about what you need help with?

[felicio]

non-blocking: Did you find the answer to why the cursor was implemented as an object and not a string/hash?

[fryorcraken]

It does contain a digest. At the end of the day, it matches the protobuf interface. I think the inner work of the Index type should be a black box for the API consumer:

1. Call createCursor and get something
2. Pass something to the store query

If you have an idea how to improve the API, please do.

[danisharora099]

I had discussed this with @jm-clius earlier and there is indeed an ideal way possible to have the entire arg as a digest of all the 4 fields.
This is simply a design that was implemented earlier and could be changed in the future.

[felicio]

Cc @LNSD

[LNSD]

FYI: I plan to simplify the current protocol as part of the nwaku v0.14.0 release. And streamline the cursor handling is one of the reasons behind this protocol overhaul.

[felicio]

non-blocking: did you consider not allowing startTime and endTime if cursor set? Could be worth ensuring with a type.

[fryorcraken]

good idea 👍 @danisharora099

[danisharora099]

We mean accepting either the timeFilter or the cursor arg here?

cc @felicio

[felicio]

We mean accepting either the timeFilter or the cursor arg here?

Yeah, but I saw the @LNSD's comment about it being allowed since then, which made me reconsider.

@LNSD thanks for the great summary at waku-org/nwaku#1400 (comment). Please,

1. What is meant by "If the cursor timestamps are not included in the time range"? Which cursor timestamps? Is it only senderTime for clients?
2. If so, shouldn't the senderTime, digest and pubsubTopic be actually required, and not optional, fields for clients?
3. And in that case, would there really be use for queries combining "time range + cursor"? Or is it then rather "end time + cursor"?
4. Would/should these combined queries be covered by the planned prepared statements and bindings too?

[danisharora099]

Thanks a lot for these valuable comments @felicio -- addressed them here: #1031

Unable to add you as a reviewer for some reason, but your review on the PR will be appreciated.