bug: inconsistent / missing messages in status.prod fleet

[richard-ramos]

I created the following project to help debug this scenario: https://github.com/richard-ramos/test-waku-query

• Topic: /waku/2/default-waku/proto
• ContentTopic: /waku/1/0xd6861a81/rfc26
• StartTime: 1668613188, // Convert it to nano
• EndTime: 1668696055, // Convert to nano

and got these results:

>>>> 119 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the first attempt (Used cursor 2 times)
>>>> 242 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the second attempt (Used cursor 3 times)
>>>> 154 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the third attempt (Used cursor 2 times)
>>>> 25 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the first attempt (Used cursor 1 times)
>>>> 20 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the second attempt (Used cursor 1 times)
>>>> 24 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the third attempt (Used cursor 1 times)
>>>> 227 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the first attempt (Used cursor 3 times)
>>>> 227 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the second attempt (Used cursor 3 times)
>>>> 227 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the third attempt (Used cursor 3 times)
>>>> 77 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the first attempt (Used cursor 1 times)
>>>> 243 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the second attempt (Used cursor 3 times)
>>>> 20 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the third attempt (Used cursor 1 times)
>>>> 111 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the first attempt (Used cursor 2 times)
>>>> 115 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the second attempt (Used cursor 2 times)
>>>> 123 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the third attempt (Used cursor 2 times)
>>>> 228 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the first attempt (Used cursor 3 times)
>>>> 56 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the second attempt (Used cursor 1 times)
>>>> 168 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the third attempt (Used cursor 2 times)

What the script does is send the same query 3 times to each node of status.prod fleet and if a cursor is returned, iterate over it. I find it interesting that the results are not consistent. (for each attempt, i expect the results to be the same for the same node for the same time period in the past).

Another attempt at running the same query:

>>>> 80 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the first attempt (Used cursor 1 times)
>>>> 242 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the second attempt (Used cursor 3 times)
>>>> 111 messages found in 16Uiu2HAkvEZgh3KLwhLwXg95e5ojM8XykJ4Kxi2T7hk22rnA7pJC in the third attempt (Used cursor 2 times)
>>>> 62 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the first attempt (Used cursor 1 times)
>>>> 100 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the second attempt (Used cursor 2 times)
>>>> 24 messages found in 16Uiu2HAm6HZZr7aToTvEBPpiys4UxajCTU97zj5v7RNR2gbniy1D in the third attempt (Used cursor 1 times)
>>>> 227 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the first attempt (Used cursor 3 times)
>>>> 100 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the second attempt (Used cursor 1 times)
>>>> 68 messages found in 16Uiu2HAkwBp8T6G77kQXSNMnxgaMky1JeyML5yqoTHRM8dbeCBNb in the third attempt (Used cursor 1 times)
>>>> 52 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the first attempt (Used cursor 1 times)
>>>> 196 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the second attempt (Used cursor 2 times)
>>>> 243 messages found in 16Uiu2HAmFy8BrJhCEmCYrUfBdSNkrPw6VHExtv4rRp1DSBnCPgx8 in the third attempt (Used cursor 3 times)
>>>> 25 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the first attempt (Used cursor 1 times)
>>>> 122 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the second attempt (Used cursor 2 times)
>>>> 16 messages found in 16Uiu2HAmSve7tR5YZugpskMv2dmJAsMUKmfWYEKRXNUxRaTCnsXV in the third attempt (Used cursor 1 times)
>>>> 228 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the first attempt (Used cursor 3 times)
>>>> 8 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the second attempt (Used cursor 1 times)
>>>> 10 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the third attempt (Used cursor 1 times)

The results are not deterministic at all for the same time period

Changing the query to retrieve 10 messages per page, returns interesting results as well: (these results are on a single node)

>>>> 25 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the first attempt (Used cursor 3 times)
>>>> 210 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the second attempt (Used cursor 21 times)
>>>> 44 messages found in 16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg in the third attempt (Used cursor 5 times)

/dns4/node-02.gc-us-central1-a.status.prod.statusim.net/tcp/30303/p2p/16Uiu2HAmDQugwDHM3YeUp86iGjrUvbdw3JPRgikC7YoGBsT2ymMg

QUERY:
{"request_id":"010203040506070800","query":{"pubsubTopic":"/waku/2/default-waku/proto","contentFilters":[{"contentTopic":"/waku/1/0xd6861a81/rfc26"}],"pagingInfo":{"pageSize":10},"startTime":1668613188000000000,"endTime":1668696055000000000}}
RESPONSE:
{"messages":[{"payload":"xjc41v2w0HnuyQtJfGFlGPmkwddTWI42Rekd4yBOnydZTLWSuyeF3Oqd/1HAwdvLSq37Rbf/nNNLbs9M4p8OoJdEqgv41BXcoQS3pRmUuvdXMw5FHbXboghEEg8ze6aO0YikG92D0dxPD0hZLrdSujU2JhMpJafsRdZH7QnpNhb+I/29TYb0x9VDFnRLRsIsXphfmznaD4clhEiLUcvSFPGj+7n+qwdATKdJgcGSUztcdL3Fu51JBhm7ksLqqy7CqcO1YrW7rPcuQXDqBlykob1tb7Ie3fQYMj04B4doAlE0SUjj7SMSRWELZ+Ht3BkAkFRK4FlhiqE5ZaFAATb3gWJ8QYQbIu+bKp/FBxv9w5prU38mh1FYB9lviezDxjPQvOTazwjRwqGVLWjwc/OoHfwZCTbMjbmWhKTlFev6+NsA0452v4Pfcz4RBdrqN9KdqTQxlv6e1S2VXkOnXVvhCLmGyYDhVqTPYZ9QDct4b/6IMWWLzQ+FFZhYxK0BiHDTidWgo1clgxcdxIPY5Fe1ESydNXnjsuUGNYibHcfPqnIlOBBJ8XF1pQ61zzLE+who+CBpHj5X4SjIvXIPqR7lSzKSv6lEWSiZcVxfMZI9iZMcQDUmseopE6Wn5Ip5XWJBH6YIwLOAoYNE5LFflWvXSNo4Nf4e0LXczlt5JxDW6Q/6UGGpwyDliXj1U0wMpEogiAykWrUzFxKdzk1s","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676731055489424},{"payload":"4CgWINgB8Bpc3SGZkARN8A+9a+yvY/s4Jsn6oIeAvFn4cdbpPl36CQyTToZLtuTluacCGwzhNV4AZSqboO7aPxQNxl+XObIuEa0buY5t8rQKE+VqJ1zSTLi/HGbS26456ZzjkrdQcFKjoRYGJ3h9eAiVpIlVFl/3JOrTUD86cZPXmI6/L6V0vTQHmcjFb/q/pCfcmK55AkHKuncL6P1pAhr5GXvVVsMg6usVRh8UmQu4hEfCaqi0lK92ZC5pMQ4b/UQ6oHHE6Wvc8n1mQib1KH0sXpRAPuIOldUSzc/loWOlo0f5erRaOfZ0Xwq2F2JwD/aE0qeLWQ3d3/9qTzfNl4NcvUW0ACVAN5RfPGnyGmFVzoaK5giO3Mk/qOMzW8aVyZT7e20GbfrHbPdnojZxvUuitSWnHgv9HZ8ESWJALGq/Mbto+4s17JdLxjXRdrH0DkYGjVm11PjAYYzLvQz8dphsJlKmpHTiLCsKLDQ9iEpzPk1y/z0HJO1oKpshtf3SyPDGxk1JKfAdaEv6Ku+YvzY/XhqG1pu0DH6Wsqos0ZWUbHaSchTqrnLkmwmXOl6A0ovdfWZuFVhr9YlM05gI6KXi3d2OsNKuklovPEuVzPzwrR4GNPlxc+W/pxgFUkMTYw+vIrGGR7Sfie0A1fMmfWe28dE67wlPFL2JvTr//a1CWP+xP0S5n4DdbFXUmIZFzhmZSNqYyNd3+pwM","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676732643428903},{"payload":"5DZcO3JeGcGWn7ChJmIgwX8ZNRwvJcCplsEow8Iv0VlOxZEZa59yZ+2XYVPcTF/OofEoxezi7mBCjBj2wJYrePaB3+p8skKleczrdi2vx/bBlsrFQGZOqt78pdDNsYHIInZeSZCMy69OtH4A3qFWSDZqRpBaa1zmqu564o1+t/UvXddreb8NwxM9tGr/uhmOiqdAd5CZACZO56rwRdTkpW/qO3rOGkhejcQ/+/Qa7RYM46tQxkUqNga4AVzl1jUiZPK9EUDmoxwNbI2xbeEyg+G4Yzq1kVECepLRs+e9o3V/L+a+uUtMRIZ1mmqISSpRAn3bA5WwBT+SINqT6DQG938NAg8oPbSY3knu0/BvoOyJZ+HwI9fvFNGqbSPywXMIVqjBgTM3yel9bP6OACeEh8YxOYIb3A19cP3/N3LusfQzDjqr1TRke5kqyrddl0kV/1gdPKghvsZ69FBams+InOMy8roZOIbxsOnT1XP3nyGcSlx5x/ENIBywELmj5eLItyjAp0iTarqZYjLITEIRflq0mvmodqdcSiricdbqv1WM3Fh/a7RCbOk0IKbQHP31wSjIqxo2jV/dxz39COUmE0jhu+3AcErF0p9IqhXYH3rn7GOwgY/FImnHKDDG/2PQqYX99awkgqJgmCs/GnCPvnl/MIga2trYcAQcOy0dRXmLd6r4nqiM3AwKrh+7jSDiGfyYl82vVBJH8PON","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676739337700046},{"payload":"sT4zm48nQqAWVOyFHma5MV2bWz2dno66rPNmxfuOGlKZYKmwlbxX/qMd9Pp3zkfYlOm4r4hRN6qUpVVAHb/qFtJ8CQuXWGks1JYPFjLPBAhOQ+2WNsdbzaa/3mrWyZW3JUpMNsPqHayB+B8zwVqAxKWytbuTTFtCWz/ADwynnW4H0KQi1Ks65VWoXYEiZ13Mb6blDCaKpsll2sTzWyoPRRGXKs1qjCyeS2bDqtRol7DKdYXHYRtoW6snGmrinO81lq4TsNABDw2x+C/MCK99OhaUbQuMUrvadhj5wxw+V8vO8AsRniNoZ0qRIyadusqPeOntUgmR2eyI3GmiGR8fr4Ld/hxwyM7NxnslMF3Kmo/y3Prcy9Li+xRtJcXh6YYo/GBICH1ojwJSA0BGnGZWUyQFUIwJ6eJiFyOxT+mRvqPHregB2gN9mxBi5iREmckfxrOx/UEn7YSpWBASkuRkvghL8y68X/47mYAsjDBlt8wh6oFJ0L+xo4Zie0J8nQfzZvnFwzdCvaIm/5bzfpauf019DPGfYRDxhnu0aKzQq6mr69L9Xjw77S/dZHOy9Mnc9RXOgoMQqeWB9aYQpUAErL/8lrtnl/bNtoa8x9G8854aWjW2YtOm0dPiaxpf2o44yZV+sWb7Nv4CoakD6n2p4Wt3MKdLm5ZWMUr/6aEK/PrU5mBX/nTys0iCzgjCs/9gUBtV+l10Gj+85mPG","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676741098989524},{"payload":"bIctDvhFF+OhJUQ360SlssTC9EOasMyDD3losNpL9uuePH8AtbLiCgYXLgATiuCH59ZwWTVgzLNXn+7HfFfYyRdLSjMB5NTW3S0VSuTQ31lnzLmR8nY0051rKSEqnd0wogQERdIkYYYcce4q7UPHn5F2MKy7sYN6CJbJiUDUIzSSwye8+dKlrurn/WrzjQKcfRi/2xeCBReLabNYeI+Bwrd81MTXPghss2McUby5wsZGEi/vaza/GJzbSDEYiC9ukvVQHh/23cht+1K9lL7AB5btEMkNHm+9cox0TkflcVnFX6C+MXOURTBxJsc0fbYoXvt/2rYocHPBvGDWOPvAgtsexRdWWSHXLQQi734kCvmsO90MLfBcDZZgHYFe7mZJvyPHFkQP73oOwRS8h4rMdMW77MbOGzQLPl0wxFZ51Q3Pc1XR+bdfnxluW0Qxxhc+IjLYxkyOJog4kp7cc+z+tbJ9EQbeK3nXdd13Etin+xbooenJTUFHPB8t8iR4ZcqBtapns5SuS9BhWiPoqohrFtqz5nYIKHPWqm1TTVqpsnHxF8TlTgHK4oQsO2us+dV+mEVGWcVrL8/RJc55O4GLfuXQoo686G+llc+zGvX3iP9VurFfPxOBTSEK9UpYkh+iWhsm9W91C7wdLXmLG08odMayyfOBApK+bK7OdzR+itSI5S+rGNdgh5Sh6pCnm2Yvwg4ocUOg63vJctgg","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676742922557701}],"pagingInfo":{"cursor":{}}}

----

QUERY:
{"request_id":"010203040506070800","query":{"pubsubTopic":"/waku/2/default-waku/proto","contentFilters":[{"contentTopic":"/waku/1/0xd6861a81/rfc26"}],"pagingInfo":{"pageSize":10},"startTime":1668613188000000000,"endTime":1668696055000000000}}
RESPONSE:
{"messages":[{"payload":"BdV2BQN2FgX4B9WjtGuPOJQVELPn2PYs+vLmxRNzZ4wr0xH15YE1aF/o819p428+RD/+J0pxBdOh0z3slcpQjCcBIioA1B2g1ZrPri3XqngyQvlZZmLsAx/+o6sqw8B1YtyXbQR6G51YP1bAdqZZMTmyFZtjiw7TCVDpTXMRxe8XLtswMHTCq7Kz/Smso7fKfSqTcvnFI0GoWAZD7zeFS7DCbu6q5n9vm4B/STcFUEK9KgA5i3dG3LODY50yZgDknVHSTttshzonpTYCsQLW3sD3MAH7Sqf3LAg+UTPcIEH8BLnWTeEIB7oOZlIrevxX/oUjryIrvORz+D9nmLTR7F/500oDzFxWBDOziLqCbaJHLnoYclbC1npCTf2I8EjQUPiQAfyMUlvFdDTdh9TgrPaFRR+awZ8G8QueU3URWVgekwuMSZsgCoolZWddlPaZh/XhzITyH3Ed35JQAZKfHrdjOixi0oW8CHqyMpGOrAAb9EgJuyPXZtUTy0CGoUSb/JFsj6Rybq2M6pw3P72x+J5PhB4DMGUdo1QUfspkT6kDV1AcF0vxf72vpsHWZhh199Ev0H7pKlCgV0oEHxUk+AwLI6OqmWYfYz5NCrjmUn9GOtVw/EbTOo0XY0nXA9j3YOi9dnkzHInCvF3Xh4q5prN0nHzzsrjqsqvGkwCEkdUWRABd+88CHqBIlqkFTWH4StDoj6quMmwjODav","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676720606173074},{"payload":"7JYelKqvySbA1k8oQha38NwJcgTNWA5Iaf7KBZfwUTBZNEbv03+gjrs0fO1JZ+HmDkQ1cT61eTAiWqSz7JQem2m1HSDZdYCEmSIuKWPqGjynZ+a8RAXYSNipAiQjqXj6DUE5aAIs7+tHe0TP5hORzcf+VeYVjbS3ZNtE6mOQswbHvRbowP2ayoGTOSomAC6Q0OFCf8LYP4yQb/d6GQFhXVTZWztjYdsd3WVDiGsies6BK7GbCK8g+OzABmiNp24EzcjFtaCCJAw3QHA6lmK7gh/fw2JdU7YGPBdk9KfBBfytO8rRm6dIvQBdLzEC55GIvL+5uQXyvdxbyLR5kk/BmRLHGAulfoOGNWmjOKNdmggfq7s5NplsnFraDLnJxOwX7EOVQdzal2PypsVaXt3a7VAee0+ZA220XVD9sSSWFMFcKr5Y/Y35THzlcxdRraBnTkn67ifM/kKNd/b41oFklJiAIyVT4Oqx32xQxD0qXwbEd8cN2CuT+RZLbhqav5jicLxUXazoZL4yjn9hu4x1n5Ie0EM3DoqLY2P+yOa6zhBANiE7zgwNTRnTcQwHJHoYcFCvXKjgUDo6SQShI9t5Zcm+19NeY88RCUi/yIejuQpe5qn/QSlnVKSGsQbYyOxnS+Nf+uDw6llGgCwTu1TWw2NkkyimpYEfFCiFoPftJxFtru4ACf+6U8C+wFJrQE58c4WL0KktJ0pzYABP","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676722688838906},{"payload":"kvVu2z3hahST5BmZTrEYQdoHhAWD30pbUA5wavig2z9hqIoP6qYJD5KBXQ8Vjf1/19xoIw0ec9dfR79xPoi1w/5800boxU619TF+pgztwgQkeG7FiwSUKawBz5ppZiyiHBfNoVaisu8QLCZ7wWwiba62x9Ur6in7wFKi5CSBeJL+7dgu2rzKxgbwympx3V5ulM3tuNIh1wJHJavdMeW/qVc5lGG3ZvL/P7VYgV+Hj3D2OcApMmEfZDR/WvC3gQzXbPvZ3tw3ja0nHj37xBmEPVMwTKysobSQDliIe4JZpxmv9Z0KdxWJPEHwNNoUirFZIn57owoypoVB8lObStaja8UOQb6ORkZZgwKFpuLF3WXCskrq7pg/2WEE3VEkYyQ8sHmv66hKfh0JzQnC+ezLK1oDEH9YM/kVvY3/hIJdtEDD2ca1UgmrZhuHRXnBKbGkdJ/+mSYxMeJjdWRMRRe0UpyMjERFn0ZHAUX4juLUkKmZ+k6gFSUF5kdm0VyFrE7R8hSWlzb1zVtj7lEJsJwfbDqJIeUpTvBoEG5psImSL7L63sCwc+r6qj5pPhTIz1y5pQ9hEeRhjsvNg/co8mbgn9t1wBcMA9chERQBDs5+823Y+iHaHjjBZFTwmgrSCNW22du8QrGOETNYVTSN61DpcEkeK8FYbYxU7bfFwDoFoiyxLBU/+OzgLxKAU51bKEVsgOirv6fqFF5Hw06z","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676725286043437},{"payload":"5lqupXcUJUxQxhaTRca7BuvHdUnNuyVicQ9W8HY3TyTnilnVBf8+ZW/r64HwIlRIy4Y3dFS7SUi9EqMx7X3k21QzqksUy1O/Tmak6Y3qE9zmxePV7yQd8S0dreIDn7+ASYN00wda5JFfuiiwLIb/KxwdWFXwdhpTABjqAwKiz0IbneGUJJ4Qrqk8qqThM7NQ8JHu0/zvPmJ9jPQ0k8Asn7XWWgZppxAGkQQQc8cH46GnBnrGZqQfTkNTGd5QArpk2eH6XazX4DqeqE8QBGivIai9Xv+0LcVIDgE2nKTceekLReoUZaEHamiye2GpddAi0HpTaKFOX1uKeUSlyf8wa9BC4FamE6lPyIQDtxrNNxkuUrkkH2N0SC92crnoUpKOOUhHCLZS4QRUsi+zU+6zbl3Z6ppjsy/fKTd+n3Yjdh2nud64rGlfSXzFn0ztuoX65KCiXwQq93wwpqKOBpbF4Vw9hSrYZYb/1caGO/9e1XgPXrYTq8c3XDf37zi7qZX4+hLZLkhbeMW5LmcA5k6oHGFJ1lJlMgV4K3lO9ZCvI18R6H2v/yAOXPgH28bf7t4NTj74Oxk2rZyxL3BhrS/hqjf554VxK+rE5tTlIsoN5xS9XtbdkdRntwEmLx8dbPZEd++uO1oGIAdhwx9e2j3VaWWGgUX3jCN/9oSCfHNbCrGsT/NTP+MIDNwrQstE8SN7m6ESRWIrRy7nzSyD","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676727062306301},{"payload":"Kjk5eDfhXo76MNj+plUTu+ETraoQVoaLyzwZ0zeYKv/oM2AaBofiAsFTMdf0TvEYVZXCvm6n7orMhgLJtet7r9l9On9E31xecXqgdyUY0+dDOBWN88pPXtdz9z4RCyIwJs4eHH65kquF2beWsRVpdqLC5RUcFa04zEve/ItzopZM9DBw8BccyLmo2brFzOvdBAjPlFVo2dketVdwBydQTIAwGTinWE1LPcH26ueoXhoSPFoqpq5Hw74evigVA/CAtpNk6RgddKPE7iCrAdNfd1/Dh9eKuw55ZAsyDufvXFDHdMPRf8VfGEwBWJMTv2y+5jwxHmw859xp7ISZni95Cq58feHekLvzBWalZLrgGiNEeXKNHh5Q24oTmAAIDtkPmbWx7ZJXp8an2Kp5IVR2rt4WgZl1jPdWqpXtmgPnAWSeojkE3k36HBrqc8Q/TznRnFFZoMkcuAhQbyXogOribEwt06ervXmovQdhLN610DzECufwTT2eFHiHS9/U6h1RjKvFURPoFZ9UzPM1DdglYzEQ8wo2d0+W1NF0IQj/0ZBlMRNJgftIxAfxuX6eU4jcXNtsbF+E9cjFvgegY1nNt94r9YhJDttH1cKUNamNBVCEKrlxqAttjWyppJ5zmcs2X2WmiBoGKsp3jbGK5mIXCFp95TwGHH0YNhopwdoBLT61h+XIYl+yeI7fyo3iNHOa/ItMh4BuY5zfQUaE","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676728774258383},{"payload":"xjc41v2w0HnuyQtJfGFlGPmkwddTWI42Rekd4yBOnydZTLWSuyeF3Oqd/1HAwdvLSq37Rbf/nNNLbs9M4p8OoJdEqgv41BXcoQS3pRmUuvdXMw5FHbXboghEEg8ze6aO0YikG92D0dxPD0hZLrdSujU2JhMpJafsRdZH7QnpNhb+I/29TYb0x9VDFnRLRsIsXphfmznaD4clhEiLUcvSFPGj+7n+qwdATKdJgcGSUztcdL3Fu51JBhm7ksLqqy7CqcO1YrW7rPcuQXDqBlykob1tb7Ie3fQYMj04B4doAlE0SUjj7SMSRWELZ+Ht3BkAkFRK4FlhiqE5ZaFAATb3gWJ8QYQbIu+bKp/FBxv9w5prU38mh1FYB9lviezDxjPQvOTazwjRwqGVLWjwc/OoHfwZCTbMjbmWhKTlFev6+NsA0452v4Pfcz4RBdrqN9KdqTQxlv6e1S2VXkOnXVvhCLmGyYDhVqTPYZ9QDct4b/6IMWWLzQ+FFZhYxK0BiHDTidWgo1clgxcdxIPY5Fe1ESydNXnjsuUGNYibHcfPqnIlOBBJ8XF1pQ61zzLE+who+CBpHj5X4SjIvXIPqR7lSzKSv6lEWSiZcVxfMZI9iZMcQDUmseopE6Wn5Ip5XWJBH6YIwLOAoYNE5LFflWvXSNo4Nf4e0LXczlt5JxDW6Q/6UGGpwyDliXj1U0wMpEogiAykWrUzFxKdzk1s","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676731055489424},{"payload":"4CgWINgB8Bpc3SGZkARN8A+9a+yvY/s4Jsn6oIeAvFn4cdbpPl36CQyTToZLtuTluacCGwzhNV4AZSqboO7aPxQNxl+XObIuEa0buY5t8rQKE+VqJ1zSTLi/HGbS26456ZzjkrdQcFKjoRYGJ3h9eAiVpIlVFl/3JOrTUD86cZPXmI6/L6V0vTQHmcjFb/q/pCfcmK55AkHKuncL6P1pAhr5GXvVVsMg6usVRh8UmQu4hEfCaqi0lK92ZC5pMQ4b/UQ6oHHE6Wvc8n1mQib1KH0sXpRAPuIOldUSzc/loWOlo0f5erRaOfZ0Xwq2F2JwD/aE0qeLWQ3d3/9qTzfNl4NcvUW0ACVAN5RfPGnyGmFVzoaK5giO3Mk/qOMzW8aVyZT7e20GbfrHbPdnojZxvUuitSWnHgv9HZ8ESWJALGq/Mbto+4s17JdLxjXRdrH0DkYGjVm11PjAYYzLvQz8dphsJlKmpHTiLCsKLDQ9iEpzPk1y/z0HJO1oKpshtf3SyPDGxk1JKfAdaEv6Ku+YvzY/XhqG1pu0DH6Wsqos0ZWUbHaSchTqrnLkmwmXOl6A0ovdfWZuFVhr9YlM05gI6KXi3d2OsNKuklovPEuVzPzwrR4GNPlxc+W/pxgFUkMTYw+vIrGGR7Sfie0A1fMmfWe28dE67wlPFL2JvTr//a1CWP+xP0S5n4DdbFXUmIZFzhmZSNqYyNd3+pwM","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676732643428903},{"payload":"5DZcO3JeGcGWn7ChJmIgwX8ZNRwvJcCplsEow8Iv0VlOxZEZa59yZ+2XYVPcTF/OofEoxezi7mBCjBj2wJYrePaB3+p8skKleczrdi2vx/bBlsrFQGZOqt78pdDNsYHIInZeSZCMy69OtH4A3qFWSDZqRpBaa1zmqu564o1+t/UvXddreb8NwxM9tGr/uhmOiqdAd5CZACZO56rwRdTkpW/qO3rOGkhejcQ/+/Qa7RYM46tQxkUqNga4AVzl1jUiZPK9EUDmoxwNbI2xbeEyg+G4Yzq1kVECepLRs+e9o3V/L+a+uUtMRIZ1mmqISSpRAn3bA5WwBT+SINqT6DQG938NAg8oPbSY3knu0/BvoOyJZ+HwI9fvFNGqbSPywXMIVqjBgTM3yel9bP6OACeEh8YxOYIb3A19cP3/N3LusfQzDjqr1TRke5kqyrddl0kV/1gdPKghvsZ69FBams+InOMy8roZOIbxsOnT1XP3nyGcSlx5x/ENIBywELmj5eLItyjAp0iTarqZYjLITEIRflq0mvmodqdcSiricdbqv1WM3Fh/a7RCbOk0IKbQHP31wSjIqxo2jV/dxz39COUmE0jhu+3AcErF0p9IqhXYH3rn7GOwgY/FImnHKDDG/2PQqYX99awkgqJgmCs/GnCPvnl/MIga2trYcAQcOy0dRXmLd6r4nqiM3AwKrh+7jSDiGfyYl82vVBJH8PON","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676739337700046},{"payload":"sT4zm48nQqAWVOyFHma5MV2bWz2dno66rPNmxfuOGlKZYKmwlbxX/qMd9Pp3zkfYlOm4r4hRN6qUpVVAHb/qFtJ8CQuXWGks1JYPFjLPBAhOQ+2WNsdbzaa/3mrWyZW3JUpMNsPqHayB+B8zwVqAxKWytbuTTFtCWz/ADwynnW4H0KQi1Ks65VWoXYEiZ13Mb6blDCaKpsll2sTzWyoPRRGXKs1qjCyeS2bDqtRol7DKdYXHYRtoW6snGmrinO81lq4TsNABDw2x+C/MCK99OhaUbQuMUrvadhj5wxw+V8vO8AsRniNoZ0qRIyadusqPeOntUgmR2eyI3GmiGR8fr4Ld/hxwyM7NxnslMF3Kmo/y3Prcy9Li+xRtJcXh6YYo/GBICH1ojwJSA0BGnGZWUyQFUIwJ6eJiFyOxT+mRvqPHregB2gN9mxBi5iREmckfxrOx/UEn7YSpWBASkuRkvghL8y68X/47mYAsjDBlt8wh6oFJ0L+xo4Zie0J8nQfzZvnFwzdCvaIm/5bzfpauf019DPGfYRDxhnu0aKzQq6mr69L9Xjw77S/dZHOy9Mnc9RXOgoMQqeWB9aYQpUAErL/8lrtnl/bNtoa8x9G8854aWjW2YtOm0dPiaxpf2o44yZV+sWb7Nv4CoakD6n2p4Wt3MKdLm5ZWMUr/6aEK/PrU5mBX/nTys0iCzgjCs/9gUBtV+l10Gj+85mPG","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676741098989524},{"payload":"bIctDvhFF+OhJUQ360SlssTC9EOasMyDD3losNpL9uuePH8AtbLiCgYXLgATiuCH59ZwWTVgzLNXn+7HfFfYyRdLSjMB5NTW3S0VSuTQ31lnzLmR8nY0051rKSEqnd0wogQERdIkYYYcce4q7UPHn5F2MKy7sYN6CJbJiUDUIzSSwye8+dKlrurn/WrzjQKcfRi/2xeCBReLabNYeI+Bwrd81MTXPghss2McUby5wsZGEi/vaza/GJzbSDEYiC9ukvVQHh/23cht+1K9lL7AB5btEMkNHm+9cox0TkflcVnFX6C+MXOURTBxJsc0fbYoXvt/2rYocHPBvGDWOPvAgtsexRdWWSHXLQQi734kCvmsO90MLfBcDZZgHYFe7mZJvyPHFkQP73oOwRS8h4rMdMW77MbOGzQLPl0wxFZ51Q3Pc1XR+bdfnxluW0Qxxhc+IjLYxkyOJog4kp7cc+z+tbJ9EQbeK3nXdd13Etin+xbooenJTUFHPB8t8iR4ZcqBtapns5SuS9BhWiPoqohrFtqz5nYIKHPWqm1TTVqpsnHxF8TlTgHK4oQsO2us+dV+mEVGWcVrL8/RJc55O4GLfuXQoo686G+llc+zGvX3iP9VurFfPxOBTSEK9UpYkh+iWhsm9W91C7wdLXmLG08odMayyfOBApK+bK7OdzR+itSI5S+rGNdgh5Sh6pCnm2Yvwg4ocUOg63vJctgg","contentTopic":"/waku/1/0xd6861a81/rfc26","version":1,"timestamp":1668676742922557701}],"pagingInfo":{"pageSize":10,"cursor":{"digest":"YuwV/LiS0J9sT72sb0NY7vmvgEC6eZcVCmYAAAAAAAA=","receiverTime":1668676718616523752,"senderTime":1668676718616523752,"pubsubTopic":"/waku/2/default-waku/proto"}}}

Here, the same request sent twice, and different responses are received. Notice the cursor is empty in one of these responses, also, the number of messages returned is different

[richard-ramos]

Related issues:

• Fetching messages not behaving as expected after restoring from seed phrase status-im/status-desktop#8291
• Messages from offline are not fetched on status.prod fleet status-im/status-mobile#14390

[fryorcraken]

1. Is it confirmed to be due to regular restarts of the nodes because of bug: underlying websocket connection leak #1186 ? Can we check missing message timestamp and restart timings to confirm?
2. If (1) is true, what work around can we apply until fd leak is resolved? Is it time to dogfood FT store at node start?

[LNSD]

The current version deployed in status.prod fleet is v0.11-107-g1cb3f4; this is a pre-release version of v0.12.0 (1 or 2 commits before v0.12.0).

We recommend upgrading to the latest release version, v0.13.0. We are not supporting previous releases.

[LNSD]

Status mobile issue: status-im/status-mobile#14390

[LNSD]

I draw some conclusions over the weekend; this is the update:

1. From a client perspective: One should establish the history query goal.

Either get all messages within a time range (e.g., get all messages stored in the last 2h). In this case, what will mark the end of the query's get page loop will be the absence of cursor.

Or get an arbitrary number of messages (e.g., fill the screen with N messages query). In this case, what will mark the end of the query's get page loop will be the total number of messages retrieved, i.e., you concatenate in a list all the items received, and once you have the N desired messages, then you stop asking for more pages.

2. Regarding the query parameters and the convenience of using a time range + cursor combination:

It depends on your goal. See #1

The startTime is redundant if the cursor is present. Both mark the search start point. But! If the cursor timestamps are not included in the time range, startTime mark the start point.

3. The direction of the query, forward or backward, affects the cursor and time range parameters.

• Forward:: The search is performed in chronological order. From older timestamps to newer timestamps. As a consequence, startTime < endTime.
• Backward: The search is performed in the direction. As a consequence, startTime > endTime.

4. Message ordering is fundamental to performing deterministic pagination using a cursor. The database, by definition, does not deliver a deterministic row sequence without a deterministic order by clause.

I am investigating this ATM.

[LNSD]

From PR #1406:

Regarding #1400, some changes have been performed in the SQLite driver queries:

• Aggregated all the query-building logic under the same proc: https://github.com/status-im/nwaku/blob/feat-waku-archive/waku/v2/protocol/waku_archive/driver/sqlite_driver/queries.nim#L237-L275
• We are running the "message query test suites" for the SQLite and Queue drivers with extended coverage.

While extending the query's test coverage, an inconsistency has been detected in the SQLite query used when providing a cursor and a pubsub topic. Ignoring the pubsub value contained in the cursor seems to fix the issue.

• The "row value" comparison no longer includes the pubsub topic value: https://github.com/status-im/nwaku/blob/feat-waku-archive/waku/v2/protocol/waku_archive/driver/sqlite_driver/queries.nim#L244-L248
• The query's order by clause no longer includes the pubsub topic column: https://github.com/status-im/nwaku/blob/feat-waku-archive/waku/v2/protocol/waku_archive/driver/sqlite_driver/queries.nim#L288

At this moment, we are working towards providing a developer preview version (a build from the master branch) to test in one of the fleets.

cc @cammellos @richard-ramos @fryorcraken

[LNSD]

While analyzing some of the queries observed in the Kibana logs, I noticed one thing:

INF 2022-11-18 14:22:39.185+00:00 received history query                     topics="waku store" tid=1 file=protocol.nim:232 peerId=16U*7HZM3b requestId=e53eadb6eef2d1186b2b query="some((contentFilters: @[(contentTopic: "/waku/1/0xd6861a81/rfc26")], pubsubTopic: some("/waku/2/default-waku/proto"), pagingInfo: some((pageSize: some(500), cursor: some((pubsubTopic: "/waku/2/default-waku/proto", senderTime: 1666380751000000000, receiverTime: 1666380751000000000, digest: (data: [110, 136, 230, 68, 51, 177, 105, 134, 91, 164, 95, 170, 254, 119, 238, 70, 75, 154, 158, 10, 10, 205, 20, 71, 120, 175, 105, 152, 14, 17, 246, 204]))), direction: some(FORWARD))), startTime: some(1668767108000000000), endTime: some(1668767638000000000)))"

The kibana record link.

This query logged here corresponds to the following query parameters:

contentFilters:
  - contentTopic: "/waku/1/0xd6861a81/rfc26" 
pubsubTopic: "/waku/2/default-waku/proto"
pagingInfo: 
  pageSize: 500  # <- This is unnecessarily high, nwaku caps this to 100
  cursor: 
    pubsubTopic: "/waku/2/default-waku/proto"
    senderTime: 1666380751000000000 # 2022-10-21 19:32:31
    receiverTime: 1666380751000000000 # 2022-10-21 19:32:31
    digest: "0x6e88e64433b169865ba45faafe77ee464b9a9e0a0acd144778af69980e11f6cc"
  direction: "FORWARD"
startTime: 1668767108000000000  # 2022-11-18 10:25:08
endTime: 1668767638000000000  # 2022-11-18 10:33:58

Even though this may not explain the inconsistencies in the query results, there are a couple of things that should be investigated:

• The cursor timestamps (senderTime and receiverTime) are not within the time range defined by startTime and endTime. In that case, the cursor won't have any effect.
• The page size is unnecessarily high. Nwaku caps the query to 100 messages per query.

@richard-ramos @cammellos Can you check where those queries for 500 messages are coming from? Where is that cursor coming from? Is it necessary to request 500 messages in one query? Do all those messages fit the Desktop/Mobile/Web app screen?

[LNSD]

The issue was located in the Nwaku SQLite bindings wrapper. See #1415 for more information.

Closing the issue since the tests performed confirmed that the problem was fixed.

[jm-clius]

Issue fixed from nwaku's perspective. Tested for consistency and performance at high query rates. Assigning to Client Team to verify:

• that store queries now return consistent responses
• that consistent responses are returned even at high query rates (e.g. worst case scenario where multiple clients perform 30 day's worth of queries).

Feel free to close when confirmed.

cc @cammellos @richard-ramos

[jm-clius]

@cammellos @richard-ramos afaik this issue has been addressed and tested? Can we close?

[richard-ramos]

Issue addressed and tested 🚀
Thank you!