v1.2.1: Low mesh peers due to p7

[twoeths]

Describe the bug

Gossipsub score has been low since v1.2.1

due to P7

Expected behavior

Mesh peers should be the same to v1.1.0

[twoeths]

when I zoom in a range of 6h using 1m rate, it shows that if messages are delivered they're always within 6s. It means peers get penalties because they don't deliver messages they promise at all (or within 30s)?! Note that gossipsub delete tracked messages after 30s

Update: this metric is the same to v1.1.1

this is not likely an I/O lag issue

[twoeths]

lodestar v1.2.1 actually does not subscribe to all subnets (even with a node of >= 64 validators)

somehow a subnet topic is subscribed then unsubscribed all the times

[twoeths]

Lodestar unsubscribe and then subscribe same subnet frequently

Nov-29 10:24:23.005[network]       ^[[36mverbose^[[39m: Unsubscribe to gossipsub topic topic=/eth2/4a26c58b/beacon_attestation_50/ssz_snappy
Nov-29 10:24:23.005[network]       ^[[36mverbose^[[39m: Unsubscribe to gossipsub topic topic=/eth2/4a26c58b/beacon_attestation_52/ssz_snappy
Nov-29 10:24:23.005[network]       ^[[36mverbose^[[39m: Subscribe to gossipsub topic topic=/eth2/4a26c58b/beacon_attestation_52/ssz_snappy
Nov-29 10:24:23.005[network]       ^[[36mverbose^[[39m: Subscribe to gossipsub topic topic=/eth2/4a26c58b/beacon_attestation_50/ssz_snappy

[twoeths]

the gossipsub 4.1.1 unsubscribe()/leave() is async and not finished so it ignores the subscribe() call if we do it immediately https://github.com/ChainSafe/js-libp2p-gossipsub/blob/v4.1.1/src/index.ts#L1680

for the context, in gossipsub 0.14.1 leave() method is sync, that's why issue did not happen in lodestar v1.1.1 https://github.com/ChainSafe/js-libp2p-gossipsub/blob/v0.14.1/ts/index.ts#L1602

from lodestar side, it should not unsubscribe() then subscribe() immediately

[twoeths]

Gossipsub broken promises on stable lg1k after we deploy v1.2 on Nov 09:

[wemeetagain]

Take a look at the metrics on the latest libp2p branch. Seems p7 issue is resolved there

[twoeths]

a lot of broken promises are due to the collision of fastMsgId function, i.e. 2 different messages have the same fastMsgId. Right now tracer ignores duplicate messages, we should also track them as delivered, see ChainSafe/js-libp2p-gossipsub#385

[wemeetagain]

Resolved with v1.3.0

[twoeths]

to be clear, this is fixed in curent unstable after libp2p is upgraded, so v1.4.0 not v1.3.0