altair processAttestations performance issue

[dapplion]

Describe the bug

Looking at the metrics for our altair nodes processing a block takes way longer (300ms) than what the performance tests say (100ms).

Taking a profile shows that all the time is spent on processing attestations

Expected behavior

Should not take so long

---

Note: this CPU profile was taken in an overloaded contabo node.

Running the altair block performance benchmarks in my laptop do

  altair processBlock
    ✓ altair processBlock - 250000 vs - 7PWei normalcase                  10.01820 ops/s    99.81831 ms/op        -         33 runs   17.2 s
    ✓ altair processBlock - 250000 vs - 7PWei worstcase                   4.261845 ops/s    234.6401 ms/op        -         26 runs   7.50 s

While running the same in that contabo's host

  altair processBlock
    ✓ altair processBlock - 250000 vs - 7PWei normalcase                  7.083636 ops/s    141.1704 ms/op   x0.601         14 runs   20.8 s
    ✓ altair processBlock - 250000 vs - 7PWei worstcase                   2.356299 ops/s    424.3943 ms/op   x0.629         24 runs   12.1 s

And Github CI does

altair processBlock - 250000 vs - 7PWei normalcase | 103.91 ms/op | 97.787 ms/op | 1.06
altair processBlock - 250000 vs - 7PWei worstcase  | 261.36 ms/op | 265.06 ms/op | 0.99

[dapplion]

The performance issue comes from having to traverse the tree many times to persist each single new status. If I comment this line out

lodestar/packages/beacon-state-transition/src/altair/block/processAttestation.ts

Line 81 in 43972fb

epochParticipation.setStatus(index, newStatus);

the cost of processing the entire block goes from 100ms -> 23.1ms.

We must apply all the changes in batch at the end of processAttestations(). Maybe with ChainSafe/persistent-merkle-tree#53 ? Also do some pre-aggregate the statuses that will end-up in the same chunk or some bit manipulation magic.

[philknows]

Note to tag this issue with SSZ refactor PR when it's ready.