Bonsai World State mismatch potential race

[garyschulte]

Description

Post-merge, when besu is asked to propose a block it first creates and executes an empty block before it assembles a block with transactions. Rarely, besu will fail to execute the empty block with a stateroot mismatch, but the block it subsequently produces for the same slot will execute just fine:

See the attached log snippet for an example of this behavior on mainnet-shadow-fork-5:
https://app.zenhub.com/files/206414745/7e454837-e202-4a69-b51e-1ac987dbee96/download

This smells like a race between the database setting the chain head and the corresponding world state and the production of the empty block. It may not be limited to block production after forkchoiceUpdated, but it seems to present most readily there, when a block is produces immediately after the chain head has been set.

Acceptance Criteria

• empty block proposals should execute without any kind of race
• ensure MergeCoordinator.updateForkChoice blocks until all world state transactions are fully committed
• ensure DefaultBlockchain.rewindToBlock blocks until all world state transactions are fully committed

Steps to Reproduce (Bug)

1. The easiest way to observe/reproduce this is on one of the besu validator nodes mainnet shadow fork 5.

[ricardolyn]

We noticed this issue in one of our Besu instances.

Important to mention that the Node didn't enter an unhealthy state. The node was not in sync anymore, so it should have gone to an unhealthy state what that happened.

[ricardolyn]

@daniellehrner created my comment simultaneously as I wrote the comment above: #4007.

Thank you 👍

[matkt]

fixed by #4013