Post-Merge some blocks contain 0 transactions #4401
Assignees
Labels
TeamRevenant
GH issues worked on by Revenant Team
Comments
|
Pre-merge prater empty blocks (old discord thread but same user reporting this with post-merge mainnet) https://discord.com/channels/905194001349627914/938504958909747250/1009795829625786418 |
|
^ empty blocks on prater do happen normally. From discord thread https://discord.com/channels/697535391594446898/1009446584611774535 When asked to produce a block, Besu starts with an empty block proposal candidate, just so it has something to respond to the CL with if it is not done building a block by the time the CL requests it. This historically has happened with nimbus because nimbus was not waiting the prescribed 500 milliseconds between asking for a block to be built and requesting the block. I don't think prysm has had this issue. The other case is that it is possible non of the transactions in besu's pool are eligible for inclusion, either because of basefee or because of nonce. in 22.7.2 besu is changing its transaction eviction policy to ensure that we prioritize future nonce transactions for eviction ahead of lower value but executable transactions. If you query the pending transactions endpoint, you can spot check some of the transactions in the pool to check whether they are executable or not |
|
https://discord.com/channels/905194001349627914/1009795829625786418/1019880263049822208 |
|
we are investigating this - working theory is that this is related to resource constraints - suggestion is to try increasing JVM memory as per https://besu.hyperledger.org/en/latest/public-networks/how-to/configure-jvm/manage-memory/ |
|
It is not resource constraints. I, and many others, have powerful machines and never experienced this pre-merge. CPU and memory are totally not-pressed. |
|
@EvilJordan can you add any logs from besu and/or teku and any other related info to this issue - this will help us a lot! Thanks |
|
from @ricardolyn Bonsai Node size Standard_D8s_v4 we are giving 16gb of ram and we have set JVM max memory percentage to 75%. geth and Besu run on the same spec |
|
Reports on discord - asked for logs - https://besu.hyperledger.org/en/latest/public-networks/how-to/configure-jvm/manage-memory/ |
|
I don't have debug logs on so this isn't terribly instructive. Teku CL and Besu EL, here are the consensus and execution logs for an empty proposal |
|
I am running a NUC8i5BEH, 1TB NVME SSD + 4TB SSD, 32 GB RAM, with a Teku-Besu pair that had run flawlessly until the merge. Post merge it began to see missing blocks in the Teku logs that were not missed onchain, with increasing missed attestations over time, decreasing attestation performance, and the publication of one transaction-free empty block at slot 4701480: https://beaconcha.in/slot/4701480 Here are the Besu logs: besu[285850]: 2022-09-15 13:35:37.739+02:00 | vert.x-worker-thread-0 | INFO | EngineForkchoiceUpdated | VALID for fork-choice-update: head:0x7ac48e0cc8f5bebd1e545661329afe205507acbf9e17604cba9f1ac2b9888e44, finalized: besu[285850]: 2022-09-15 13:36:23.229+02:00 | vert.x-worker-thread-0 | INFO | EngineGetPayload | Fetch block proposal by identifier: 0x000000000e2e9537, hash: 0xbc12bb44a42f5bcc9e00eec4b3c24370111f6740b775d96160b4f16993a958e0, number: 15538842, coinbase:x9c93698ee475f6a62c6f930c410270972ff10019, transaction count: 0 Here are the Teku logs: 2022-09-15 13:36:24.246+02:00 | CompletableFutureDelayScheduler | ERROR | ExecutionLayerManagerImpl | Unable to obtain a valid bid from builder. Falling back to local execution engine. |
|
Since the merge I've proposed two blocks, the first one with just 2 transactions, the second one with only 1 transaction. I also experience frequent missed attestations. I'm running besu/teku (solo), my specs are:
I'm getting the following errors in besu and teku at irregular intervals: |
Our current setup is: Restart doesn't help us to stop producing empty blocks :( |
|
Logs from Besu/Teku which produced empty block |
|
Using geth/teku, I proposed an empty block: https://etherscan.io/block/15545143. Maybe it's just a teku issue instead of a besu issue? Logs are completely uninteresting. Teku: geth: |
|
We have identified a case when there is timeout in RocksDB I am working on a PR for this retry mechanism |
|
@EvilJordan had a crash of its Besu node and it stopped. I also found a potential problem with the transaction pool but I need to investigate further. I don't believe @EvilJordan had exceptions when it had empty blocks. that's why for my part I look at the pool level |
|
it is a user who uses MEV. So for this user it doesn't seem related to Besu. he just did an MEV update and we'll see if that's better |
|
I've since had two more proposals, the first with zero transactions, the second with 341 transactions. I haven't changed or updated anything since the merge. The errors and missed attestations remain. Edit: and just now another proposal containing 1 transaction. |
|
I'm using besu/teku via rocket pool and I've had 3 empty proposals out of 5 since the merge (0 0 1 0 1) |
|
This above error is slightly different and should be fixed by #4407 - still investigating empty blocks. |
2 tasks
Merged
2 tasks
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Issue stub - more details to come. Post-Merge we are seeing some blocks with 0 included transactions.
Acceptance Criteria
Steps to Reproduce (Bug)
The text was updated successfully, but these errors were encountered: