Transaction that is added to a node's pool may not be propagated to the pools of its peer nodes

[nano-adhara]

Introduction

In certain situations, a transaction that is added to a node's pool (of sequenced type) may not be propagated to the pools of its peer nodes, resulting in the transaction being present only in the transaction receptor node's pool rather than being distributed across all nodes' pools.

Summary

We are using the following five nodes setup for a private Besu network with QBFT protocol:

• A non-validator node that doesn’t participate in the consensus protocol.
• One node acting as a validator.

Users connect to the non-validator node to send transactions, and it propagates them to the validator so that they are included in new blocks.

Starting from a scenario where all the nodes have a transaction (let’s call it “tx_a”) in the pool, as shown in the first image.

If the transaction is dropped from the pool of the nodes for any reason, the pools of the nodes are empty as we can see in the second image.

Then, if the transaction is sent again to the non-validator, it is added to its pool, but it is not added to the validator pool as we can see in the third image.

Detailed description

We have observed 2 problematic scenarios related to how the nodes handle their caches:

1. When a client sends a transaction to the non-validator node, if this node dropped the transaction before, the node doesn't propagate the transaction to the peers in the network. If the transaction is not propagated to a validator, it will never be mined.
2. When a peer propagates a transaction to the validator node, if this node has already received the transaction from a peer, the transaction is not promoted to the transactions pool. If the transaction is not promoted to the transactions pool, it will never be mined.

Both scenarios are problematic if the transaction is valid but the node dropped it from the pool before mining the transaction for any reason, e.g. exceeding tx-pool-retention-hours limit.

We were able to add that transaction again to the validator’s pool only by restarting all the nodes and sending the transaction again.

It seems to be a bug by which internal caches are not updated properly when transactions are dropped/evicted from the pool, not allowing those transactions to be added into new blocks as they are not propagated by the non-validator node and they are rejected by the validator as well.

Versions

This issue occurs in the latest version of the system, as of the time of this writing (v24.3.3). We believe that this occurs in previous versions as well.

Steps to reproduce

This is an explanation of how we were able to reproduce the bug:

1 . Set up at least one non-validator node and at least one validator node with a QBFT network configuration.
3. Send a new transaction using a higher nonce than the expected one, so a nonce gap is created, and that transaction will remain in the pool of both the non-validator and validator nodes.
4. Force the transaction to be dropped from the pool (for example waiting for the tx-pool-retention-hours to expire).
5. Resend the transaction to the non-validator node and you will be able to check that it won’t appear in the validator’s pool (as it does not get propagated by the non-validator nor accepted by the validator because it gets filtered due to be categorized as an already seen transaction).

Expected behavior

✅ = already happening
❌ = not happening

1. The user sends a transaction, which is received by the non-validator node and added to the pool ✅
2. The transaction is propagated to the validator nodes. ✅
3. The validator nodes accept the transaction and add it to their pool. ✅
4. Every node drops the transaction from its pool. ✅
5. The user resends the transaction, which is received by the non-validator node and added to the pool. ✅
6. The same as step 2. ❌
7. The same as step 3. ❌

Nodes execution arguments

--tx-pool-max-size=5
--tx-pool=sequenced
--tx-pool-limit-by-account-percentage=1

Nodes config

Non-validator node

# Network
p2p-host="127.0.0.1"
p2p-port=1232
max-peers=42

rpc-http-enabled=true
rpc-http-api=["ETH","NET","WEB3","IBFT","QBFT","TXPOOL","ADMIN"]

host-whitelist=["*"]
rpc-http-cors-origins=["all"]

rpc-http-host="0.0.0.0"
rpc-http-port=8545

rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=30303

# Mining
miner-enabled=true
miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"

min-gas-price="0"
revert-reason-enabled=true

metrics-category=[ "ETHEREUM", "BLOCKCHAIN","EXECUTORS","JVM","NETWORK","PEERS","PROCESS","KVSTORE_ROCKSDB","KVSTORE_ROCKSDB_STATS","RPC","SYNCHRONIZER", "TRANSACTION_POOL" ]
metrics-enabled=true
metrics-host="0.0.0.0"
metrics-port=9095

Validator node config

# Network
p2p-host="127.0.0.1"
p2p-port=1234
max-peers=42

rpc-http-enabled=true
rpc-http-api=["ETH","NET","WEB3","IBFT","QBFT","TXPOOL"]

host-whitelist=["*"]
rpc-http-cors-origins=["all"]

rpc-http-host="0.0.0.0"
rpc-http-port=8585

rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=30305

# Mining
miner-enabled=true
miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"

min-gas-price="0"
revert-reason-enabled=true

metrics-category=[ "ETHEREUM", "BLOCKCHAIN","EXECUTORS","JVM","NETWORK","PEERS","PROCESS","KVSTORE_ROCKSDB","KVSTORE_ROCKSDB_STATS","RPC","SYNCHRONIZER", "TRANSACTION_POOL" ]
metrics-enabled=true
metrics-host="0.0.0.0"
metrics-port=9097

genesis.json

{
 "config": {
   "muirGlacierBlock": 0,
   "chainId": 44844,
   "contractSizeLimit": 2147483647,
   "qbft": {
     "blockperiodseconds": 1,
     "epochlength": 30000,
     "requesttimeoutseconds": 10
   }
 },
 "nonce": "0x0",
 "timestamp": "0x58ee40ba",
 "gasLimit": "0x5F5E100",
 "difficulty": "0x1",
 "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
 "coinbase": "0x0000000000000000000000000000000000000000",
 "alloc": {
   "fe3b557e8fb62b89f4916b721be55ceb828dbd73": {
     "privateKey": "8f2a55949038a9610f50fb23b5883af3b4ecb3c3bb792cbcefbd1542c692be63",
     "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
     "balance": "0xad78ebc5ac6200000"
   },
   "627306090abaB3A6e1400e9345bC60c78a8BEf57": {
     "privateKey": "c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3",
     "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
     "balance": "90000000000000000000000"
   },
   "f17f52151EbEF6C7334FAD080c5704D77216b732": {
     "privateKey": "ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f",
     "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
     "balance": "90000000000000000000000"
   }
 },
 "extraData": "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f85494792fc5093a85bd8fb52c781aefee7da96d2180cf9414275b2f4cefb4c72f12ca32ce0044578923e1b694fcbc96c1e8a673b7cdc333c6687f07fa2c28befe94b5ec93ab0a6ad8f8c0e404e3225b16cb2ea23a1ec080c0"
}

[nano-adhara]

I added the node execution arguments into the issue. This is using the sequenced node pool

[fab-10]

Hi @nano-adhara, I see where the issue is coming from, Besu has caches to remember txs exchanges with other peers, basically because we want to avoid to resend a tx multiple time to a peer (actually also the p2p protocol state that peers resending txs should be disconnected) and we also want to avoid reprocessing a tx that we have already seen.

Those caches are quite basic a the moment, so they could be improved to better handle scenarios like the one you are reporting, and I could take a look at them, trying to remove from these cache, valid txs that are dropped from the txpool, this needs to be done with care to avoid being exposed to attacks.

As last a question about your network, is it common for a valid tx to stay in the pool for so long that it could be evicted by the timer?

[nano-adhara]

Hello @fab-10,

So as an answer to your last question, is not happening that much, once a month, but the real problem with that is that whenever happens, the only solution is to restart the besu clients.

Ill include Fernando @chookly314 and Coenie @coeniebeyers in the topic to move forward in the conversation.

[non-fungible-nelson]

Hi @nano-adhara - sorry this hasn't moved. I am wondering if this may be an issue that we could work with @matthew1001 as its in QBFT and the sequenced pool. Was there other context needed from the folks you tagged in your comment?

[matthew1001]

I'd be happy to look into this at some point if @fab-10 doesn't get there first. I'm a bit focused on bonsai archive and empty block period for the time being so might not be that soon for me to take a look

[nano-adhara]

No context from the other folks i tagged. Just did so they have all the context as well

[louisrushin]

@matthew1001 Hope you are doing well? I was wondering if you could advise when you would be able to look into the issue above? Kind regards

[fab-10]

sorry for being slow on this one, but I can start working on it, and hope to have something by the next week.

[louisrushin]

Hi @fab-10 , thank you very much, much appreciated! Regards

[louisrushin]

Hi @fab-10 hope you are doing well? Was just wondering if still possible for something this week? Kind regards, Louis

[fab-10]

Hi @louisrushin, still not done, definitely not for this week

[fab-10]

it is on main now, so you can start trying the develop version, before the next release is out

[louisrushin]

Thank you very much!

[nano-adhara]

@fab-10 after testing latest develop (main branch) functionality that includes the PR #7755, we identified that the issue is still not solved. We reviewed the code and execute the tests as described below. The issue looks like is related more on how transactions are being shared between nodes (seenTransactions and transactionsToSend maps on PeerTransactionTracker file)

This is a detailed description on the test we did.

Setup

We used latest main commit and started a setup with two nodes. The network is a private besu network using QBFT protocol using sequenced transaction pool:

• A non-validator node (client node) that doesn’t participate in the consensus protocol.
• One node acting as a validator.

Client node configuration:

# Network
p2p-host="127.0.0.1"
p2p-port=1232
max-peers=42

rpc-http-enabled=true
rpc-http-api=["ETH","NET","WEB3","IBFT","QBFT","TXPOOL","ADMIN"]

host-whitelist=["*"]
rpc-http-cors-origins=["all"]

rpc-http-host="0.0.0.0"
rpc-http-port=8545

rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=30303

# Mining
miner-enabled=true
miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"

min-gas-price="0"
revert-reason-enabled=true

metrics-category=[ "ETHEREUM", "BLOCKCHAIN","EXECUTORS","JVM","NETWORK","PEERS","PROCESS","KVSTORE_ROCKSDB","KVSTORE_ROCKSDB_STATS","RPC","SYNCHRONIZER", "TRANSACTION_POOL" ]
metrics-enabled=true
metrics-host="0.0.0.0"
metrics-port=9095

Validator node configuration:

# Network
p2p-host="127.0.0.1"
p2p-port=1234
max-peers=42

rpc-http-enabled=true
rpc-http-api=["ETH","NET","WEB3","IBFT","QBFT","TXPOOL"]

host-whitelist=["*"]
rpc-http-cors-origins=["all"]

rpc-http-host="0.0.0.0"
rpc-http-port=8585

rpc-ws-enabled=true
rpc-ws-host="0.0.0.0"
rpc-ws-port=30305

# Mining
miner-enabled=true
miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"

min-gas-price="0"
revert-reason-enabled=true

metrics-category=[ "ETHEREUM", "BLOCKCHAIN","EXECUTORS","JVM","NETWORK","PEERS","PROCESS","KVSTORE_ROCKSDB","KVSTORE_ROCKSDB_STATS","RPC","SYNCHRONIZER", "TRANSACTION_POOL" ]
metrics-enabled=true
metrics-host="0.0.0.0"
metrics-port=9100

Genesis.json file:

{
  "config": {
    "muirGlacierBlock": 0,
    "chainId": 44844,
    "contractSizeLimit": 2147483647,
    "qbft": {
      "blockperiodseconds": 1,
      "epochlength": 30000,
      "requesttimeoutseconds": 10
    }
  },
  "nonce": "0x0",
  "timestamp": "0x58ee40ba",
  "gasLimit": "0x5F5E100",
  "difficulty": "0x1",
  "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "alloc": {
    "fe3b557e8fb62b89f4916b721be55ceb828dbd73": {
      "privateKey": "8f2a55949038a9610f50fb23b5883af3b4ecb3c3bb792cbcefbd1542c692be63",
      "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance": "0xad78ebc5ac6200000"
    },
    "627306090abaB3A6e1400e9345bC60c78a8BEf57": {
      "privateKey": "c87509a1c067bbde78beb793e6fa76530b6382a4c0241e5e4a9ec0a0f44dc0d3",
      "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance": "90000000000000000000000"
    },
    "f17f52151EbEF6C7334FAD080c5704D77216b732": {
      "privateKey": "ae6ae8e5ccbfb04590405997ee2d52d2b330726137b875053c36d94e974d162f",
      "comment": "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance": "90000000000000000000000"
    }
  },
  "extraData": "0xf83aa00000000000000000000000000000000000000000000000000000000000000000d594792fc5093a85bd8fb52c781aefee7da96d2180cfc080c0"
}

Client node program arguments:

--config-file
<path_to_above_client_node_config>
--data-path
<path_to_data>
--genesis-file
<path_to_above_genesis>
--tx-pool-max-size=5
--tx-pool=sequenced
--tx-pool-retention-hours=1
--tx-pool-limit-by-account-percentage=1

Validator node program arguments:

--config-file
<path_to_above_validator_node_config>
--data-path
<path_to_data>
--genesis-file
<path_to_above_genesis>
--tx-pool-max-size=5
--tx-pool=sequenced
--tx-pool-retention-hours=1
--tx-pool-limit-by-account-percentage=1
--Xdns-enabled=true
--Xdns-update-enabled=true
--bootnodes=enode://f9c38091886901dffe3f574f02b1d2e3381135fb105d6a5f4a86e941809e0277fa0329c888dbffeb3c84665e9f508368f346ac26dbbc3fb39d7ef9a516277ae6@127.0.0.1:1232

Test

1. We send a signed transaction to the client node. This transaction has a nonce=2, so a nonce gap its already in place, and transaction won't be mined.

Input: 
curl -X POST http://localhost:8545 -d '{"jsonrpc":"2.0","method":"eth_sendRawTransaction", "params":["0xf86302018301676094655b6e7934ffb546119a509e0d53ddc05716c1e1038083015e7ba0efeafc4d0aa31bf247df546ef05f0342b25f57aa73dc0d5001d48ce976af35afa00457410b72b8df38455668ea4e052cf576bc5f723191757ef613ac473093b05f"], "id": "1"}'

Output:
{"jsonrpc":"2.0","id":"1","result":"0x34ecab1fb036a364ac602ecfd4580c40129f56641987eb73413b7bbc8823ef71"}%                                                                                                                        

2. We validate that tx is in both pools.

Client node pool:

Input:
curl -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8545

Output:
{"jsonrpc":"2.0","id":1,"result":[{"hash":"0x34ecab1fb036a364ac602ecfd4580c40129f56641987eb73413b7bbc8823ef71","isReceivedFromLocalSource":true,"addedToPoolAt":"2024-10-22T13:57:01.736Z"}]}%

Validator node pool:

Input:
url -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8585

Output:
{"jsonrpc":"2.0","id":1,"result":[{"hash":"0x34ecab1fb036a364ac602ecfd4580c40129f56641987eb73413b7bbc8823ef71","isReceivedFromLocalSource":false,"addedToPoolAt":"2024-10-22T13:57:01.736Z"}]}%

NOTE: the isReceivedFromLocalSource confirms that transaction was shared from client peer.

3. We wait an hour so eviction policy drops the transaction from the pool in both nodes:

Client node pool:

Input:
curl -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8545

Output:
{"jsonrpc":"2.0","id":1,"result":[]}%  

Validator node pool:

Input:
url -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8585

Output:
{"jsonrpc":"2.0","id":1,"result":[]}%  

4. We send the same transaction, as it was already dropped from both pools to the client node:

Input: 
curl -X POST http://localhost:8545 -d '{"jsonrpc":"2.0","method":"eth_sendRawTransaction", "params":["0xf86302018301676094655b6e7934ffb546119a509e0d53ddc05716c1e1038083015e7ba0efeafc4d0aa31bf247df546ef05f0342b25f57aa73dc0d5001d48ce976af35afa00457410b72b8df38455668ea4e052cf576bc5f723191757ef613ac473093b05f"], "id": "1"}'

Output:
{"jsonrpc":"2.0","id":"1","result":"0x34ecab1fb036a364ac602ecfd4580c40129f56641987eb73413b7bbc8823ef71"}%                                                                                                                        

5. The transaction is in the client nodes pool, but not in the validator pool:

Client node pool:

Input:
curl -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8545

Output:
{"jsonrpc":"2.0","id":1,"result":[{"hash":"0x34ecab1fb036a364ac602ecfd4580c40129f56641987eb73413b7bbc8823ef71","isReceivedFromLocalSource":true,"addedToPoolAt":"2024-10-22T15:09:19.198Z"}]}% 

Validator node pool:

Input:
url -X POST --data '{"jsonrpc":"2.0","method":"txpool_besuTransactions","params":[],"id":1}' http://127.0.0.1:8585

Output:
{"jsonrpc":"2.0","id":1,"result":[]}%  

Expected outcome

Transaction should be shared to the validator node as the transaction was dropped due to a time eviction, but client node does not send the transaction, so the validator node is not aware of this transaction. The transaction cannot be executed so a nonce gap is perpetualy created for that address. This problem is not solved until all nodes are restarted with the current version

[petermunnings]

@fab-10 @matthew1001 what is the process now? This ticket is closed, but the problem is not solved as demonstrated by @nano-adhara above. Can you give us some guidance on how we should proceed. We have a live environment with this problem in which is getting a lot of attention.

[fab-10]

@petermunnings I am reopening the ticket, and will take a look at your report to reproduce it and fix in the next days, if in the meantime if you already have in idea of the solution, you can propose a PR for review.

[coeniebeyers]

Hi @fab-10 just following up, any progress on this?

[fab-10]

thanks to @nano-adhara for the detailed steps to reproduce the issue, I was able to reproduce the issue and submit a fix in PR #7858.
Could you try out that branch and confirm that it works for you?

One important note on an edge case: the eviction task is triggered every minute, so depending on the start time of each node, there could be 1 minute delay between the eviction of a tx between 2 nodes in the worst case, this means that if you try to resubmit the tx just after it has evicted on one node, it is possible that the same tx isn't still evicted on every node, so I suggest to wait 1 minute to resend the tx after it has been evicted.

[nano-adhara]

let me review the branch and let you know the outcome. thank you @fab-10

[nano-adhara]

hello @fab-10. I was able to test the scenario and now its working 👍.

i was considering the edge case you provided. i was able to test it aswell and i have some doubts that id like to know the answer on how to proceed when we apply this besu patch.

• For long-living networks, i guess this should not be a problem as the eviction is a one time minute compared to default thirteen hours default retencion policy. Still, as this is something that can happen in a very edgy scenario. Is there something that can be done to force manually (or via API) an eviction check?
• In the scenario that transaction was evicted from client node, but not from validator node (because we are in that 1minute gap). If we provide the transaction again, there are two transactions that look like the same, but are different for each node. The client node will hold the new one, and the validator node is still holding the old one until evicted. Are they the same transaction?
  • Considering the scenario above. Imagine that i send another transaction in that 1 minute gap that fixes the nonce gap. What will happen is that the validator node inmediately mine both transactions from the pool. Is that right? what will happen with the transaction in the client node?
• Lastly, even for this edge cases, i can assume that for this patch, even with the time eviction gaps, the important thing is that if BOTH transactions ends up evicted in both pools, the moment we resend the transaction that will be shared again inthe two nodes, is this assumption right? This is important as we are looking forward to a solution that does not require to restart the full network

[fab-10]

hello @fab-10. I was able to test the scenario and now its working 👍.

great, I will merge the PR then

* For long-living networks, i guess this should not be a problem as the eviction is a one time minute compared to default thirteen hours default retencion policy. Still, as this is something that can happen in a very edgy scenario. Is there something that can be done to force manually (or via API) an eviction check?

at the moment there is way to trigger the eviction manually.

* In the scenario that transaction was evicted from client node, but not from validator node (because we are in that 1minute gap). If we provide the transaction again, there are two transactions that look like the same, but are different for each node. The client node will hold the new one, and the validator node is still holding the old one until evicted. Are they the same transaction?

yes it will be the same tx (same hash), otherwise in case any of the tx field has changed then it will be a different one, and it could replace the previous one, according to the transaction replacement rule.

  * Considering the scenario above. Imagine that i send another transaction in that 1 minute gap that fixes the nonce gap. What will happen is that the validator node inmediately mine both transactions from the pool. Is that right? what will happen with the transaction in the client node?

Yes the validator could mine both, and the one in the node will be removed as soon as the new block will be imported.

* Lastly, even for this edge cases, i can assume that for this patch, even with the time eviction gaps, the important thing is that if BOTH transactions ends up evicted in both pools, the moment we resend the transaction that will be shared again inthe two nodes, is this assumption right? This is important as we are looking forward to a solution that **does not require to restart the full network**

Correct, with this fix, if the tx has been evicted by all pools, then after re-sending it will be added again to every pool