A potential fix to raw typed tx encoding

[MarekM25]

Fixes #8256

Changes

Not tested, not merge, requires adding tests, draft for now

Types of changes

What types of changes does your code introduce?

• Bugfix (a non-breaking change that fixes an issue)
• New feature (a non-breaking change that adds functionality)
• Breaking change (a change that causes existing functionality not to work as expected)
• Optimization
• Refactoring
• Documentation update
• Build-related changes
• Other: Description

Testing

Requires testing

• Yes
• No

If yes, did you write tests?

• Yes
• No

Notes on testing

Optional. Remove if not applicable.

Documentation

Requires documentation update

• Yes
• No

If yes, link the PR to the docs update or the issue with the details labeled docs. Remove if not applicable.

Requires explanation in Release Notes

• Yes
• No

If yes, fill in the details here. Remove if not applicable.

Remarks

Optional. Remove if not applicable.

[mralj]

Tested against the rbuilder, this fixes the issue 🥳

[mralj]

Unfortunately, after more testing, I see that this works for all TX types, except blobs. The RLP decoding fails with: unexpected string.
Here is an example of response:

0x03f9021b0182adc384f59236008502e0b6a200832dc6c09406a9ab27c7e2255df1815e6cc0168d7755feb19a80b90184648885fb00000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009a5cc6e3a3325cdc19fc76926cc9666c80139c0900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0843b9aca00e1a00144d0f9e43bd2e6a911e8d540e92feaed5d079c941bc698bf2574d44d79403601a08e3f46fb0756be06e9fcf3785f707d11f89e4542b38f1925a654132daa87b37aa0291ec8005a93b113e464fa4e46bef99cd8aaa6ecb1d862b1dc819ce5f654ab4a

Here is the tx in question on etherscan

[flcl42]

Unfortunately, after more testing, I see that this works for all TX types, except blobs. The RLP decoding fails with: unexpected string. Here is an example of response:

0x03f9021b0182adc384f59236008502e0b6a200832dc6c09406a9ab27c7e2255df1815e6cc0168d7755feb19a80b90184648885fb00000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009a5cc6e3a3325cdc19fc76926cc9666c80139c0900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0843b9aca00e1a00144d0f9e43bd2e6a911e8d540e92feaed5d079c941bc698bf2574d44d79403601a08e3f46fb0756be06e9fcf3785f707d11f89e4542b38f1925a654132daa87b37aa0291ec8005a93b113e464fa4e46bef99cd8aaa6ecb1d862b1dc819ce5f654ab4a

Here is the tx in question on etherscan

The tool does not support decoding of such tx type. Btw how do you use this method? Just so you know eth_sendRawTransaction may receive such transaction in form with blobs, but eth_getRawTransactionByHash will return without them

[mralj]

Unfortunately, after more testing, I see that this works for all TX types, except blobs. The RLP decoding fails with: unexpected string. Here is an example of response:

0x03f9021b0182adc384f59236008502e0b6a200832dc6c09406a9ab27c7e2255df1815e6cc0168d7755feb19a80b90184648885fb00000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000160000000000000000000000000000000000000000000000000000000000000010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000009a5cc6e3a3325cdc19fc76926cc9666c80139c0900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008e1900000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000c0843b9aca00e1a00144d0f9e43bd2e6a911e8d540e92feaed5d079c941bc698bf2574d44d79403601a08e3f46fb0756be06e9fcf3785f707d11f89e4542b38f1925a654132daa87b37aa0291ec8005a93b113e464fa4e46bef99cd8aaa6ecb1d862b1dc819ce5f654ab4a

Here is the tx in question on etherscan

The tool does not support decoding of such tx type. Btw how do you use this method? Just so you know eth_sendRawTransaction may receive such transaction in form with blobs, but eth_sendRawTransaction will return without them

Hey, I'm using this from rbuilder, here is how this is decoded.
The above-pasted bytes are raw bytes I log after receiving them from NMC/eth_sendRawTransaction.
The code fails even before blobs are extracted, i.e. here

[flcl42]

To send tx using Nethermind and its eth_sendRawTransaction API you firstly need to encode it, with blobs in it. How do you encode it?

[mralj]

To send tx using Nethermind and its eth_sendRawTransaction API you firstly need to encode it, with blobs in it. How do you encode it?

Hey, I think there has been some misunderstanding here, I apologies if I haven't described the issue well in #8256, I've updated the description with section "rbuilder repro steps", here is a copy of it:

rbuilder repro steps

1. rbuilder connects to NMC via WS/IPC and subscribes to mainnet pending transactions (eth_subscribe/newPendingTransactions)
2. When new TX hash is received by rbuilder it calls eth_getRawTransactionByHash
3. After receiving raw TX (bytes), rbuilder will fail to decode these bytes: here

So, it's not that I'm calling eth_sendRawTransaction, I'm just receiving real mainnet transactions on rbuilder via connection to NMC.

[flcl42]

To send tx using Nethermind and its eth_sendRawTransaction API you firstly need to encode it, with blobs in it. How do you encode it?

Hey, I think there has been some misunderstanding here, I apologies if I haven't described the issue well in #8256, I've updated the description with section "rbuilder repro steps", here is a copy of it:

rbuilder repro steps

1. `rbuilder` connects to NMC via WS/IPC and subscribes to mainnet pending transactions (`eth_subscribe/newPendingTransactions`)

2. When new TX hash is received by `rbuilder` it calls `eth_getRawTransactionByHash`

3. After receiving raw TX (bytes), `rbuilder` will fail to decode these bytes: [here](https://github.com/flashbots/rbuilder/blob/70e396298cb4961b863c093be603fdfbbd4f7e1e/crates/rbuilder/src/primitives/mod.rs#L619)

So, it's not that I'm calling eth_sendRawTransaction, I'm just receiving real mainnet transactions on rbuilder via connection to NMC.

Thank you! Yeah it will not work, because eth_getRawTransactionByHash does not return blobs with tx. We return payload form(=no blobs only blob hashes), because we don't know if requested tx is from a block(so no blobs available) or from tx pool(where we can still find blobs). But let's think what we can do. Maybe a subscription that returns txs instead of hashes? Maybe we can pass argument to search only in tx pool and return txs in pool form?
@MarekM25 @LukaszRozmej @marcindsobczak wdyt?

[LukaszRozmej]

Ok so Reth probably returns eth_getRawTransactionByHash with blobs if from mempool?

@flcl42 can you check how Geth and Reth handle eth_getRawTransactionByHash for blob transactions?

If it is a standard thing - change Nethermind, if it is Reth invention, maybe we could include optional parameter, that rbuilder could always add and other clients would just ignore?

[mralj]

Ok so Reth probably returns eth_getRawTransactionByHash with blobs if from mempool?

@flcl42 can you check how Geth and Reth handle eth_getRawTransactionByHash for blob transactions?

If it is a standard thing - change Nethermind, if it is Reth invention, maybe we could include optional parameter, that rbuilder could always add and other clients would just ignore?

As far as I can see it's a standard thing @LukaszRozmej @flcl42

geth

1. The call starts here
2. Which will try to fetch TX from pool: here
3. Which calls this and eventually Get in BlobPool, which, unless I'm misunderstanding code, is attaching blobs
4. Here is how the geth handles encdoding/decoding of blobs: link

reth

1. The call starts here
2. Which will call this and subsequently this
3. All of which finally leads us to to_pooled_transaction call which loads and attaches blobs.

There is interesting observation made in reth codebase regarding this call:
// Note: this is mostly used to fetch pooled transactions so we check the pool first this is opposite of geth which will firstly check finalised transactions - and if I'm remembering correctly this is how NMC works, but I thing reth team is correct here in their assumption :)