Ability to incrementally build a block

[pipermerriam]

What is wrong?

This PR added optimizations to the process of importing blocks #565

The underlying changes to apply_transaction are causing problems in eth-tester. Now, there is no way to leave the chain in a half state with some transactions applied to the HEAD block. This removes the ability to incrementally apply transactions to a block without digging into internals.

How can it be fixed

Not sure but:

• Need a way for import_block to delay building the transaction trie until the end of the block.
• Need a way to do what the previous apply_transaction functionality did, executing the transaction and fully updating the pending block with the latest state.

[pipermerriam]

cc @gsalgado (just an FYI)

[pipermerriam]

link: ethereum/eth-tester#81

[pipermerriam]

Trying to reason about this....

When doing a full block import we need to re-execute the block and verify that the final resulting block is identical to the input block. Optimization here includes skipping rebuilding the receipt and transaction tries.

When incrementally building a block we need to execute the transaction and build the transaction and receipt tries.

This seems like we should be able to fix this as follows by separating concerns to the appropriate level.

• Chain.apply_transaction()
  • delegates to VM.apply_transaction()
• VM.apply_transaction()
  • delegates to State.apply_transaction()
  • updates local VM.block.header.state_root
  • updates local VM.block.transactions
  • updates local VM.block.header.transactions_root
  • updates local VM.block.header.receipts_root
  • updates local VM.block.header.bloom
  • updates local VM.block.header.gas_used
• State.apply_transaction()
  • executes the transaction
  • returns the computations, receipt, state_root (modified from previously returning block.

This isn't exactly right, but the idea is to move more of the block stuff up to the VM level so that import_block can strictly use the State level functions to shortcut the overhead of rebuilding each of these block/header values after each transaction is executed.

[gsalgado]

When doing a full block import we need to re-execute the block and verify that the final resulting block is identical to the input block. Optimization here includes skipping rebuilding the receipt and transaction tries.

I don't think this is an optimization, really, but maybe my understanding is not correct. More below...

I'm not familiar with eth-tester APIs but if we keep the current block's transactions/receipts in memory, we should only need to generate the tx/receipt tries when finalizing the block, no? I mean, if the transaction_root and receipt_root are only needed to lookup the transactions/receipts from the DB, we shouldn't need to generate them if the txs/receipts are available from somewhere else, but even if there's a need for transaction_root and receipt_root to be exposed before a block is finalized, we could still calculate them on the fly when needed.

[pipermerriam]

I think you're probably right. I think what we're really talking about here is shuffling around what state we store, where we store it, and what parts of the architecture are responsible for what