Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve flat trace generation performance #6472

Merged
merged 17 commits into from
Feb 19, 2024
Merged

Improve flat trace generation performance #6472

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
c89e4db
Improve flat trace generation performance
ahamlat Jan 26, 2024
45448b2
Use opCode number for comparisons instead of opCode name
ahamlat Jan 29, 2024
395a89a
Test with primitives to avoid unboxing overhead
ahamlat Jan 29, 2024
663a065
Merge branch 'main' into improve-trace-block-on-subcalls
ahamlat Jan 30, 2024
da0ce79
Manage the case where opCode is null
ahamlat Jan 30, 2024
5b9de6e
Merge remote-tracking branch 'origin/improve-trace-block-on-subcalls'…
ahamlat Jan 30, 2024
8f78a07
Refactoring
ahamlat Jan 30, 2024
ce36c61
Merge branch 'main' into improve-trace-block-on-subcalls
ahamlat Jan 30, 2024
0de955a
Add a changelog
ahamlat Jan 30, 2024
9d058a3
Merge remote-tracking branch 'origin/improve-trace-block-on-subcalls'…
ahamlat Jan 30, 2024
4a8fa04
Merge branch 'main' into improve-trace-block-on-subcalls
ahamlat Feb 13, 2024
dd553c3
Merge branch 'main' into improve-trace-block-on-subcalls
ahamlat Feb 15, 2024
c0cc9fd
Add the OpCode enum in the EVM to store all the opcodes.
ahamlat Feb 15, 2024
ec123a6
Spotless
ahamlat Feb 15, 2024
b8e548f
Add license header
ahamlat Feb 15, 2024
51b06f8
Revert opcodes enum and add constants in RevertOperation and ReturnOp…
ahamlat Feb 15, 2024
35e246d
Merge branch 'main' into improve-trace-block-on-subcalls
ahamlat Feb 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@
- Log blob count when importing a block via Engine API [#6466](https://github.com/hyperledger/besu/pull/6466)
- Introduce `--Xbonsai-limit-trie-logs-enabled` experimental feature which by default will only retain the latest 512 trie logs, saving about 3GB per week in database growth [#5390](https://github.com/hyperledger/besu/issues/5390)
- Introduce `besu storage x-trie-log prune` experimental offline subcommand which will prune all redundant trie logs except the latest 512 [#6303](https://github.com/hyperledger/besu/pull/6303)
- Improve flat trace generation performance [#6472](https://github.com/hyperledger/besu/pull/6472)
- SNAP and CHECKPOINT sync - early access flag removed so now simply SNAP and CHECKPOINT [#6405](https://github.com/hyperledger/besu/pull/6405)
- X_SNAP and X_CHECKPOINT are marked for deprecation and will be removed in 24.4.0
- X_SNAP and X_CHECKPOINT are marked for deprecation and will be removed in 24.4.0
- Github Actions based build.
- Introduce caching mechanism to optimize Keccak hash calculations for account storage slots during block processing [#6452](https://github.com/hyperledger/besu/pull/6452)
- Added configuration options for `pragueTime` to genesis file for Prague fork development [#6473](https://github.com/hyperledger/besu/pull/6473)
Expand Down
18 changes: 13 additions & 5 deletions ...perledger/besu/ethereum/api/jsonrpc/internal/results/tracing/flat/FlatTraceGenerator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -530,11 +530,19 @@ private static FlatTrace.Context handleCallDataLoad(

private static boolean hasRevertInSubCall(
final TransactionTrace transactionTrace, final TraceFrame callFrame) {
return transactionTrace.getTraceFrames().stream()
.filter(traceFrame -> !traceFrame.equals(callFrame))
.takeWhile(traceFrame -> !traceFrame.getOpcode().equals("RETURN"))
.filter(traceFrame -> traceFrame.getOpcode().equals("REVERT"))
.anyMatch(traceFrame -> traceFrame.getDepth() == callFrame.getDepth());
for (int i = 0; i < transactionTrace.getTraceFrames().size(); i++) {
if (i + 1 < transactionTrace.getTraceFrames().size()) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kind of a random question (Side note: I dont really have Java exp) but what is the point of this If condition? Wouldn't starting the loop at 1 and running it until size() - 1 have the same result? (Or if iterating in reverse has no side-effects, even starting at size() - 1 and running while its > 0)

final TraceFrame next = transactionTrace.getTraceFrames().get(i + 1);
if (next.getDepth() == callFrame.getDepth()) {
if (next.getOpcodeNumber() == 0xFD) { // REVERT opCode
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth making AbstractOperation.getOpcode() static and doing the comparison to that to avoid creating another range of enums.

Copy link
Contributor Author

@ahamlat ahamlat Feb 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This commit is outdated. Making AbstractOperation.getOpcode() static means that we need to make opcode static in AbstractOperation which is not the case, opcode depends on the implementation.
What suggested by Danno is very similar in some way to what you suggest, add a static constants directly in the implementation, that is stored in the opcode field.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding an OPCODE constant to RevertOperation and ReturnOperation so we can reduce magic constants in the code.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion, I was thinking actually of an Enum OpCode to store all the opcodes and use only the Enum in the EVM and outside, like Opcode.REVERT.getOpcodeNumber() for 0xFD
public enum Opcode {
STOP(0x00, "STOP"),
ADD(0x01, "ADD"),
MUL(0x02, "MUL"),
...
}
In terms of performance, I guess JIT will replace by the opCodeNumber or the opcode name at runtime

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shemnon I've added the OpCode enum within the EVM and would like to use it across all EVM opcode operations. An implementation example has been provided for ADDMOD. Does this approach work for you ?

Copy link
Contributor

@shemnon shemnon Feb 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not what I meant. Such an OpCode enum breaks extensibility to downstream users use. Please revert.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I intended was something like this, where a constant is added to the Operation classes, not an entirely different enum.

besu/evm/src/main/java/org/hyperledger/besu/evm/operation/BlobHashOperation.java

Line 37 in c89e4db

public static final int OPCODE = 0x49;

Copy link
Contributor Author

@ahamlat ahamlat Feb 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually understood your suggestion but wanted to suggest an implementation that would work for similar use cases in the future, but it looks like it wasn't a good idea. I'm going to revert the last changes.

return true;
} else if (next.getOpcodeNumber() == 0xF3) { // RETURN opCode
return false;
}
}
}
}
return false;
}

private static String calculateCallingAddress(final FlatTrace.Context lastContext) {
Expand Down
1 change: 1 addition & 0 deletions ...org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByHashTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ public void shouldReturnCorrectResponse() {
new TraceFrame(
12,
Optional.of("NONE"),
Integer.MAX_VALUE,
45L,
OptionalLong.of(56L),
0L,
Expand Down
1 change: 1 addition & 0 deletions ...g/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByNumberTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ public void shouldReturnCorrectResponse() {
new TraceFrame(
12,
Optional.of("NONE"),
Integer.MAX_VALUE,
45L,
OptionalLong.of(56L),
0L,
Expand Down
1 change: 1 addition & 0 deletions .../java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ public void shouldReturnCorrectResponse() {
new TraceFrame(
12,
Optional.of("NONE"),
Integer.MAX_VALUE,
45L,
OptionalLong.of(56L),
0L,
Expand Down
2 changes: 2 additions & 0 deletions ...org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceTransactionTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,7 @@ public void shouldTraceTheTransactionUsingTheTransactionTracer() {
new TraceFrame(
12,
Optional.of("NONE"),
Integer.MAX_VALUE,
45L,
OptionalLong.of(56L),
0L,
Expand Down Expand Up @@ -183,6 +184,7 @@ public void shouldNotTraceTheTransactionIfNotFound() {
new TraceFrame(
12,
Optional.of("NONE"),
Integer.MAX_VALUE,
45L,
OptionalLong.of(56L),
0L,
Expand Down
7 changes: 7 additions & 0 deletions ethereum/core/src/main/java/org/hyperledger/besu/ethereum/debug/TraceFrame.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ public class TraceFrame {

private final int pc;
private final Optional<String> opcode;
private final int opcodeNumber;
private final long gasRemaining;
private final OptionalLong gasCost;
private final long gasRefund;
Expand Down Expand Up @@ -63,6 +64,7 @@ public class TraceFrame {
public TraceFrame(
final int pc,
final Optional<String> opcode,
final int opcodeNumber,
final long gasRemaining,
final OptionalLong gasCost,
final long gasRefund,
Expand All @@ -86,6 +88,7 @@ public TraceFrame(
final Optional<StorageEntry> maybeUpdatedStorage) {
this.pc = pc;
this.opcode = opcode;
this.opcodeNumber = opcodeNumber;
this.gasRemaining = gasRemaining;
this.gasCost = gasCost;
this.gasRefund = gasRefund;
Expand Down Expand Up @@ -118,6 +121,10 @@ public String getOpcode() {
return opcode.orElse("");
}

public int getOpcodeNumber() {
return opcodeNumber;
}

public long getGasRemaining() {
return gasRemaining;
}
Expand Down
4 changes: 4 additions & 0 deletions ethereum/core/src/main/java/org/hyperledger/besu/ethereum/vm/DebugOperationTracer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,7 @@ public void tracePreExecution(final MessageFrame frame) {
public void tracePostExecution(final MessageFrame frame, final OperationResult operationResult) {
final Operation currentOperation = frame.getCurrentOperation();
final String opcode = currentOperation.getName();
final int opcodeNumber = (opcode != null) ? currentOperation.getOpcode() : Integer.MAX_VALUE;
final WorldUpdater worldUpdater = frame.getWorldUpdater();
final Bytes outputData = frame.getOutputData();
final Optional<Bytes[]> memory = captureMemory(frame);
Expand All @@ -82,6 +83,7 @@ public void tracePostExecution(final MessageFrame frame, final OperationResult o
new TraceFrame(
pc,
Optional.of(opcode),
opcodeNumber,
gasRemaining,
operationResult.getGasCost() == 0
? OptionalLong.empty()
Expand Down Expand Up @@ -117,6 +119,7 @@ public void tracePrecompileCall(
new TraceFrame(
frame.getPC(),
Optional.empty(),
Integer.MAX_VALUE,
frame.getRemainingGas(),
OptionalLong.empty(),
frame.getGasRefund(),
Expand Down Expand Up @@ -163,6 +166,7 @@ public void traceAccountCreationResult(
new TraceFrame(
frame.getPC(),
Optional.empty(),
Integer.MAX_VALUE,
frame.getRemainingGas(),
OptionalLong.empty(),
frame.getGasRefund(),
Expand Down
Loading