Add Block Benchmark Suite #1955

RodrigoVillar · 2025-03-05T18:43:07Z

This PR adds a benchmarking test suite for benchmarking the processor execution time of a list of blocks. This test suite can be used for benchmarking both the HyperSDK itself and any custom VM which wants to benchmark its own block execution time.

As part of the test suite, users are required to pass in a TxListGenerator which when called, returns a list of transactions that will be set to the transaction list of a block. An implementation of TxListGenerator can be used to benchmark the HyperSDK's pessimistic concurrency model against various state access distributions (i.e. TXs with no conflicting state accesses, TXs with all conflicting state accesses, TXs with state accesses under a Zipf distribution).

An example of how benchmarking would look like for the HyperSDK (with empty blocks) is as follows:

func BenchmarkProcessorExecuteEmptyBlocks(b *testing.B) {
	test := &chaintest.BlockBenchmark{
		MetadataManager: metadata.NewDefaultManager(),
		BalanceHandler:  &mockBalanceHandler{},
		AuthVM: &chaintest.TestAuthVM{
			GetAuthBatchVerifierF: func(authTypeID uint8, cores int, count int) (chain.AuthBatchVerifier, bool) {
				bv, ok := auth.Engines()[authTypeID]
				if !ok {
					return nil, false
				}
				return bv.GetBatchVerifier(cores, count), ok
			},
			Log: logging.NoLog{},
		},
		RuleFactory:      &genesis.ImmutableRuleFactory{Rules: genesis.NewDefaultRules()},
		TxListGenerator:  chaintest.NoopTxListGenerator{},
		Config:           chain.NewDefaultConfig(),
		NumOfBlocks:      10_000,
		NumOfTxsPerBlock: 0,
	}
	test.Run(context.Background(), b)
}

aaronbuchwald · 2025-03-05T22:03:55Z

chain/chaintest/block.go

+	switch numOfTxsPerBlock {
+	case 0:
+		timestampOffset = rules.GetMinEmptyBlockGap()
+	default:
+		timestampOffset = rules.GetMinBlockGap()
+	}


Can we handle this with a single if block instead of a switch?

Not identical, but this is a case of "unnecessary else" - https://github.com/uber-go/guide/blob/master/style.md#unnecessary-else

aaronbuchwald · 2025-03-05T22:17:16Z

chain/chaintest/block.go

+	factories := []chain.AuthFactory{}
+	customAllocs := []*genesis.CustomAllocation{}
+	for range test.NumOfTxsPerBlock {
+		pk, err := ed25519.GeneratePrivateKey()
+		r.NoError(err)
+
+		factory := auth.NewED25519Factory(pk)
+		factories = append(factories, factory)
+
+		customAllocs = append(customAllocs, &genesis.CustomAllocation{
+			Address: factory.Address(),
+			Balance: uint64(allocBalance),
+		})
+	}


Should we leave the auth factories behind the generator interface? This is a VM-level detail that will have a non-trivial performance impact.

aaronbuchwald · 2025-03-05T22:18:19Z

chain/chaintest/block.go

+		tsv := ts.NewView(state.CompletePermissions, parentView, 0)
+		if err := tsv.Insert(ctx, chain.HeightKey(metadataManager.HeightPrefix()), binary.BigEndian.AppendUint64(nil, height)); err != nil {
+			return nil, err
+		}
+		if err := tsv.Insert(ctx, chain.TimestampKey(metadataManager.TimestampPrefix()), binary.BigEndian.AppendUint64(nil, uint64(timestamp))); err != nil {
+			return nil, err
+		}
+		if err := tsv.Insert(ctx, chain.FeeKey(metadataManager.FeePrefix()), feeManager.Bytes()); err != nil {
+			return nil, err
+		}
+		tsv.Commit()
+
+		root, err := parentView.GetMerkleRoot(ctx)
+		if err != nil {
+			return nil, err
+		}
+
+		parentView, err = parentView.NewView(ctx, merkledb.ViewChanges{
+			MapOps:       ts.ChangedKeys(),
+			ConsumeBytes: true,
+		})
+		if err != nil {
+			return nil, err
+		}
+
+		statelessBlock, err := chain.NewStatelessBlock(
+			parentID,
+			timestamp,
+			height,
+			txs,
+			root,
+			nil,
+		)
+		if err != nil {
+			return nil, err
+		}
+		parentID = statelessBlock.GetID()
+
+		blk := chain.NewExecutionBlock(statelessBlock)
+		executionBlocks[i] = blk
+	}
+	return executionBlocks, nil


This code is getting duplicated across a growing number of places - can we make this more re-usable?

For DSMR - we need the ability to assemble a block given a parent state / block, metadata for the next block (assigned from DSMR wrapper block), and a set of transactions that should be included. Maybe we can implement an Assembler and use that functionality here

aaronbuchwald · 2025-03-05T22:19:12Z

chain/chaintest/block.go

+	var view merkledb.View
+	view = db
+	b.ResetTimer()
+	for i := 0; i < b.N; i++ {
+		for j := 0; j < int(test.NumOfBlocks); j++ {
+			outputBlock, err := processor.Execute(
+				ctx,
+				view,
+				blocks[j],
+				false,
+			)
+			r.NoError(err)
+			view = outputBlock.View
+		}
+		view = db
+	}
+}


Could we add a comment explaining the handling of views here?

aaronbuchwald · 2025-03-05T22:20:02Z

chain/chaintest/block.go

+				ctx,
+				view,
+				blocks[j],
+				false,


Why use false here? In normal operation, this will be set to true and we will need to check the expiry window

Changed to true.

aaronbuchwald · 2025-03-05T22:38:07Z

chain/chaintest/block.go

+	switch test.AuthVerificationCores {
+	case 0, 1:
+		processorWorkers = workers.NewSerial()
+	default:
+		processorWorkers = workers.NewParallel(
+			test.AuthVerificationCores,
+			numOfAuthVerificationJobs,
+		)
+	}


Can we use a simple if condition instead of a switch since there are only two options?

aaronbuchwald · 2025-03-05T22:38:42Z

chain/chaintest/block.go

+		customAllocs = append(customAllocs, &genesis.CustomAllocation{
+			Address: factory.Address(),
+			Balance: uint64(allocBalance),
+		})
+	}


Hmm - this limits the usability of this package to VMs that use the default genesis / CustomAllocation types.

It seems like the caller should define their application logic including the genesis, transactions (auth/actions), and pass in the components they would have control over as opposed to making this a responsibility of the benchmark tool

RodrigoVillar · 2025-03-06T17:16:03Z

Regarding #1955 (comment) and #1955 (comment), I've decided to switch from users providing a tx-based helper (TxListGenerator) to providing instead a block-based helper (BlockBenchmarkHelper). BlockBenchmarkHelper still produces the TX lists required for GenerateExecutionBlocks() but in addition, it has the method GenerateGenesis which produces the genesis that the benchmark tests will use to generate the genesis commit. I found GenerateGenesis to be an improvement for the following reasons:

Since GenerateGenesis returns an interface, we have that the tests are more VM agnostic (i.e. no longer using genesis.DefaultGenesis)
In GenerateGenesis, the test user can also instantiate any factories required for GenerateTxList, instead of having the benchmark tests produce the factories on their behalf.

tsachiherman · 2025-03-07T17:00:01Z

chain/processor_test.go

+		MetadataManager: metadata.NewDefaultManager(),
+		BalanceHandler:  &mockBalanceHandler{},
+		AuthVM: &chaintest.TestAuthVM{
+			GetAuthBatchVerifierF: func(authTypeID uint8, cores int, count int) (chain.AuthBatchVerifier, bool) {


would it make sense to have a single copy of the GetAuthBatchVerifierF function and use it across these three locations ?

Going to wait until #1960 is merged to address this.

tsachiherman · 2025-03-07T17:02:16Z

chain/processor_test.go

+	test.Run(context.Background(), b)
+}
+
+func BenchmarkExecuteBlocksZipfTxs(b *testing.B) {


given that all these benchmarks are almost identical, I'd opt into having a single benchmark containing multiple sub-benchmarks.

I agree, switched to using table benchmarks (looks much better as well!)

RodrigoVillar added 5 commits March 5, 2025 09:34

add testauthvm

a1aeff7

add empty instance

de0386e

Merge branch 'main' into add-testauthvm

3a9a564

move genesis init to chain

5468398

add block benchmark test suite

dc1fa0a

RodrigoVillar marked this pull request as ready for review March 5, 2025 18:55

RodrigoVillar requested a review from aaronbuchwald as a code owner March 5, 2025 18:55

RodrigoVillar linked an issue Mar 5, 2025 that may be closed by this pull request

Benchmark Processor #1910

Open

add morpheus block tests

6fc0d4d

aaronbuchwald requested changes Mar 5, 2025

View reviewed changes

Base automatically changed from add-genesis-to-chain to main March 5, 2025 22:43

RodrigoVillar added 5 commits March 6, 2025 09:38

style

3f90c9c

Merge branch 'main' into add-block-benchmark-test-suite

7dcfc39

nit

6c15225

lint

b155775

vm developers defines genesis/factories

7a1078d

RodrigoVillar marked this pull request as draft March 6, 2025 16:22

RodrigoVillar added 2 commits March 6, 2025 11:57

txlistgenerator => blockbenchmarkhelper

91951b0

enforce checking expiry window

a88642f

RodrigoVillar self-assigned this Mar 6, 2025

RodrigoVillar marked this pull request as ready for review March 6, 2025 17:16

RodrigoVillar added 3 commits March 6, 2025 13:21

add hypersdk block benchmarks

92fe069

distributions

9d268e0

nit

9913756

tsachiherman reviewed Mar 7, 2025

View reviewed changes

RodrigoVillar added 2 commits March 7, 2025 13:22

use table benchmarks for processor

f097fcc

use table benchmarks in morpheusvm

ff912f6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Block Benchmark Suite #1955

Add Block Benchmark Suite #1955

RodrigoVillar commented Mar 5, 2025 •

edited

Loading

aaronbuchwald Mar 5, 2025

RodrigoVillar Mar 6, 2025

aaronbuchwald Mar 5, 2025

aaronbuchwald Mar 5, 2025

aaronbuchwald Mar 5, 2025

aaronbuchwald Mar 5, 2025

RodrigoVillar Mar 6, 2025

aaronbuchwald Mar 5, 2025

RodrigoVillar Mar 6, 2025

aaronbuchwald Mar 5, 2025

RodrigoVillar Mar 6, 2025

aaronbuchwald Mar 5, 2025

RodrigoVillar commented Mar 6, 2025 •

edited

Loading

tsachiherman Mar 7, 2025

RodrigoVillar Mar 7, 2025

tsachiherman Mar 7, 2025

RodrigoVillar Mar 7, 2025

Add Block Benchmark Suite #1955

Are you sure you want to change the base?

Add Block Benchmark Suite #1955

Conversation

RodrigoVillar commented Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RodrigoVillar commented Mar 6, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RodrigoVillar commented Mar 5, 2025 •

edited

Loading

RodrigoVillar commented Mar 6, 2025 •

edited

Loading