test: migrate cli unit/e2e tests to vitest #6192

nazarhussain · 2023-12-14T11:39:26Z

Motivation

Consolidate the testing frameworks and migrate to vitest.

Description

Migrate cli unit/e2e tests to vitest.

Steps to test or reproduce

Run all tests

github-actions · 2023-12-14T12:35:22Z

⚠️ Performance Alert ⚠️

Possible performance regression was detected for some benchmarks.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold.

Benchmark suite	Current: `495cc97`	Previous: `d3005bf`	Ratio
forkChoice updateHead vc 600000 bc 64 eq 300000	68.009 ms/op	16.275 ms/op	4.18
altair processAttestation - 250000 vs - 7PWei normalcase	7.0621 ms/op	2.2627 ms/op	3.12
altair processAttestation - 250000 vs - 7PWei worstcase	10.366 ms/op	3.4085 ms/op	3.04

Full benchmark results

Benchmark suite	Current: `495cc97`	Previous: `d3005bf`	Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc	569.40 us/op	617.22 us/op	0.92
getPubkeys - validatorsArr - req 1000 vs - 250000 vc	85.714 us/op	84.778 us/op	1.01
BLS verify - blst-native	1.3059 ms/op	1.3312 ms/op	0.98
BLS verifyMultipleSignatures 3 - blst-native	2.7706 ms/op	2.8139 ms/op	0.98
BLS verifyMultipleSignatures 8 - blst-native	5.9489 ms/op	6.1753 ms/op	0.96
BLS verifyMultipleSignatures 32 - blst-native	21.842 ms/op	22.547 ms/op	0.97
BLS verifyMultipleSignatures 64 - blst-native	42.941 ms/op	44.476 ms/op	0.97
BLS verifyMultipleSignatures 128 - blst-native	85.346 ms/op	88.753 ms/op	0.96
BLS deserializing 10000 signatures	937.96 ms/op	920.68 ms/op	1.02
BLS deserializing 100000 signatures	9.4516 s/op	9.6734 s/op	0.98
BLS verifyMultipleSignatures - same message - 3 - blst-native	1.3255 ms/op	1.3640 ms/op	0.97
BLS verifyMultipleSignatures - same message - 8 - blst-native	1.4944 ms/op	1.6998 ms/op	0.88
BLS verifyMultipleSignatures - same message - 32 - blst-native	3.0062 ms/op	2.9494 ms/op	1.02
BLS verifyMultipleSignatures - same message - 64 - blst-native	4.7194 ms/op	4.2028 ms/op	1.12
BLS verifyMultipleSignatures - same message - 128 - blst-native	5.7204 ms/op	7.2895 ms/op	0.78
BLS aggregatePubkeys 32 - blst-native	27.212 us/op	27.930 us/op	0.97
BLS aggregatePubkeys 128 - blst-native	102.23 us/op	101.14 us/op	1.01
getAttestationsForBlock	41.353 ms/op	53.125 ms/op	0.78
getSlashingsAndExits - default max	112.21 us/op	159.65 us/op	0.70
getSlashingsAndExits - 2k	539.32 us/op	423.24 us/op	1.27
proposeBlockBody type=full, size=empty	6.0635 ms/op	5.4065 ms/op	1.12
isKnown best case - 1 super set check	615.00 ns/op	328.00 ns/op	1.88
isKnown normal case - 2 super set checks	603.00 ns/op	309.00 ns/op	1.95
isKnown worse case - 16 super set checks	613.00 ns/op	312.00 ns/op	1.96
CheckpointStateCache - add get delete	6.3830 us/op	5.6810 us/op	1.12
validate api signedAggregateAndProof - struct	2.8154 ms/op	2.7903 ms/op	1.01
validate gossip signedAggregateAndProof - struct	2.8043 ms/op	2.8286 ms/op	0.99
validate gossip attestation - vc 640000	1.4009 ms/op	1.3615 ms/op	1.03
batch validate gossip attestation - vc 640000 - chunk 32	172.07 us/op	164.73 us/op	1.04
batch validate gossip attestation - vc 640000 - chunk 64	164.97 us/op	144.69 us/op	1.14
batch validate gossip attestation - vc 640000 - chunk 128	194.22 us/op	136.49 us/op	1.42
batch validate gossip attestation - vc 640000 - chunk 256	161.85 us/op	138.43 us/op	1.17
pickEth1Vote - no votes	2.8316 ms/op	1.1773 ms/op	2.41
pickEth1Vote - max votes	20.544 ms/op	10.561 ms/op	1.95
pickEth1Vote - Eth1Data hashTreeRoot value x2048	51.608 ms/op	21.590 ms/op	2.39
pickEth1Vote - Eth1Data hashTreeRoot tree x2048	61.596 ms/op	29.080 ms/op	2.12
pickEth1Vote - Eth1Data fastSerialize value x2048	1.2894 ms/op	613.04 us/op	2.10
pickEth1Vote - Eth1Data fastSerialize tree x2048	11.641 ms/op	4.5670 ms/op	2.55
bytes32 toHexString	1.3620 us/op	506.00 ns/op	2.69
bytes32 Buffer.toString(hex)	692.00 ns/op	293.00 ns/op	2.36
bytes32 Buffer.toString(hex) from Uint8Array	693.00 ns/op	451.00 ns/op	1.54
bytes32 Buffer.toString(hex) + 0x	313.00 ns/op	285.00 ns/op	1.10
Object access 1 prop	0.29400 ns/op	0.15900 ns/op	1.85
Map access 1 prop	0.18400 ns/op	0.15000 ns/op	1.23
Object get x1000	11.070 ns/op	7.4470 ns/op	1.49
Map get x1000	0.95300 ns/op	0.78500 ns/op	1.21
Object set x1000	89.806 ns/op	52.329 ns/op	1.72
Map set x1000	60.455 ns/op	40.201 ns/op	1.50
Return object 10000 times	0.31640 ns/op	0.24360 ns/op	1.30
Throw Error 10000 times	4.4063 us/op	3.8881 us/op	1.13
fastMsgIdFn sha256 / 200 bytes	3.8070 us/op	3.3250 us/op	1.14
fastMsgIdFn h32 xxhash / 200 bytes	444.00 ns/op	278.00 ns/op	1.60
fastMsgIdFn h64 xxhash / 200 bytes	464.00 ns/op	338.00 ns/op	1.37
fastMsgIdFn sha256 / 1000 bytes	12.438 us/op	11.364 us/op	1.09
fastMsgIdFn h32 xxhash / 1000 bytes	518.00 ns/op	409.00 ns/op	1.27
fastMsgIdFn h64 xxhash / 1000 bytes	530.00 ns/op	408.00 ns/op	1.30
fastMsgIdFn sha256 / 10000 bytes	123.19 us/op	106.47 us/op	1.16
fastMsgIdFn h32 xxhash / 10000 bytes	2.2170 us/op	2.0190 us/op	1.10
fastMsgIdFn h64 xxhash / 10000 bytes	1.4350 us/op	1.3740 us/op	1.04
send data - 1000 256B messages	24.035 ms/op	19.153 ms/op	1.25
send data - 1000 512B messages	32.235 ms/op	25.787 ms/op	1.25
send data - 1000 1024B messages	48.705 ms/op	41.459 ms/op	1.17
send data - 1000 1200B messages	57.723 ms/op	36.869 ms/op	1.57
send data - 1000 2048B messages	67.752 ms/op	43.427 ms/op	1.56
send data - 1000 4096B messages	52.785 ms/op	40.855 ms/op	1.29
send data - 1000 16384B messages	124.08 ms/op	110.65 ms/op	1.12
send data - 1000 65536B messages	499.74 ms/op	393.80 ms/op	1.27
enrSubnets - fastDeserialize 64 bits	1.9910 us/op	1.3990 us/op	1.42
enrSubnets - ssz BitVector 64 bits	555.00 ns/op	436.00 ns/op	1.27
enrSubnets - fastDeserialize 4 bits	263.00 ns/op	178.00 ns/op	1.48
enrSubnets - ssz BitVector 4 bits	591.00 ns/op	498.00 ns/op	1.19
prioritizePeers score -10:0 att 32-0.1 sync 2-0	155.13 us/op	112.36 us/op	1.38
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25	168.75 us/op	138.33 us/op	1.22
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5	208.88 us/op	210.82 us/op	0.99
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75	399.25 us/op	328.38 us/op	1.22
prioritizePeers score 0:0 att 64-1 sync 4-1	429.45 us/op	363.63 us/op	1.18
array of 16000 items push then shift	1.8491 us/op	1.6982 us/op	1.09
LinkedList of 16000 items push then shift	11.850 ns/op	9.2030 ns/op	1.29
array of 16000 items push then pop	131.41 ns/op	92.952 ns/op	1.41
LinkedList of 16000 items push then pop	10.048 ns/op	9.1290 ns/op	1.10
array of 24000 items push then shift	2.7627 us/op	2.5526 us/op	1.08
LinkedList of 24000 items push then shift	12.794 ns/op	9.1850 ns/op	1.39
array of 24000 items push then pop	194.67 ns/op	139.72 ns/op	1.39
LinkedList of 24000 items push then pop	11.817 ns/op	9.3410 ns/op	1.27
intersect bitArray bitLen 8	7.9200 ns/op	6.5000 ns/op	1.22
intersect array and set length 8	142.42 ns/op	64.549 ns/op	2.21
intersect bitArray bitLen 128	57.116 ns/op	34.527 ns/op	1.65
intersect array and set length 128	1.6666 us/op	913.79 ns/op	1.82
bitArray.getTrueBitIndexes() bitLen 128	2.3450 us/op	1.5750 us/op	1.49
bitArray.getTrueBitIndexes() bitLen 248	4.2450 us/op	2.6710 us/op	1.59
bitArray.getTrueBitIndexes() bitLen 512	8.4600 us/op	5.0820 us/op	1.66
Buffer.concat 32 items	1.4460 us/op	995.00 ns/op	1.45
Uint8Array.set 32 items	3.0880 us/op	1.9390 us/op	1.59
Set add up to 64 items then delete first	5.3964 us/op	4.3020 us/op	1.25
OrderedSet add up to 64 items then delete first	7.8154 us/op	5.4105 us/op	1.44
Set add up to 64 items then delete last	6.7586 us/op	4.5840 us/op	1.47
OrderedSet add up to 64 items then delete last	8.8028 us/op	5.7861 us/op	1.52
Set add up to 64 items then delete middle	7.0773 us/op	4.6169 us/op	1.53
OrderedSet add up to 64 items then delete middle	13.742 us/op	7.0188 us/op	1.96
Set add up to 128 items then delete first	13.952 us/op	9.3565 us/op	1.49
OrderedSet add up to 128 items then delete first	17.548 us/op	12.271 us/op	1.43
Set add up to 128 items then delete last	11.325 us/op	9.2541 us/op	1.22
OrderedSet add up to 128 items then delete last	16.449 us/op	11.698 us/op	1.41
Set add up to 128 items then delete middle	12.034 us/op	9.1948 us/op	1.31
OrderedSet add up to 128 items then delete middle	21.707 us/op	16.720 us/op	1.30
Set add up to 256 items then delete first	23.005 us/op	18.887 us/op	1.22
OrderedSet add up to 256 items then delete first	34.912 us/op	25.468 us/op	1.37
Set add up to 256 items then delete last	23.442 us/op	18.273 us/op	1.28
OrderedSet add up to 256 items then delete last	31.861 us/op	23.172 us/op	1.37
Set add up to 256 items then delete middle	24.330 us/op	18.396 us/op	1.32
OrderedSet add up to 256 items then delete middle	67.089 us/op	44.760 us/op	1.50
transfer serialized Status (84 B)	2.1470 us/op	1.7380 us/op	1.24
copy serialized Status (84 B)	1.7670 us/op	1.5060 us/op	1.17
transfer serialized SignedVoluntaryExit (112 B)	2.2730 us/op	1.9820 us/op	1.15
copy serialized SignedVoluntaryExit (112 B)	2.0660 us/op	1.5850 us/op	1.30
transfer serialized ProposerSlashing (416 B)	2.7210 us/op	2.2780 us/op	1.19
copy serialized ProposerSlashing (416 B)	3.5080 us/op	3.1300 us/op	1.12
transfer serialized Attestation (485 B)	3.5340 us/op	3.2250 us/op	1.10
copy serialized Attestation (485 B)	2.9530 us/op	3.1250 us/op	0.94
transfer serialized AttesterSlashing (33232 B)	3.4360 us/op	3.2180 us/op	1.07
copy serialized AttesterSlashing (33232 B)	11.859 us/op	6.6130 us/op	1.79
transfer serialized Small SignedBeaconBlock (128000 B)	3.6720 us/op	3.0560 us/op	1.20
copy serialized Small SignedBeaconBlock (128000 B)	33.220 us/op	17.824 us/op	1.86
transfer serialized Avg SignedBeaconBlock (200000 B)	3.6980 us/op	3.1870 us/op	1.16
copy serialized Avg SignedBeaconBlock (200000 B)	48.368 us/op	21.438 us/op	2.26
transfer serialized BlobsSidecar (524380 B)	4.4380 us/op	3.1260 us/op	1.42
copy serialized BlobsSidecar (524380 B)	160.48 us/op	97.942 us/op	1.64
transfer serialized Big SignedBeaconBlock (1000000 B)	8.3730 us/op	3.2070 us/op	2.61
copy serialized Big SignedBeaconBlock (1000000 B)	307.47 us/op	219.30 us/op	1.40
pass gossip attestations to forkchoice per slot	5.7211 ms/op	4.2704 ms/op	1.34
forkChoice updateHead vc 100000 bc 64 eq 0	817.16 us/op	686.62 us/op	1.19
forkChoice updateHead vc 600000 bc 64 eq 0	5.5018 ms/op	4.6733 ms/op	1.18
forkChoice updateHead vc 1000000 bc 64 eq 0	9.8150 ms/op	7.1046 ms/op	1.38
forkChoice updateHead vc 600000 bc 320 eq 0	4.8723 ms/op	4.2697 ms/op	1.14
forkChoice updateHead vc 600000 bc 1200 eq 0	5.0085 ms/op	4.3061 ms/op	1.16
forkChoice updateHead vc 600000 bc 7200 eq 0	6.7392 ms/op	5.5790 ms/op	1.21
forkChoice updateHead vc 600000 bc 64 eq 1000	12.252 ms/op	11.315 ms/op	1.08
forkChoice updateHead vc 600000 bc 64 eq 10000	15.438 ms/op	12.085 ms/op	1.28
forkChoice updateHead vc 600000 bc 64 eq 300000	68.009 ms/op	16.275 ms/op	4.18
computeDeltas 500000 validators 300 proto nodes	7.7037 ms/op	6.8184 ms/op	1.13
computeDeltas 500000 validators 1200 proto nodes	7.4590 ms/op	6.6462 ms/op	1.12
computeDeltas 500000 validators 7200 proto nodes	6.9548 ms/op	6.4632 ms/op	1.08
computeDeltas 750000 validators 300 proto nodes	10.693 ms/op	9.7178 ms/op	1.10
computeDeltas 750000 validators 1200 proto nodes	10.467 ms/op	9.6766 ms/op	1.08
computeDeltas 750000 validators 7200 proto nodes	10.669 ms/op	9.9186 ms/op	1.08
computeDeltas 1400000 validators 300 proto nodes	19.604 ms/op	18.761 ms/op	1.04
computeDeltas 1400000 validators 1200 proto nodes	19.214 ms/op	19.513 ms/op	0.98
computeDeltas 1400000 validators 7200 proto nodes	18.689 ms/op	19.919 ms/op	0.94
computeDeltas 2100000 validators 300 proto nodes	28.612 ms/op	29.037 ms/op	0.99
computeDeltas 2100000 validators 1200 proto nodes	27.760 ms/op	29.040 ms/op	0.96
computeDeltas 2100000 validators 7200 proto nodes	30.290 ms/op	29.548 ms/op	1.03
computeProposerBoostScoreFromBalances 500000 validators	4.3096 ms/op	3.8326 ms/op	1.12
computeProposerBoostScoreFromBalances 750000 validators	5.6654 ms/op	3.8195 ms/op	1.48
computeProposerBoostScoreFromBalances 1400000 validators	5.4585 ms/op	3.8045 ms/op	1.43
computeProposerBoostScoreFromBalances 2100000 validators	4.8534 ms/op	3.7577 ms/op	1.29
altair processAttestation - 250000 vs - 7PWei normalcase	7.0621 ms/op	2.2627 ms/op	3.12
altair processAttestation - 250000 vs - 7PWei worstcase	10.366 ms/op	3.4085 ms/op	3.04
altair processAttestation - setStatus - 1/6 committees join	283.04 us/op	144.29 us/op	1.96
altair processAttestation - setStatus - 1/3 committees join	388.36 us/op	294.06 us/op	1.32
altair processAttestation - setStatus - 1/2 committees join	687.96 us/op	383.54 us/op	1.79
altair processAttestation - setStatus - 2/3 committees join	629.12 us/op	481.10 us/op	1.31
altair processAttestation - setStatus - 4/5 committees join	1.2058 ms/op	676.40 us/op	1.78
altair processAttestation - setStatus - 100% committees join	1.3916 ms/op	811.41 us/op	1.72
altair processBlock - 250000 vs - 7PWei normalcase	14.182 ms/op	9.9211 ms/op	1.43
altair processBlock - 250000 vs - 7PWei normalcase hashState	59.548 ms/op	34.756 ms/op	1.71
altair processBlock - 250000 vs - 7PWei worstcase	67.652 ms/op	37.009 ms/op	1.83
altair processBlock - 250000 vs - 7PWei worstcase hashState	167.19 ms/op	95.838 ms/op	1.74
phase0 processBlock - 250000 vs - 7PWei normalcase	6.6614 ms/op	3.2664 ms/op	2.04
phase0 processBlock - 250000 vs - 7PWei worstcase	48.918 ms/op	33.923 ms/op	1.44
altair processEth1Data - 250000 vs - 7PWei normalcase	1.2208 ms/op	599.97 us/op	2.03
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15	21.768 us/op	11.181 us/op	1.95
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219	135.39 us/op	62.899 us/op	2.15
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42	44.922 us/op	16.665 us/op	2.70
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18	32.420 us/op	11.500 us/op	2.82
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020	256.14 us/op	189.38 us/op	1.35
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777	2.3986 ms/op	1.4632 ms/op	1.64
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384	3.4126 ms/op	1.6798 ms/op	2.03
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384	3.5247 ms/op	2.9176 ms/op	1.21
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384	11.733 ms/op	3.9795 ms/op	2.95
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384	4.8262 ms/op	2.6734 ms/op	1.81
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384	12.662 ms/op	5.7619 ms/op	2.20
Tree 40 250000 create	1.3026 s/op	369.36 ms/op	3.53
Tree 40 250000 get(125000)	370.78 ns/op	211.84 ns/op	1.75
Tree 40 250000 set(125000)	4.1526 us/op	1.0979 us/op	3.78
Tree 40 250000 toArray()	42.743 ms/op	21.738 ms/op	1.97
Tree 40 250000 iterate all - toArray() + loop	43.026 ms/op	20.062 ms/op	2.14
Tree 40 250000 iterate all - get(i)	110.25 ms/op	67.584 ms/op	1.63
MutableVector 250000 create	23.971 ms/op	17.852 ms/op	1.34
MutableVector 250000 get(125000)	11.408 ns/op	6.6380 ns/op	1.72
MutableVector 250000 set(125000)	868.47 ns/op	281.89 ns/op	3.08
MutableVector 250000 toArray()	7.3188 ms/op	3.3645 ms/op	2.18
MutableVector 250000 iterate all - toArray() + loop	7.3988 ms/op	3.9873 ms/op	1.86
MutableVector 250000 iterate all - get(i)	3.2918 ms/op	1.5522 ms/op	2.12
Array 250000 create	7.0261 ms/op	4.0630 ms/op	1.73
Array 250000 clone - spread	7.4386 ms/op	1.2261 ms/op	6.07
Array 250000 get(125000)	4.2690 ns/op	1.0260 ns/op	4.16
Array 250000 set(125000)	6.5820 ns/op	4.1370 ns/op	1.59
Array 250000 iterate all - loop	324.46 us/op	167.68 us/op	1.94
effectiveBalanceIncrements clone Uint8Array 300000	95.105 us/op	30.217 us/op	3.15
effectiveBalanceIncrements clone MutableVector 300000	2.3500 us/op	346.00 ns/op	6.79
effectiveBalanceIncrements rw all Uint8Array 300000	389.61 us/op	202.79 us/op	1.92
effectiveBalanceIncrements rw all MutableVector 300000	293.48 ms/op	83.258 ms/op	3.52
phase0 afterProcessEpoch - 250000 vs - 7PWei	166.49 ms/op	114.51 ms/op	1.45
phase0 beforeProcessEpoch - 250000 vs - 7PWei	89.774 ms/op	54.807 ms/op	1.64
altair processEpoch - mainnet_e81889	700.21 ms/op	557.44 ms/op	1.26
mainnet_e81889 - altair beforeProcessEpoch	129.72 ms/op	86.784 ms/op	1.49
mainnet_e81889 - altair processJustificationAndFinalization	29.414 us/op	15.000 us/op	1.96
mainnet_e81889 - altair processInactivityUpdates	7.6515 ms/op	6.5800 ms/op	1.16
mainnet_e81889 - altair processRewardsAndPenalties	81.182 ms/op	73.486 ms/op	1.10
mainnet_e81889 - altair processRegistryUpdates	10.918 us/op	2.2140 us/op	4.93
mainnet_e81889 - altair processSlashings	1.6890 us/op	504.00 ns/op	3.35
mainnet_e81889 - altair processEth1DataReset	1.9820 us/op	596.00 ns/op	3.33
mainnet_e81889 - altair processEffectiveBalanceUpdates	3.0666 ms/op	1.7281 ms/op	1.77
mainnet_e81889 - altair processSlashingsReset	10.387 us/op	7.3270 us/op	1.42
mainnet_e81889 - altair processRandaoMixesReset	13.415 us/op	7.5840 us/op	1.77
mainnet_e81889 - altair processHistoricalRootsUpdate	2.5550 us/op	820.00 ns/op	3.12
mainnet_e81889 - altair processParticipationFlagUpdates	5.8200 us/op	1.8720 us/op	3.11
mainnet_e81889 - altair processSyncCommitteeUpdates	1.9810 us/op	744.00 ns/op	2.66
mainnet_e81889 - altair afterProcessEpoch	167.34 ms/op	121.97 ms/op	1.37
capella processEpoch - mainnet_e217614	3.3078 s/op	2.3516 s/op	1.41
mainnet_e217614 - capella beforeProcessEpoch	756.37 ms/op	509.63 ms/op	1.48
mainnet_e217614 - capella processJustificationAndFinalization	36.378 us/op	25.358 us/op	1.43
mainnet_e217614 - capella processInactivityUpdates	44.842 ms/op	20.848 ms/op	2.15
mainnet_e217614 - capella processRewardsAndPenalties	646.06 ms/op	424.85 ms/op	1.52
mainnet_e217614 - capella processRegistryUpdates	46.018 us/op	30.864 us/op	1.49
mainnet_e217614 - capella processSlashings	2.1650 us/op	651.00 ns/op	3.33
mainnet_e217614 - capella processEth1DataReset	767.00 ns/op	500.00 ns/op	1.53
mainnet_e217614 - capella processEffectiveBalanceUpdates	8.1889 ms/op	5.5342 ms/op	1.48
mainnet_e217614 - capella processSlashingsReset	8.5760 us/op	4.7140 us/op	1.82
mainnet_e217614 - capella processRandaoMixesReset	13.470 us/op	5.2700 us/op	2.56
mainnet_e217614 - capella processHistoricalRootsUpdate	1.2510 us/op	506.00 ns/op	2.47
mainnet_e217614 - capella processParticipationFlagUpdates	2.6510 us/op	1.8630 us/op	1.42
mainnet_e217614 - capella afterProcessEpoch	316.67 ms/op	336.52 ms/op	0.94
phase0 processEpoch - mainnet_e58758	647.13 ms/op	519.63 ms/op	1.25
mainnet_e58758 - phase0 beforeProcessEpoch	196.37 ms/op	159.54 ms/op	1.23
mainnet_e58758 - phase0 processJustificationAndFinalization	31.728 us/op	21.727 us/op	1.46
mainnet_e58758 - phase0 processRewardsAndPenalties	59.429 ms/op	67.684 ms/op	0.88
mainnet_e58758 - phase0 processRegistryUpdates	17.357 us/op	16.654 us/op	1.04
mainnet_e58758 - phase0 processSlashings	1.8870 us/op	674.00 ns/op	2.80
mainnet_e58758 - phase0 processEth1DataReset	1.2310 us/op	638.00 ns/op	1.93
mainnet_e58758 - phase0 processEffectiveBalanceUpdates	2.1064 ms/op	2.1456 ms/op	0.98
mainnet_e58758 - phase0 processSlashingsReset	5.9410 us/op	4.3970 us/op	1.35
mainnet_e58758 - phase0 processRandaoMixesReset	8.9130 us/op	7.6070 us/op	1.17
mainnet_e58758 - phase0 processHistoricalRootsUpdate	1.2450 us/op	678.00 ns/op	1.84
mainnet_e58758 - phase0 processParticipationRecordUpdates	9.0470 us/op	5.8900 us/op	1.54
mainnet_e58758 - phase0 afterProcessEpoch	113.46 ms/op	93.071 ms/op	1.22
phase0 processEffectiveBalanceUpdates - 250000 normalcase	2.8745 ms/op	2.5930 ms/op	1.11
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5	3.4456 ms/op	1.5247 ms/op	2.26
altair processInactivityUpdates - 250000 normalcase	44.398 ms/op	30.993 ms/op	1.43
altair processInactivityUpdates - 250000 worstcase	37.378 ms/op	32.809 ms/op	1.14
phase0 processRegistryUpdates - 250000 normalcase	16.019 us/op	12.953 us/op	1.24
phase0 processRegistryUpdates - 250000 badcase_full_deposits	460.42 us/op	643.69 us/op	0.72
phase0 processRegistryUpdates - 250000 worstcase 0.5	163.92 ms/op	138.76 ms/op	1.18
altair processRewardsAndPenalties - 250000 normalcase	64.947 ms/op	72.214 ms/op	0.90
altair processRewardsAndPenalties - 250000 worstcase	57.169 ms/op	69.276 ms/op	0.83
phase0 getAttestationDeltas - 250000 normalcase	9.2097 ms/op	9.3050 ms/op	0.99
phase0 getAttestationDeltas - 250000 worstcase	9.6381 ms/op	9.3875 ms/op	1.03
phase0 processSlashings - 250000 worstcase	97.786 us/op	131.98 us/op	0.74
altair processSyncCommitteeUpdates - 250000	159.78 ms/op	152.37 ms/op	1.05
BeaconState.hashTreeRoot - No change	293.00 ns/op	255.00 ns/op	1.15
BeaconState.hashTreeRoot - 1 full validator	135.69 us/op	189.73 us/op	0.72
BeaconState.hashTreeRoot - 32 full validator	1.3958 ms/op	1.6575 ms/op	0.84
BeaconState.hashTreeRoot - 512 full validator	19.072 ms/op	18.921 ms/op	1.01
BeaconState.hashTreeRoot - 1 validator.effectiveBalance	219.00 us/op	176.72 us/op	1.24
BeaconState.hashTreeRoot - 32 validator.effectiveBalance	2.3049 ms/op	2.9053 ms/op	0.79
BeaconState.hashTreeRoot - 512 validator.effectiveBalance	26.407 ms/op	32.017 ms/op	0.82
BeaconState.hashTreeRoot - 1 balances	122.91 us/op	139.72 us/op	0.88
BeaconState.hashTreeRoot - 32 balances	1.2666 ms/op	1.2460 ms/op	1.02
BeaconState.hashTreeRoot - 512 balances	11.594 ms/op	13.688 ms/op	0.85
BeaconState.hashTreeRoot - 250000 balances	208.14 ms/op	226.81 ms/op	0.92
aggregationBits - 2048 els - zipIndexesInBitList	18.874 us/op	17.145 us/op	1.10
byteArrayEquals 32	75.099 ns/op	75.458 ns/op	1.00
Buffer.compare 32	55.928 ns/op	55.700 ns/op	1.00
byteArrayEquals 1024	2.0463 us/op	2.0285 us/op	1.01
Buffer.compare 1024	72.013 ns/op	71.235 ns/op	1.01
byteArrayEquals 16384	32.076 us/op	32.605 us/op	0.98
Buffer.compare 16384	267.06 ns/op	262.48 ns/op	1.02
byteArrayEquals 123687377	251.74 ms/op	260.36 ms/op	0.97
Buffer.compare 123687377	9.1837 ms/op	9.0453 ms/op	1.02
byteArrayEquals 32 - diff last byte	78.231 ns/op	81.448 ns/op	0.96
Buffer.compare 32 - diff last byte	58.316 ns/op	62.322 ns/op	0.94
byteArrayEquals 1024 - diff last byte	2.1606 us/op	2.2273 us/op	0.97
Buffer.compare 1024 - diff last byte	74.127 ns/op	75.671 ns/op	0.98
byteArrayEquals 16384 - diff last byte	33.367 us/op	35.119 us/op	0.95
Buffer.compare 16384 - diff last byte	288.11 ns/op	311.74 ns/op	0.92
byteArrayEquals 123687377 - diff last byte	254.77 ms/op	268.16 ms/op	0.95
Buffer.compare 123687377 - diff last byte	10.572 ms/op	8.8174 ms/op	1.20
byteArrayEquals 32 - random bytes	6.7580 ns/op	6.5850 ns/op	1.03
Buffer.compare 32 - random bytes	62.537 ns/op	64.080 ns/op	0.98
byteArrayEquals 1024 - random bytes	6.6920 ns/op	6.5200 ns/op	1.03
Buffer.compare 1024 - random bytes	61.849 ns/op	63.601 ns/op	0.97
byteArrayEquals 16384 - random bytes	7.1460 ns/op	6.5320 ns/op	1.09
Buffer.compare 16384 - random bytes	63.116 ns/op	63.880 ns/op	0.99
byteArrayEquals 123687377 - random bytes	11.610 ns/op	10.140 ns/op	1.14
Buffer.compare 123687377 - random bytes	79.240 ns/op	77.620 ns/op	1.02
regular array get 100000 times	46.940 us/op	47.681 us/op	0.98
wrappedArray get 100000 times	49.782 us/op	50.827 us/op	0.98
arrayWithProxy get 100000 times	15.309 ms/op	14.774 ms/op	1.04
ssz.Root.equals	56.565 ns/op	60.890 ns/op	0.93
byteArrayEquals	54.849 ns/op	59.525 ns/op	0.92
Buffer.compare	12.393 ns/op	12.994 ns/op	0.95
shuffle list - 16384 els	7.2670 ms/op	7.4534 ms/op	0.97
shuffle list - 250000 els	106.39 ms/op	113.34 ms/op	0.94
processSlot - 1 slots	22.981 us/op	21.388 us/op	1.07
processSlot - 32 slots	3.6936 ms/op	4.1495 ms/op	0.89
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei	60.405 ms/op	60.627 ms/op	1.00
getCommitteeAssignments - req 1 vs - 250000 vc	2.5283 ms/op	2.6066 ms/op	0.97
getCommitteeAssignments - req 100 vs - 250000 vc	3.7500 ms/op	3.8584 ms/op	0.97
getCommitteeAssignments - req 1000 vs - 250000 vc	4.1292 ms/op	4.1734 ms/op	0.99
findModifiedValidators - 10000 modified validators	521.06 ms/op	597.54 ms/op	0.87
findModifiedValidators - 1000 modified validators	412.83 ms/op	496.45 ms/op	0.83
findModifiedValidators - 100 modified validators	389.00 ms/op	492.26 ms/op	0.79
findModifiedValidators - 10 modified validators	407.21 ms/op	470.94 ms/op	0.86
findModifiedValidators - 1 modified validators	394.88 ms/op	473.01 ms/op	0.83
findModifiedValidators - no difference	448.49 ms/op	488.08 ms/op	0.92
compare ViewDUs	5.1098 s/op	5.1864 s/op	0.99
compare each validator Uint8Array	1.9371 s/op	1.7828 s/op	1.09
compare ViewDU to Uint8Array	1.4425 s/op	1.3830 s/op	1.04
migrate state 1000000 validators, 24 modified, 0 new	1.1076 s/op	900.06 ms/op	1.23
migrate state 1000000 validators, 1700 modified, 1000 new	1.2380 s/op	1.3101 s/op	0.94
migrate state 1000000 validators, 3400 modified, 2000 new	1.2662 s/op	1.5017 s/op	0.84
migrate state 1500000 validators, 24 modified, 0 new	760.93 ms/op	1.0023 s/op	0.76
migrate state 1500000 validators, 1700 modified, 1000 new	1.0125 s/op	1.2620 s/op	0.80
migrate state 1500000 validators, 3400 modified, 2000 new	1.3237 s/op	1.3991 s/op	0.95
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei	4.2900 ns/op	5.2500 ns/op	0.82
state getBlockRootAtSlot - 250000 vs - 7PWei	752.62 ns/op	939.23 ns/op	0.80
computeProposers - vc 250000	8.8219 ms/op	10.659 ms/op	0.83
computeEpochShuffling - vc 250000	102.83 ms/op	109.01 ms/op	0.94
getNextSyncCommittee - vc 250000	150.30 ms/op	159.40 ms/op	0.94
computeSigningRoot for AttestationData	27.441 us/op	30.713 us/op	0.89
hash AttestationData serialized data then Buffer.toString(base64)	2.3242 us/op	2.3574 us/op	0.99
toHexString serialized data	1.0631 us/op	1.1396 us/op	0.93
Buffer.toString(base64)	215.26 ns/op	233.54 ns/op	0.92

by benchmarkbot/action

codecov · 2023-12-14T17:05:58Z

Codecov Report

Merging #6192 (b49aeb0) into unstable (e09702d) will decrease coverage by 4.06%.
Report is 2 commits behind head on unstable.
The diff coverage is n/a.

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #6192      +/-   ##
============================================
- Coverage     90.35%   86.30%   -4.06%     
============================================
  Files            78      141      +63     
  Lines          8087    12157    +4070     
  Branches        490      721     +231     
============================================
+ Hits           7307    10492    +3185     
- Misses          772     1644     +872     
- Partials          8       21      +13

wemeetagain · 2023-12-18T15:43:01Z

packages/cli/src/cmds/validator/keymanager/decryptKeystores/threadPool.ts

 import {spawn, Pool, Worker, ModuleThread, QueuedTask} from "@chainsafe/threads";
 import {DecryptKeystoreArgs, DecryptKeystoreWorkerAPI} from "./types.js";
 import {maxPoolSize} from "./poolSize.js";

+// Worker constructor consider the path relative to the current working directory
+const workerDir =
+  process.env.NODE_ENV === "test" ? "../../../../../lib/cmds/validator/keymanager/decryptKeystores" : "./";


why is this necessary?

Mocha and Vitest handles the runtime dependencies differently. Mocha passes everything through ts-node where it translates internally to right compiled file. While vitest not ties to to load the worker with exact path given. If we don't do that it tries to load ./worker.js which does not exits in that directory ./worker.ts exists.

So why we have to point the worker path to lib path in test env.

Have you tried if new URL('./worker.js', import.meta.url) works?

Yes tried that it didn't work. import.meta.url points to the current src directory.

Can we make this a bit more robust at least right now it breaks if

someone runs tests without NODE_ENV set (we should avoid on relying on this env to be set) <-- vitest might set this haven't tested

the file is moved or folders are renamed (this will not be detected in a refactoring that moves files / folders)

@nazarhussain I noticed we are using this pattern already on unstable branch. This looks like a bug in vitest as this workaround is not reasonable in my opinion, as testing framework should not force you to use weird hacks in your src code.

I think what we should do is

open a issue on vitest repo to get this resolved (if it doesn't exist already)

create an issue on lodestar repository, we also should make sure to reference all lines that have this in the issue

lodestar/packages/beacon-node/src/chain/bls/multithread/index.ts

Line 33 in d3005bf

const workerDir = process.env.NODE_ENV === "test" ? "../../../../lib/chain/bls/multithread" : "./";

lodestar/packages/beacon-node/src/network/core/networkCoreWorkerHandler.ts

Line 33 in d3005bf

const workerDir = process.env.NODE_ENV === "test" ? "../../../lib/network/core/" : "./";

I had created an issue for it. #6208

It's not a bug in vitest neither a hack. Mocha uses runtime NODE_OPTIONS to specify a node loader to ts-node, so every code that you run in Node had to go through ts-node, that's why it's slow.

Vitest compile and deliver the code to test runner, on runtime, so why when it requests for a file src/worker.js it does not find it there.

Will try to find a neat workaround to address this thing in separate PR.

@nazarhussain I noticed we are using this pattern already on unstable branch. This looks like a bug in vitest as this workaround is not reasonable in my opinion, as testing framework should not force you to use weird hacks in your src code.

I think what we should do is

open a issue on vitest repo to get this resolved (if it doesn't exist already)

create an issue on lodestar repository, we also should make sure to reference all lines that have this in the issue

lodestar/packages/beacon-node/src/chain/bls/multithread/index.ts

Line 33 in d3005bf

const workerDir = process.env.NODE_ENV === "test" ? "../../../../lib/chain/bls/multithread" : "./";

lodestar/packages/beacon-node/src/network/core/networkCoreWorkerHandler.ts

Line 33 in d3005bf

const workerDir = process.env.NODE_ENV === "test" ? "../../../lib/network/core/" : "./";

It's not a bug in vitest neither a hack.

Having to use process.env.NODE_ENV === "test" in your source code is definitely a hack and ../../../../lib/ as well, if there is no other way to make it work we have to bite the bullet but this is not pretty.

The use of process.env.NODE_ENV to production or test is common practice in JS projects to optimize some logic for particular use cases. And mostly bundling the code such usages are stripped down.

That path looks ugly, because we are not using any import aliases.

The use of process.env.NODE_ENV to production or test is common practice in JS projects to optimize some logic for particular use cases

See NODE ENV=production is a lie, I don't think it good to rely on that. For testing it might be still ok, I would still consider it a hack though.

That path looks ugly, because we are not using any import aliases.

My main concern is not that it is ugly but rather that it is really fragile and will break if any folder changes in that path.

packages/cli/test/unit/validator/decryptKeystoreDefinitions.test.ts

nflaig · 2023-12-18T21:26:31Z

packages/cli/src/cmds/validator/keymanager/decryptKeystores/threadPool.ts

 import {spawn, Pool, Worker, ModuleThread, QueuedTask} from "@chainsafe/threads";
 import {DecryptKeystoreArgs, DecryptKeystoreWorkerAPI} from "./types.js";
 import {maxPoolSize} from "./poolSize.js";

+// Worker constructor consider the path relative to the current working directory
+const workerDir =
+  process.env.NODE_ENV === "test" ? "../../../../../lib/cmds/validator/keymanager/decryptKeystores" : "./";


Have you tried if new URL('./worker.js', import.meta.url) works?

packages/cli/test/unit/validator/decryptKeystoreDefinitions.test.ts

nflaig · 2023-12-19T12:24:52Z

packages/cli/test/utils/runUtils.ts

 }

 /**
 * Similar to `expectDeepEquals` but only checks presence of all elements in array, irrespective of their order.
 */
 export function expectDeepEqualsUnordered<T>(a: T[], b: T[], message: string): void {
-  expect(a).to.have.deep.members(b, message);
+  try {
+    expect(a.sort()).toEqual(b.sort());


@nazarhussain I don't think this works as expected, see failed test run.

Note that tests might pass sometimes, there was a previous issue related to that

test: make results of keymanager e2e tests deterministic #5641

Also noticing from that issue, our test output is not really detailed compared to previous output if tests fail (see screenshot in linked issue). Previously the output showed exactly what value was wrong while now it does tell me anything about what part of the data is wrong

wemeetagain · 2024-01-16T04:10:32Z

🎉 This PR is included in v1.14.0 🎉

* Add vitest config * Migrate unit tests * Migrate e2e tests * Revert test-util changes * Update worker path * Fix e2e tests * Fix the validator process crash * Update single thread value * Fix the exit hooks * Remove timeout indentation * Remvoe some test comments

nazarhussain requested a review from a team as a code owner December 14, 2023 11:39

nazarhussain self-assigned this Dec 14, 2023

nazarhussain added 5 commits December 18, 2023 11:53

Add vitest config

4d9657e

Migrate unit tests

f1fe62f

Migrate e2e tests

62f1adb

Revert test-util changes

8fedb7d

Update worker path

363c568

nazarhussain force-pushed the nh/vitest-cli branch from fa06515 to 363c568 Compare December 18, 2023 10:53

nazarhussain added 4 commits December 18, 2023 14:42

Fix e2e tests

b5ed676

Fix the validator process crash

794427f

Update single thread value

0784e58

Fix the exit hooks

2107b1e

wemeetagain reviewed Dec 18, 2023

View reviewed changes

nflaig reviewed Dec 18, 2023

View reviewed changes

Remove timeout indentation

f8d46b9

nazarhussain requested a review from nflaig December 19, 2023 09:29

nflaig reviewed Dec 19, 2023

View reviewed changes

packages/cli/test/unit/validator/decryptKeystoreDefinitions.test.ts Show resolved Hide resolved

nazarhussain requested a review from nflaig December 19, 2023 10:14

Remvoe some test comments

b49aeb0

nazarhussain force-pushed the nh/vitest-cli branch from f3cd22b to b49aeb0 Compare December 19, 2023 10:55

nazarhussain mentioned this pull request Dec 19, 2023

Use the worker loading from src directory #6208

Open

nflaig approved these changes Dec 19, 2023

View reviewed changes

nazarhussain merged commit 6670c08 into unstable Dec 19, 2023

nazarhussain deleted the nh/vitest-cli branch December 19, 2023 11:56

nflaig reviewed Dec 19, 2023

View reviewed changes

nflaig mentioned this pull request Dec 22, 2023

test: fix flaky e2e tests #6231

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test: migrate cli unit/e2e tests to vitest #6192

test: migrate cli unit/e2e tests to vitest #6192

nazarhussain commented Dec 14, 2023

github-actions bot commented Dec 14, 2023 •

edited

Loading

codecov bot commented Dec 14, 2023 •

edited

Loading

wemeetagain Dec 18, 2023

nazarhussain Dec 18, 2023

nflaig Dec 18, 2023

nazarhussain Dec 19, 2023

nflaig Dec 19, 2023 •

edited

Loading

nflaig Dec 19, 2023

nazarhussain Dec 19, 2023

nflaig Dec 19, 2023

nazarhussain Dec 19, 2023 •

edited

Loading

nflaig Dec 19, 2023

nflaig Dec 18, 2023

nflaig Dec 19, 2023

wemeetagain commented Jan 16, 2024

test: migrate cli unit/e2e tests to vitest #6192

test: migrate cli unit/e2e tests to vitest #6192

Conversation

nazarhussain commented Dec 14, 2023

github-actions bot commented Dec 14, 2023 • edited Loading

⚠️ Performance Alert ⚠️

codecov bot commented Dec 14, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nflaig Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nazarhussain Dec 19, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wemeetagain commented Jan 16, 2024

github-actions bot commented Dec 14, 2023 •

edited

Loading

codecov bot commented Dec 14, 2023 •

edited

Loading

nflaig Dec 19, 2023 •

edited

Loading

nazarhussain Dec 19, 2023 •

edited

Loading