feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best #7513

aakoshh · 2025-02-25T14:37:28Z

Description

Problem*

Related to #7511

Summary*

The comments under the PR contain charts showing the impact the impact on bytecode size of switching to a different serialisation format. Protobuf turned out to be better than bincode, so this PR implements the boilerplate to serialise Program and WitnessStack using protobuf. We can't switch to using it yet, because the Barretenberg wouldn't be able to handle it, and the integration tests use the latest bb to parse bytecode and witness files as well.

Once bb can handle both formats (try one, fall back to the other, or take a format parameter), then we can update the integration tests to use the new format and sunset the other. We can potentially also just keep both available, since we will most likely keep using serde-reflect as a convenient way to generate the domain objects for C++, into which we can deserialise from protobuf.

Additional Context

CBOR was relatively good for larger contracts, not much worse than bincode. But on very small contracts it has substantial overhead, so hopefully the extra maintenance of protobuf will be worth it.

The PR contains scripts to produce the comparisons in the comments (see below).

I added the GITHUB_TOKEN to the workflow jobs that build nargo so that don't get rate limited on CI when downloading pre-built protoc binaries, but there are other ways we could procure it: I'm sure there is a GitHub action already, but also there is a crate which would build it from source. I wanted it to be low maintenance but quick, which was the appeal of the crate I used that downloads it on the fly.

Compiling contracts

Run these commands to compile noir protocol circuits and contracts in aztec-packages after rebuilding nargo:

cargo build -p nargo_cli --release
./target/release/nargo --program-dir ../aztec-packages/noir-projects/noir-protocol-circuits compile --force --silence-warnings --skip-underconstrained-check
./target/release/nargo --program-dir ../aztec-packages/noir-projects/noir-contracts compile --force --silence-warnings --skip-underconstrained-check

Baseline

Recorded the bytecode size with bincode before switching to other implementations:

./scripts/bytecode-sizes/print-bytecode-size.sh ../aztec-packages > ./scripts/bytecode-sizes/bincode.jsonl

Comparing bytecode

Run these commands to record a new measurement (e.g. flexbuffers here) and compare against bincode.

./scripts/bytecode-sizes/print-bytecode-size.sh ../aztec-packages > ./scripts/bytecode-sizes/flexbuffers.jsonl
./scripts/bytecode-sizes/compare-bytecode-size.sh ./scripts/bytecode-sizes/bincode.jsonl ./scripts/bytecode-sizes/flexbuffers.jsonl > ./scripts/bytecode-sizes/bincode-vs-flexbuffers.jsonl
./scripts/bytecode-sizes/plot-bytecode-size.sh ./scripts/bytecode-sizes/bincode-vs-flexbuffers.jsonl

Documentation*

Check one:

No documentation needed.
Documentation included in this PR.
[For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

I have tested the changes locally.
I have formatted the changes with Prettier and/or cargo fmt on default settings.

github-actions

⚠️ Performance Alert ⚠️

Possible performance regression was detected for benchmark 'Compilation Time'.
Benchmark result of this commit is worse than the previous benchmark result exceeding threshold 1.20.

Benchmark suite	Current: `67e98a9`	Previous: `6f78848`	Ratio
`global_var_regression_entry_points`	`2.03` s	`0.541` s	`3.75`

This comment was automatically generated by workflow using github-action-benchmark.

CC: @TomAFrench

aakoshh · 2025-02-25T16:07:35Z

CBOR shows a stable 5-10% overhead for circuits over 10KB in size; below that the overhead goes up to +170% percent.

aakoshh · 2025-02-25T16:08:33Z

FlexBuffers has a higher base overhead and doesn't show any trend of converging to a lower value for larger circuits, going from +200% to +700%:

…m-ser

aakoshh · 2025-02-28T12:41:51Z

This is the results of comparing bincode to protobuf: Protobuf is 10%-20% worse until 1KB, then 10%-20% better. Beats CBOR, as expected.

…m-ser

acvm-repo/acir/src/lib.rs

asterite

Looks great!

* master: (65 commits) feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best (#7513) chore: track more critical libraries (#7604) chore!: bump bb version to v0.77.0 (#7599) chore: bump external pinned commits (#7601) feat(cli): Log and replay oracle transcript (#7417) chore: some SSA improvements (#7588) chore(profiler): Add option to only get the total sample count for the `execution-opcodes` command (#7578) feat(experimental): Issue errors for unreachable match branches (#7556) fix: Log to `stderr` (#7585) chore!: remove merkle module from stdlib (#7582) fix: Display causes but not stack trace in CLI error report (#7584) chore: bump `light-poseidon` (#7568) chore: bump external pinned commits (#7581) chore!: remove deprecated hash functions from stdlib (#7477) fix(frontend)!: Restrict capturing mutable variable in lambdas (#7488) feat: perform constant sha256 compressions at compile-time (#7566) chore: bump external pinned commits (#7565) chore(ssa): Turn the Brillig constraints check back on by default (#7404) chore: bump external pinned commits (#7561) chore: address some frontend tests TODOs (#7554) ...

… the correct… (noir-lang/noir#7598) feat: Sync from aztec-packages (noir-lang/noir#7606) feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best (noir-lang/noir#7513)

Automated pull of development from the [noir](https://github.com/noir-lang/noir) programming language, a dependency of Aztec. BEGIN_COMMIT_OVERRIDE fix(LSP): references/rename only when underlying span has the correct… (noir-lang/noir#7598) feat: Sync from aztec-packages (noir-lang/noir#7606) feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best (noir-lang/noir#7513) END_COMMIT_OVERRIDE

🤖 I have created a new Aztec Packages release --- ## [0.78.0](v0.77.1...v0.78.0) (2025-03-07) ### ⚠ BREAKING CHANGES * convert `TraitMethodNotInScope` to error (noir-lang/noir#7427) * bump bb version to v0.77.0 (noir-lang/noir#7599) * remove merkle module from stdlib (noir-lang/noir#7582) * remove deprecated hash functions from stdlib (noir-lang/noir#7477) * **frontend:** Restrict capturing mutable variable in lambdas (noir-lang/noir#7488) * remove U128 struct from stdlib (noir-lang/noir#7529) ### Features * **barretenberg:** Graph methods for circuit analysis (part 2) ([#12130](#12130)) ([ec4c0c4](ec4c0c4)) * **cli:** Log and replay oracle transcript (noir-lang/noir#7417) ([f13b729](f13b729)) * Compare bincode to CBOR, FlexBuffers and Protobuf - implement best (noir-lang/noir#7513) ([8eb727c](8eb727c)) * **experimental:** Enable ownership syntax (noir-lang/noir#7603) ([1a3c112](1a3c112)) * **experimental:** Issue errors for unreachable match branches (noir-lang/noir#7556) ([f13b729](f13b729)) * nullify just-added notes ([#12552](#12552)) ([dcba7a4](dcba7a4)) * perform constant sha256 compressions at compile-time (noir-lang/noir#7566) ([f13b729](f13b729)) * relate errors to macro built-ins errors (noir-lang/noir#7609) ([fbaa634](fbaa634)) * simplify simple conditionals for brillig (noir-lang/noir#7205) ([f13b729](f13b729)) * Support `<Type as Trait>::method` in expressions (noir-lang/noir#7551) ([f13b729](f13b729)) * Sync from aztec-packages (noir-lang/noir#7606) ([8eb727c](8eb727c)) * teardown in call interface ([#12499](#12499)) ([062df02](062df02)) * translation evaluations with zk ([#12222](#12222)) ([568982d](568982d)) ### Bug Fixes * **avm:** use the correct number of rows in check_interaction ([#12519](#12519)) ([b1284ef](b1284ef)) * aztec-up ([#12509](#12509)) ([3ddb6de](3ddb6de)) * bbup ([#12555](#12555)) ([e7b5353](e7b5353)) * Bitwise lookup ([#12471](#12471)) ([a38f353](a38f353)) * **ci:** remove regex - transfer explicitly ([#12525](#12525)) ([352bb1d](352bb1d)) * Cl/fix arm anvil ([#12565](#12565)) ([e4bfbd1](e4bfbd1)) * compare Quoted by expanding interned values (noir-lang/noir#7602) ([1a3c112](1a3c112)) * Display causes but not stack trace in CLI error report (noir-lang/noir#7584) ([f13b729](f13b729)) * **experimental:** Fix execution of match expressions with multiple branches (noir-lang/noir#7570) ([1a3c112](1a3c112)) * fix a few cases where safety comment wasn't correctly identified (noir-lang/noir#7548) ([f13b729](f13b729)) * fix bbup and add CI ([#12541](#12541)) ([1b2604c](1b2604c)) * Fix the config ([#12513](#12513)) ([fb9fac6](fb9fac6)) * **frontend:** Restrict capturing mutable variable in lambdas (noir-lang/noir#7488) ([f13b729](f13b729)) * FunctionDefinition::as_typed_expr didn't work well for trait imp… (noir-lang/noir#7611) ([1a3c112](1a3c112)) * Log to `stderr` (noir-lang/noir#7585) ([f13b729](f13b729)) * **LSP:** references/rename only when underlying span has the correct… (noir-lang/noir#7598) ([8eb727c](8eb727c)) * make vk metadata actual witnesses ([#12459](#12459)) ([dada06f](dada06f)) * no fast deployments when the boot node needs to restart. ([#12557](#12557)) ([866582e](866582e)) * **node:** drop log level of handler not registered ([#12523](#12523)) ([cb7e42d](cb7e42d)) * override bb path in cli-wallet PXE config ([#12511](#12511)) ([0c3024e](0c3024e)) * publish-bb-mac.yml version replace ([#12554](#12554)) ([7e89dfb](7e89dfb)) * release bb-mac ([fac5fb5](fac5fb5)) * Revert "make vk metadata actual witnesses" ([#12534](#12534)) ([ed46a3c](ed46a3c)) * shift right overflow in ACIR with unknown var now returns zero (noir-lang/noir#7509) ([f13b729](f13b729)) * TokensPrettyPrinter was missing some spaces between tokens (noir-lang/noir#7607) ([1a3c112](1a3c112)) * yarn-project e2e bench ([#12547](#12547)) ([b40b904](b40b904)) ### Miscellaneous * add some extra tests (noir-lang/noir#7544) ([f13b729](f13b729)) * add underscore parameter documentation (noir-lang/noir#7562) ([1a3c112](1a3c112)) * add yaml aliases in .test_patterns.yml ([#12516](#12516)) ([3ee8d51](3ee8d51)) * address some frontend tests TODOs (noir-lang/noir#7554) ([f13b729](f13b729)) * addressing remaining feedback in PR 12182 ([#12494](#12494)) ([f733879](f733879)), closes [#12193](#12193) * bump `light-poseidon` (noir-lang/noir#7568) ([f13b729](f13b729)) * bump bb version to v0.77.0 (noir-lang/noir#7599) ([f13b729](f13b729)) * bump external pinned commits (noir-lang/noir#7561) ([f13b729](f13b729)) * bump external pinned commits (noir-lang/noir#7565) ([f13b729](f13b729)) * bump external pinned commits (noir-lang/noir#7581) ([f13b729](f13b729)) * bump external pinned commits (noir-lang/noir#7601) ([f13b729](f13b729)) * bump external pinned commits (noir-lang/noir#7618) ([fbaa634](fbaa634)) * bump ring to address advisory (noir-lang/noir#7619) ([fbaa634](fbaa634)) * Cleaner PXE ([#12515](#12515)) ([a69f416](a69f416)) * cleanup committing and masking utility ([#12514](#12514)) ([9f57048](9f57048)) * **cli:** exclude kind smoke test from flake list ([#12518](#12518)) ([778bfa6](778bfa6)) * **cli:** Forward `nargo execute` to `noir_artifact_cli` (noir-lang/noir#7406) ([f13b729](f13b729)) * convert `TraitMethodNotInScope` to error (noir-lang/noir#7427) ([fbaa634](fbaa634)) * explode aliases when looking up owners in `.test_patterns.yml` ([#12526](#12526)) ([2e0d791](2e0d791)) * fix trait import issues ([#12500](#12500)) ([fd9f145](fd9f145)) * Fix yarn install immutable issues ([#12539](#12539)) ([fb9ada3](fb9ada3)), closes [#12538](#12538) * More config defaults and forward p2p ports ([#12529](#12529)) ([2c45fb9](2c45fb9)) * **node:** return correct node version ([#12520](#12520)) ([5502901](5502901)) * **profiler:** Add option to only get the total sample count for the `execution-opcodes` command (noir-lang/noir#7578) ([f13b729](f13b729)) * put RcTracker as part of the DIE context (noir-lang/noir#7309) ([f13b729](f13b729)) * remove deprecated hash functions from stdlib (noir-lang/noir#7477) ([f13b729](f13b729)) * remove FileDiagnostic (noir-lang/noir#7546) ([f13b729](f13b729)) * remove merkle module from stdlib (noir-lang/noir#7582) ([f13b729](f13b729)) * Remove scope interpolation from env vars ([#12522](#12522)) ([70942e9](70942e9)) * remove U128 struct from stdlib (noir-lang/noir#7529) ([f13b729](f13b729)) * replace relative paths to noir-protocol-circuits ([f20c0dd](f20c0dd)) * replace relative paths to noir-protocol-circuits ([4365064](4365064)) * restore bb --version ([#12542](#12542)) ([ab13d43](ab13d43)) * restore method syntax on `get_storage_slot` calls ([#12532](#12532)) ([8e9f594](8e9f594)) * rm unused methods ([#12544](#12544)) ([ed1dbdc](ed1dbdc)) * some SSA improvements (noir-lang/noir#7588) ([f13b729](f13b729)) * **spartan:** kind test speedup ([#12478](#12478)) ([8ede7b1](8ede7b1)) * **ssa:** Turn the Brillig constraints check back on by default (noir-lang/noir#7404) ([f13b729](f13b729)) * track more critical libraries (noir-lang/noir#7604) ([f13b729](f13b729)) * update and lock AVM's lockfile ([#12533](#12533)) ([2babc50](2babc50)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).

TomAFrench · 2025-03-07T19:05:42Z

acvm-repo/acir/Cargo.toml

 [dev-dependencies]
 serde_json = "1.0"
 serde-reflection = "0.3.6"
 serde-generate = "0.25.1"
 fxhash.workspace = true
 criterion.workspace = true
 pprof.workspace = true
+num-bigint.workspace = true
+
+acir = { path = ".", features = ["arb"] } # Self to turn on `arb`.


Why are we having this crate depend on itself to turn on a feature flag? If we need to have this feature on all the time like this then we should just not have a feature flag.

Ah, it's just for testing.

It is for testing, yes, but I somehow again encountered the problem that if I just enabled this feature then noir-wasm didn't compile. I think it happened when it was like acir = { workspace = true, features = ["arb"] } instead of using path.

Originally I tried to make it work like this:

cfg_attr(any(test, feature = "arb"), derive(proptest_derive::Arbitrary))]

The problem is that half of the stuff that needs Arbitrary live in the brillig crate, so for that to kick in I either need to add a dev-dependency on that crate with arb, which can trigger the wasm problem, or I can rely on the arb feature of acir turning on brillig/arb.

TomAFrench · 2025-03-07T19:07:52Z

acvm-repo/acir/src/circuit/opcodes.rs

@@ -30,7 +31,8 @@ impl BlockType {

 #[allow(clippy::large_enum_variant)]
 #[derive(Clone, PartialEq, Eq, Serialize, Deserialize, Hash)]
-pub enum Opcode<F> {
+#[cfg_attr(feature = "arb", derive(proptest_derive::Arbitrary))]
+pub enum Opcode<F: AcirField> {


😬 Is this trait bound really necessary?

😮‍💨 seems like a really leaky solution to just needing a Debug implementation for Arbitrary

Yes, it's annoying. I thought I'm not adding any bound that won't be ultimately satisfied, so it's just making what would inevitably be true official, but it's certainly viral.

Use ciborium for serialization of Program

f2640c0

github-actions bot reviewed Feb 25, 2025

View reviewed changes

aakoshh added 2 commits February 25, 2025 14:49

fixup! Use ciborium for serialization of Program

348e43a

Compared bincode against CBOR

d9df721

aakoshh force-pushed the 7511-compare-program-ser branch from bf8a365 to d9df721 Compare February 25, 2025 14:49

aakoshh mentioned this pull request Feb 25, 2025

Backwards compatible Program serialisation #7511

Open

aakoshh added 2 commits February 25, 2025 15:45

Use FlexBuffers for serialization

948cd8e

Measure and display FlexBuffer sizes

67e98a9

aakoshh changed the title ~~test: Compare bincode against other serialisation formats~~ feat(experiment): Compare bincode against other serialisation formats Feb 25, 2025

aakoshh added 15 commits February 26, 2025 19:02

Define protobuf schema for Program

13a440f

Refactor package names

44c8afe

Merge remote-tracking branch 'origin/master' into 7511-compare-progra…

d89166f

…m-ser

Brillig opcodes

17654ca

Add ProtoReprF

ebb1369

Switch to ProtoSchema

08d17fb

BrilligOpcode encode

33a32b6

Split up conversions

f2fcc5c

Encoder for usize

0734118

All brillig encoding

a2494c2

Separate native. Work with IntoIterator

f96c33e

Move Expression to native

d4e8745

Finish encoding acir

36701d8

WIP

70deca7

Switch to protobuf serialization in Program

a66a9e7

aakoshh force-pushed the 7511-compare-program-ser branch from c14a35f to 52da1a6 Compare February 28, 2025 12:38

Bincode vs Protobuf

e21fccc

aakoshh force-pushed the 7511-compare-program-ser branch from 52da1a6 to e21fccc Compare February 28, 2025 13:04

Merge remote-tracking branch 'origin/master' into 7511-compare-progra…

27514bb

…m-ser

aakoshh changed the title ~~feat(experiment): Compare bincode against other serialisation formats~~ feat: Compare bincode against CBOR, FlexBuffers and Protobuf - implement Protobuf Mar 4, 2025

aakoshh changed the title ~~feat: Compare bincode against CBOR, FlexBuffers and Protobuf - implement Protobuf~~ feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best Mar 4, 2025

aakoshh marked this pull request as ready for review March 4, 2025 15:31

aakoshh marked this pull request as draft March 4, 2025 15:31

Use bincode in WitnessStack and WitnessMap again

e8df457

aakoshh marked this pull request as ready for review March 4, 2025 16:07

aakoshh requested a review from a team March 4, 2025 16:17

aakoshh added 4 commits March 5, 2025 10:56

Fix docstring

da0e82a

Add readme for the scripts

b9daf52

Merge branch 'master' into 7511-compare-program-ser

4281e4b

Add GITHUB_TOKEN for prost_prebuilt

7ecbf3d

aakoshh mentioned this pull request Mar 5, 2025

feat: C++ bindings for protobuf #7590

Closed

5 tasks

asterite reviewed Mar 5, 2025

View reviewed changes

acvm-repo/acir/src/lib.rs Outdated Show resolved Hide resolved

Comment about why unsafe is needed

93212ec

asterite approved these changes Mar 5, 2025

View reviewed changes

aakoshh added this pull request to the merge queue Mar 6, 2025

Merged via the queue into master with commit d047779 Mar 6, 2025
105 checks passed

aakoshh deleted the 7511-compare-program-ser branch March 6, 2025 09:50

noirwhal mentioned this pull request Mar 6, 2025

chore: Release Noir(1.0.0-beta.4) #7475

Open

AztecBot mentioned this pull request Mar 6, 2025

feat: Sync from noir AztecProtocol/aztec-packages#12524

Merged

AztecBot mentioned this pull request Mar 6, 2025

chore(master): release 0.78.0 AztecProtocol/aztec-packages#12508

Merged

TomAFrench reviewed Mar 7, 2025

View reviewed changes

TomAFrench mentioned this pull request Mar 7, 2025

chore: remove unnecessary trait bounds #7635

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best #7513

feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best #7513

aakoshh commented Feb 25, 2025 •

edited

Loading

github-actions bot left a comment •

edited

Loading

aakoshh commented Feb 25, 2025 •

edited

Loading

aakoshh commented Feb 25, 2025 •

edited

Loading

aakoshh commented Feb 28, 2025 •

edited

Loading

asterite left a comment

TomAFrench Mar 7, 2025

TomAFrench Mar 7, 2025

aakoshh Mar 7, 2025

TomAFrench Mar 7, 2025

TomAFrench Mar 7, 2025

aakoshh Mar 7, 2025

feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best #7513

feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best #7513

Conversation

aakoshh commented Feb 25, 2025 • edited Loading

Description

Problem*

Summary*

Additional Context

Compiling contracts

Baseline

Comparing bytecode

Documentation*

PR Checklist*

github-actions bot left a comment • edited Loading

Choose a reason for hiding this comment

⚠️ Performance Alert ⚠️

aakoshh commented Feb 25, 2025 • edited Loading

aakoshh commented Feb 25, 2025 • edited Loading

aakoshh commented Feb 28, 2025 • edited Loading

asterite left a comment

Choose a reason for hiding this comment

TomAFrench Mar 7, 2025

Choose a reason for hiding this comment

TomAFrench Mar 7, 2025

Choose a reason for hiding this comment

aakoshh Mar 7, 2025

Choose a reason for hiding this comment

TomAFrench Mar 7, 2025

Choose a reason for hiding this comment

TomAFrench Mar 7, 2025

Choose a reason for hiding this comment

aakoshh Mar 7, 2025

Choose a reason for hiding this comment

aakoshh commented Feb 25, 2025 •

edited

Loading

github-actions bot left a comment •

edited

Loading

aakoshh commented Feb 25, 2025 •

edited

Loading

aakoshh commented Feb 25, 2025 •

edited

Loading

aakoshh commented Feb 28, 2025 •

edited

Loading