feat: Barretenberg C++ binary overhaul (#11459)

Overhaul the Barretenberg binary and its API.
- Breaks up bb main into different files organized by proving system /
IVC scheme.
- Make UltraHonk conform to the new API interface introduced earlier for
Client IVC.
- Refines the API a bit.
- Introduces [CLI11](https://github.com/CLIUtils/CLI11) to: provide help
/ documentation; validate opts (options can be required, exlusive of
each other, validated against predicates like "path exists" or "string
is in list"); also allows for easy environment variable aliasing.

This could definitely use some more a help. 
 - Lots of documentation needed
- Defaults are set in a weird and inconsistent way and that information
isn't included in the documentation.
- The help menus are perhaps too verbose. Subcommands can't inherit
options or flags so we end up repeating.
- Empty string cannot be passed and parsed to a "nothing argument" which
can lead to frustrating debugging...
 - Little option validation is actually implemented.
 - Deprecated options aren't noted but they could be.

It was requested that the default change from UltraPlonk to UltraHonk,
but we get rid of a default set of commands altogether. As a workaround,
we can have users set `BB_SCHEME=ultra_honk`.

Newly created issues:
#1252,
#1253,
#1254,
#1255,
#1256,
#1257,
#1258,
#1259

Resolves #1260

NB the line count is large because 1) CLI11 is a single 11k-line header;
2) I moved a lot of functions and some git mvs didn't show up as such.
Main new code is api_ultra_honk.hpp.

---------

Co-authored-by: ludamad <adam.domurad@gmail.com>

README.md

@@ -30,16 +30,15 @@ Barretenberg (or `bb` for short) is an optimized elliptic curve library for the
       - [Testing locally in docker](#testing-locally-in-docker)
   - [Docs Build](#docs-build)
   - [Benchmarks](#benchmarks)
-    - [x86\_64](#x86_64)
+    - [x86_64](#x86_64)
     - [WASM](#wasm)
     - [How to run](#how-to-run)
   - [Debugging](#debugging)
     - [Debugging Verification Failures](#debugging-verifification-failures)
     - [Improving LLDB Debugging](#improving-lldb-debugging)
     - [Using Tracy to Profile Memory/CPU](#using-tracy-to-profile-memorycpu)
 
-> [!CAUTION]
-> **This code is highly experimental, use at your own risk!**
+> [!CAUTION] > **This code is highly experimental, use at your own risk!**
 
 ## Installation
 
@@ -66,14 +65,14 @@ All available `bb` commands:
 Prove the valid execution of your program:
 
 ```bash
-bb prove_ultra_honk -b ./target/hello_world.json -w ./target/witness-name.gz -o ./target/proof
+bb prove --scheme ultra_honk -b ./target/hello_world.json -w ./target/witness-name.gz -o ./target/proof
 ```
 
 You can then compute the verification key for your Noir program and verify the proof:
 
 ```bash
-bb write_vk_ultra_honk -b ./target/hello_world.json -o ./target/vk
-bb verify_ultra_honk -k ./target/vk -p ./target/proof
+bb write_vk --scheme ultra_honk -b ./target/hello_world.json -o ./target/vk
+bb verify --scheme ultra_honk -k ./target/vk -p ./target/proof
 
 ```
 
@@ -83,7 +82,7 @@ If successful, the verification will complete in silence.
 
 The usage with MegaHonk is similar to the above UltraHonk. Refer to all the available `bb` commands, using the `bb <command>_mega_honk` syntax.
 
->[!WARNING]
+> [!WARNING]
 > MegaHonk generates insecure recursion circuits when Goblin recursive verifiers are not present.
 
 ### Solidity verifier
@@ -93,21 +92,20 @@ Barretenberg can generate a smart contract that verifies proofs in Solidity (i.e
 First, prove the valid execution of your Noir program and export the verification key:
 
 ```bash
-bb prove_ultra_keccak_honk -b ./target/hello_world.json -w ./target/witness-name.gz -o ./target/proof
-bb write_vk_ultra_honk -b ./target/hello_world.json -o ./target/vk
+bb prove --sceme ultra_honk -b ./target/hello_world.json -w ./target/witness-name.gz -o ./target/proof
+bb write_vk --scheme ultra_honk -b ./target/hello_world.json -o ./target/vk
 ```
 
-> [!IMPORTANT]
-> `prove_ultra_keccak_honk` is used to generate UltraHonk proofs with Keccak hashes, making them gas-efficient. `prove_ultra_honk` in comparison generates proofs with Poseidon hashes, more efficient in recursions but not on-chain verifications.
+> [!IMPORTANT] > `prove --scheme ultra_honk --oracle_hash keccak` is used to generate UltraHonk proofs with Keccak hashes, making them gas-efficient. `prove --scheme ultra_honk` in comparison generates proofs with Poseidon hashes, more efficient in recursions but not on-chain verifications.
 
 You can now use the verification key to generate a Solidity verifier contract:
 
 ```bash
-bb contract_ultra_honk -k ./target/vk -c $CRS_PATH -b ./target/hello_world.json -o ./target/Verifier.sol
+bb write_contract --scheme ultra_honk -k ./target/vk -c $CRS_PATH -b ./target/hello_world.json -o ./target/Verifier.sol
 ```
 
->[!CAUTION]
-> Solidity verifier contracts are work-in-progress. Expect significant optimizations and breaking changes, and *do NOT use it in production!*
+> [!CAUTION]
+> Solidity verifier contracts are work-in-progress. Expect significant optimizations and breaking changes, and _do NOT use it in production!_
 
 ## Development
 
@@ -270,7 +268,7 @@ Alternatively you can build separate test binaries, e.g. honk_tests or numeric_t
 
 Code is formatted using `clang-format` and the `./cpp/format.sh` script which is called via a git pre-commit hook.
 
->[!TIP]
+> [!TIP]
 > A default configuration for VS Code is provided by the file [`barretenberg.code-workspace`](barretenberg.code-workspace). These settings can be overridden by placing configuration files in `.vscode/`.
 > If you've installed the C++ Vscode extension, configure it to format on save!
 
@@ -410,7 +408,7 @@ Usage instructions:
 Caveats:
 
 - This works best for code that is not overly generic, i.e. where just the sequence of function calls carries a lot of information. It is possible to tag extra data along with the stack trace, this can be done as a followup, please leave feedback if desired.
-- There are certain functions like `assert_equals` that can cause gates that occur *before* them to fail. If this would be useful to automatically report, please leave feedback.
+- There are certain functions like `assert_equals` that can cause gates that occur _before_ them to fail. If this would be useful to automatically report, please leave feedback.
 
 Example:
 

acir_tests/bootstrap.sh

@@ -36,8 +36,9 @@ function build {
     echo "Regenerating verify_honk_proof and verify_rollup_honk_proof recursive inputs."
     local bb=$(realpath ../cpp/build/bin/bb)
     (cd ./acir_tests/assert_statement && \
-      $bb write_recursion_inputs_ultra_honk -b ./target/program.json -o ../verify_honk_proof --recursive && \
-      $bb write_recursion_inputs_rollup_honk -b ./target/program.json -o ../verify_rollup_honk_proof --recursive)
+      # TODO(https://github.com/AztecProtocol/barretenberg/issues/1253) Deprecate command and construct TOML (e.g., via yq or via conversion from a JSON)
+      $bb OLD_API write_recursion_inputs_ultra_honk -b ./target/program.json -o ../verify_honk_proof --recursive && \
+      $bb OLD_API write_recursion_inputs_ultra_honk --ipa_accumulation -b ./target/program.json -o ../verify_rollup_honk_proof --recursive)
 
     cache_upload $tests_tar acir_tests
   fi
@@ -69,9 +70,9 @@ function test_cmds {
 # Paths are all relative to the repository root.
 function test_cmds_internal {
   local plonk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
-    grep -vE 'verify_honk_proof|double_verify_honk_proof|verify_rollup_honk_proof')
+    grep -vE 'verify_honk_proof|double_verify_honk_proof|verify_rollup_honk_proof|fold')
   local honk_tests=$(find ./acir_tests -maxdepth 1 -mindepth 1 -type d | \
-    grep -vE 'single_verify_proof|double_verify_proof|double_verify_nested_proof|verify_rollup_honk_proof')
+    grep -vE 'single_verify_proof|double_verify_proof|double_verify_nested_proof|verify_rollup_honk_proof|fold')
 
   local run_test=$(realpath --relative-to=$root ./run_test.sh)
   local run_test_browser=$(realpath --relative-to=$root ./run_test_browser.sh)
@@ -99,25 +100,23 @@ function test_cmds_internal {
   # echo ecdsa_secp256r1_3x through bb.js on node to check 256k support.
   echo BIN=$bbjs_bin FLOW=prove_then_verify $run_test ecdsa_secp256r1_3x
   # echo the prove then verify flow for UltraHonk. This makes sure we have the same circuit for different witness inputs.
-  echo BIN=$bbjs_bin SYS=ultra_honk FLOW=prove_then_verify $run_test 6_array
-  # echo a single arbitrary test not involving recursion through bb.js for MegaHonk
-  echo BIN=$bbjs_bin SYS=mega_honk FLOW=prove_and_verify $run_test 6_array
+  echo BIN=$bbjs_bin SYS=ultra_honk_deprecated FLOW=prove_then_verify $run_test 6_array
   # echo 1_mul through bb.js build, all_cmds flow, to test all cli args.
   echo BIN=$bbjs_bin FLOW=all_cmds $run_test 1_mul
 
   # barretenberg-acir-tests-bb:
-  # Fold and verify an ACIR program stack using ClientIvc, recursively verify as part of the Tube circuit and produce and verify a Honk proof
+  # Fold and verify an ACIR program stack using ClientIVC, recursively verify as part of the Tube circuit and produce and verify a Honk proof
   echo FLOW=prove_then_verify_tube $run_test 6_array
   # echo 1_mul through native bb build, all_cmds flow, to test all cli args.
-  echo FLOW=all_cmds $run_test 1_mul
+  echo NATIVE=1 FLOW=all_cmds $run_test 1_mul
 
   # barretenberg-acir-tests-bb-ultra-plonk:
   # Exclude honk tests.
   for t in $plonk_tests; do
-    echo FLOW=prove_then_verify $run_test $(basename $t)
+    echo SYS=ultra_plonk_deprecated FLOW=prove_then_verify $run_test $(basename $t)
   done
-  echo FLOW=prove_then_verify RECURSIVE=true $run_test assert_statement
-  echo FLOW=prove_then_verify RECURSIVE=true $run_test double_verify_proof
+  echo SYS=ultra_plonk_deprecated FLOW=prove_then_verify RECURSIVE=true $run_test assert_statement
+  echo SYS=ultra_plonk_deprecated FLOW=prove_then_verify RECURSIVE=true $run_test double_verify_proof
 
   # barretenberg-acir-tests-bb-ultra-honk:
   # Exclude plonk tests.
@@ -126,15 +125,16 @@ function test_cmds_internal {
   done
   echo SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true $run_test assert_statement
   echo SYS=ultra_honk FLOW=prove_then_verify RECURSIVE=true $run_test double_verify_honk_proof
-  echo SYS=ultra_honk FLOW=prove_and_verify_program $run_test merkle_insert
-  echo SYS=ultra_rollup_honk FLOW=prove_then_verify $run_test verify_rollup_honk_proof
+  echo SYS=ultra_honk FLOW=prove_then_verify HASH=keccak $run_test assert_statement
+  echo SYS=ultra_honk FLOW=prove_then_verify ROLLUP=true $run_test verify_rollup_honk_proof
 
   # barretenberg-acir-tests-bb-client-ivc:
   echo FLOW=prove_then_verify_client_ivc $run_test 6_array
   echo FLOW=prove_then_verify_client_ivc $run_test databus
   echo FLOW=prove_then_verify_client_ivc $run_test databus_two_calldata
 }
 
+# TODO(https://github.com/AztecProtocol/barretenberg/issues/1254): More complete testing, including failure tests
 function bench {
   # TODO: Move to scripts dir along with run_test.sh.
   LOG_FILE=bench-acir.jsonl ./bench_acir_tests.sh

acir_tests/flows/all_cmds.sh

@@ -4,19 +4,23 @@ set -eu
 VFLAG=${VERBOSE:+-v}
 BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH $VFLAG"
+# the commands called here are subcommands of the OLD_API command in the native bb binary,
+# but no such refactoring was done to the node binary. This is because the node binary is
+# deprecated and UltraPlonk is also deprecated.
+MAYBE_OLD_API=${NATIVE:+OLD_API}
 
 # Test we can perform the proof/verify flow.
-$BIN gates $FLAGS $BFLAG > /dev/null
-$BIN prove -o proof $FLAGS $BFLAG
-$BIN write_vk -o vk $FLAGS $BFLAG
-$BIN write_pk -o pk $FLAGS $BFLAG
-$BIN verify -k vk -p proof $FLAGS
+$BIN $MAYBE_OLD_API gates $FLAGS $BFLAG > /dev/null
+$BIN $MAYBE_OLD_API prove -o proof $FLAGS $BFLAG
+$BIN $MAYBE_OLD_API write_vk -o vk $FLAGS $BFLAG
+$BIN $MAYBE_OLD_API write_pk -o pk $FLAGS $BFLAG
+$BIN $MAYBE_OLD_API verify -k vk -p proof $FLAGS
 
 # Check supplemental functions.
 # Grep to determine success.
-$BIN contract -k vk $BFLAG -o - | grep "Verification Key Hash" > /dev/null
+$BIN $MAYBE_OLD_API contract -k vk $BFLAG -o - | grep "Verification Key Hash" > /dev/null
 # Use jq to determine success, and also check result not empty.
-OUTPUT=$($BIN proof_as_fields -k vk -p proof -o - | jq .)
+OUTPUT=$($BIN $MAYBE_OLD_API proof_as_fields -p proof -k vk -o - | jq .)
 [ -n "$OUTPUT" ] || exit 1
-OUTPUT=$($BIN vk_as_fields -k vk -o - | jq .)
+OUTPUT=$($BIN $MAYBE_OLD_API vk_as_fields -k vk -o - | jq .)
 [ -n "$OUTPUT" ] || exit 1

acir_tests/flows/prove_and_verify.sh

@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/bash
 # prove_and_verify produces no output, so is parallel safe.
 set -eu
 
@@ -11,7 +11,11 @@ case ${SYS:-} in
     ;;
   "client_ivc")
     cmd=prove_and_verify
-    flags+=" --scheme client_ivc --input_type ${INPUT_TYPE:-compiletime_stack}"
+    flags+=" --scheme client_ivc ${INPUT_TYPE:---input_type compiletime_stack}"
+    ;;
+  "ultra_honk")
+    cmd=prove_and_verify
+    flags+=" --scheme ultra_honk ${INPUT_TYPE:---input_type compiletime_stack}"
     ;;
   *)
     cmd=prove_and_verify_$SYS

acir_tests/flows/prove_then_verify.sh

@@ -5,7 +5,6 @@ set -eu
 BFLAG="-b ./target/program.json"
 FLAGS="-c $CRS_PATH ${VERBOSE:+-v}"
 [ "${RECURSIVE}" = "true" ] && FLAGS+=" --recursive"
-[ -n "${SYS:-}" ] && SYS="_$SYS" || SYS=""
 
 # TODO: Use this when client ivc support write_vk. Currently it keeps its own flow.
 # case ${SYS:-} in
@@ -16,7 +15,7 @@ FLAGS="-c $CRS_PATH ${VERBOSE:+-v}"
 #   "client_ivc")
 #     prove_cmd=prove
 #     verify_cmd=verify
-#     flags+=" --scheme client_ivc --input_type ${INPUT_TYPE:-compiletime_stack}"
+#     flags+=" --scheme client_ivc --input_type ${INPUT_TYPE:-single_circuit}"
 #     ;;
 #   *)
 #     prove_cmd=prove_$SYS
@@ -27,6 +26,40 @@ FLAGS="-c $CRS_PATH ${VERBOSE:+-v}"
 # Test we can perform the proof/verify flow.
 # This ensures we test independent pk construction through real/garbage witness data paths.
 # We use process substitution pipes to avoid temporary files, which need cleanup, and can collide with parallelism.
-$BIN verify$SYS $FLAGS \
-    -k <($BIN write_vk$SYS -o - $FLAGS $BFLAG) \
-    -p <($BIN prove$SYS -o - $FLAGS $BFLAG)
+
+case ${SYS:-} in
+  "")
+    # Deprecated; used for old node cli
+    [ -n "${SYS:-}" ] && SYS="_$SYS" || SYS=""
+    $BIN verify$SYS $FLAGS \
+        -k <($BIN write_vk$SYS -o - $FLAGS $BFLAG) \
+        -p <($BIN prove$SYS -o - $FLAGS $BFLAG)
+  ;;
+  "ultra_plonk_deprecated")
+    # This used to be the default but now it's on its way out.
+    $BIN OLD_API verify $FLAGS \
+        -k <($BIN OLD_API write_vk -o - $FLAGS $BFLAG) \
+        -p <($BIN OLD_API prove -o - $FLAGS $BFLAG)
+  ;;
+  "ultra_honk")
+    FLAGS+=" --scheme $SYS --oracle_hash ${HASH:-poseidon2}"
+    [ "${ROLLUP:-false}" = "true" ] && FLAGS+=" --ipa_accumulation"
+    [ "${RECURSIVE}" = "true" ] && FLAGS+=" --init_kzg_accumulator"
+    $BIN verify $FLAGS \
+        -k <($BIN write_vk $FLAGS $BFLAG -o - ) \
+        -p <($BIN prove $FLAGS $BFLAG -o - )
+  ;;
+  "ultra_honk_deprecated")
+    # deprecated flow is necessary until we finish C++ api refactor and then align ts api
+    SYS_DEP=_ultra_honk
+    $BIN verify$SYS_DEP $FLAGS \
+        -k <($BIN write_vk$SYS_DEP -o - $FLAGS $BFLAG) \
+        -p <($BIN prove$SYS_DEP -o - $FLAGS $BFLAG)
+    ;;
+  *)
+    [ -n "${SYS:-}" ] && SYS="_$SYS" || SYS=""
+    $BIN verify$SYS $FLAGS \
+        -k <($BIN write_vk$SYS -o - $FLAGS $BFLAG) \
+        -p <($BIN prove$SYS -o - $FLAGS $BFLAG)
+    ;;
+esac

acir_tests/flows/prove_then_verify_client_ivc.sh

@@ -1,6 +1,6 @@
 #!/bin/bash
 # Create intermediate state in a directory. Uses a temp dir to ensure parallel safe and cleansup on exit.
-set -eu
+set -eux
 
 CRS_PATH=${CRS_PATH:-$HOME/.bb-crs}
 BIN=$(realpath ${BIN:-../cpp/build/bin/bb})
@@ -10,7 +10,7 @@ BIN=$(realpath ${BIN:-../cpp/build/bin/bb})
 outdir=$(mktemp -d)
 trap "rm -rf $outdir" EXIT
 
-flags="--scheme client_ivc -c $CRS_PATH ${VERBOSE:+-v} -o $outdir"
+flags="--scheme client_ivc -c $CRS_PATH ${VERBOSE:+-v}"
 
-$BIN prove $flags -b ./target/program.json --input_type ${INPUT_TYPE:-compiletime_stack}
-$BIN verify $flags
+$BIN prove $flags -b ./target/program.json ${INPUT_TYPE:---input_type compiletime_stack} -o $outdir
+$BIN verify $flags -p $outdir/proof -k $outdir/vk

acir_tests/flows/prove_then_verify_tube.sh

@@ -13,6 +13,7 @@ trap "rm -rf $outdir" EXIT
 
 flags="-c $CRS_PATH ${VERBOSE:+-v} -o $outdir"
 
-$BIN write_arbitrary_valid_proof_and_vk_to_file --scheme client_ivc $flags
+# TODO(https://github.com/AztecProtocol/barretenberg/issues/1252): deprecate in favor of normal proving flow
+$BIN OLD_API write_arbitrary_valid_client_ivc_proof_and_vk_to_file $flags
 $BIN prove_tube $flags
-$BIN verify_tube $flags
+$BIN verify_tube $flags

acir_tests/flows/sol.sh

@@ -10,10 +10,10 @@ export PROOF_AS_FIELDS="$PWD/sol_proof_fields.json"
 export VK="$PWD/sol_vk"
 
 # Create a proof, write the solidity contract, write the proof as fields in order to extract the public inputs
-$BIN prove -o $PROOF $FLAGS
-$BIN write_vk  -o $VK $FLAGS
-$BIN proof_as_fields -k $VK $FLAGS -p $PROOF
-$BIN contract -k $VK $FLAGS $BFLAG -o Key.sol
+$BIN OLD_API prove -o $PROOF $FLAGS
+$BIN OLD_API write_vk  -o $VK $FLAGS
+$BIN OLD_API proof_as_fields -k $VK $FLAGS -p $PROOF
+$BIN OLD_API contract -k $VK $FLAGS $BFLAG -o Key.sol
 
 # Export the paths to the environment variables for the js test runner
 export KEY_PATH="$PWD/Key.sol"

acir_tests/flows/sol_honk.sh

@@ -1,28 +1,34 @@
 #!/bin/sh
-set -eux
+set -eu
 
 VFLAG=${VERBOSE:+-v}
 BFLAG="-b ./target/program.json"
-FLAGS="-c $CRS_PATH $VFLAG"
+FLAGS="-c $CRS_PATH $VFLAG --scheme ultra_honk"
+[ "${RECURSIVE}" = "true" ] && FLAGS+=" --recursive"
+PROVE_FLAGS="$FLAGS $BFLAG --oracle_hash keccak --output_data bytes_and_fields --output_content proof_and_vk --input_type single_circuit"
+VERIFY_FLAGS="$FLAGS --oracle_hash keccak"
 
-export PROOF="$PWD/sol_honk_proof"
-export PROOF_AS_FIELDS="$PWD/sol_honk_proof_fields.json"
-export VK="$PWD/sol_honk_vk"
+outdir=$(mktemp -d)
+trap "rm -rf $outdir" EXIT
+
+# Export the paths to the environment variables for the js test runner
+export PROOF="$outdir/proof"
+export PROOF_AS_FIELDS="$outdir/proof_fields.json"
+export VK="$outdir/vk"
+export VERIFIER_CONTRACT="$outdir/Verifier.sol"
 
 # Create a proof, write the solidity contract, write the proof as fields in order to extract the public inputs
-$BIN prove_ultra_keccak_honk -o $PROOF $FLAGS $BFLAG
-$BIN write_vk_ultra_keccak_honk -o $VK $FLAGS $BFLAG
-$BIN verify_ultra_keccak_honk -k $VK -p $PROOF $FLAGS
-$BIN proof_as_fields_honk $FLAGS -p $PROOF -o $PROOF_AS_FIELDS
-$BIN contract_ultra_honk -k $VK $FLAGS -o Verifier.sol
+$BIN prove $PROVE_FLAGS -o $outdir
+$BIN verify $VERIFY_FLAGS -k $VK -p $PROOF
+$BIN write_contract $FLAGS -k $VK -o $VERIFIER_CONTRACT
 
 # Export the paths to the environment variables for the js test runner
-export VERIFIER_PATH="$PWD/Verifier.sol"
+export VERIFIER_PATH="$outdir/Verifier.sol"
 export TEST_PATH=$(realpath "../../sol-test/HonkTest.sol")
 export TESTING_HONK="true"
 
 # Use solcjs to compile the generated key contract with the template verifier and test contract
 # index.js will start an anvil, on a random port
 # Deploy the verifier then send a test transaction
-export TEST_NAME=$(basename $PWD)
+export TEST_NAME=$(basename $outdir)
 node ../../sol-test/src/index.js