Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cli): Log and replay oracle transcript #7417

Merged
merged 86 commits into from
Mar 5, 2025

Conversation

aakoshh
Copy link
Contributor

@aakoshh aakoshh commented Feb 18, 2025

Description

Problem*

Resolves #7379 and #7382

Summary*

  • Added NARGO_TEST_FOREIGN_CALL_LOG to nargo test which can be used to log the foreign calls made by a test either to stdout or a file. It only makes sense with a single test. The format of the log is JSON Line with a call and response as it appears in the JSON-RPC.
  • Added a LoggingForeignCallExecutor that prints foreign calls to a transcript log.
  • Added a ReplayForeignCallExecutor that deserialises the transcript, and implements the handler by popping off the next call, comparing the logged function name and inputs, rejecting any unexpected call to avoid confusion, then returns the logged response.
  • Added --oracle-file to noir-execute and nargo execute to be able to point at the transcript
  • Changed noir-execute to use the ReplayForeignCallExecutor as the final layer in the foreign call executor stack, if the file is present.
  • Added examples/oracle_transcript based on test_programs/noir_test_success/mock_oracle, which calls nargo test to capture a transcript and verify the expected input/output, then nargo execute with the transcript file to verify the expected return value in Prover.toml. The oracle file is checked in for reference.
  • Repeats the same call with noir-execute to demonstrate how it's used.
  • Build the noir-execute artifact on CI to be able to use it in the test script.

Additional Context

Here's a link to show the new example passing on CI: https://github.com/noir-lang/noir/actions/runs/13394747019/job/37411091682

The format in this PR, or something similar, will be the one used in AztecProtocol/aztec-packages#11855

Documentation*

Check one:

  • No documentation needed.
  • Documentation included in this PR.
  • [For Experimental Features] Documentation to be submitted in a separate PR.

PR Checklist*

  • I have tested the changes locally.
  • I have formatted the changes with Prettier and/or cargo fmt on default settings.

@asterite
Copy link
Collaborator

Added NARGO_TEST_FOREIGN_CALL_LOG to nargo test which can be used to log the foreign calls made by a test either to stdout or a file. It only makes sense with a single test.

I'm about to start reviewing this PR, but why does it only make sense with a single test? I know oracle calls from multiple tests might lead to different results, but I was wondering if we could key them under the test name.

Copy link
Collaborator

@asterite asterite left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! I guess, if it's possible, we could support running multiple tests in a future PR (is it an error right now if you run it with multiple tests?)

@aakoshh
Copy link
Contributor Author

aakoshh commented Feb 19, 2025

I'm about to start reviewing this PR, but why does it only make sense with a single test?

Only because I didn't spend any effort to delineate them:

  • if we set NARGO_TEST_FOREIGN_CALL_LOG=stdout then it will print things as it goes
  • if we set NARGO_TEST_FOREIGN_CALL_LOG=path/to/file.jsonl then the last test will overwrite the previous ones
  • there is no option to log to a string and print it to the stdout as one unit like you are doing it with the test result reporter for println collected during execution

@aakoshh
Copy link
Contributor Author

aakoshh commented Feb 19, 2025

I didn't want to change the signature of nargo::ops::run_test to add another PrintOutput for these, and also didn't want to have to copy paste from the terminal into a file, but if we think this is useful, or should be displayed along with the println output of the test, I'm happy to revisit. Currently it was just an idea for getting something to work with.

Base automatically changed from 7380-nargo-exec-fwd to master February 28, 2025 11:45
@aakoshh aakoshh requested a review from TomAFrench February 28, 2025 11:55
@aakoshh
Copy link
Contributor Author

aakoshh commented Mar 3, 2025

The TypeScript side of the transaction dump capturing the transcript can be seen in AztecProtocol/aztec-packages#12148

@aakoshh aakoshh added this pull request to the merge queue Mar 5, 2025
Merged via the queue into master with commit 95ac7a8 Mar 5, 2025
103 checks passed
@aakoshh aakoshh deleted the 7379-exec-oracle-transcript branch March 5, 2025 19:44
AztecBot added a commit to AztecProtocol/aztec-packages that referenced this pull request Mar 6, 2025
chore!: bump bb version to v0.77.0 (noir-lang/noir#7599)
chore: bump external pinned commits (noir-lang/noir#7601)
feat(cli): Log and replay oracle transcript (noir-lang/noir#7417)
chore: some SSA improvements (noir-lang/noir#7588)
chore(profiler): Add option to only get the total sample count for the `execution-opcodes` command (noir-lang/noir#7578)
feat(experimental): Issue errors for unreachable match branches (noir-lang/noir#7556)
fix: Log to `stderr` (noir-lang/noir#7585)
chore!: remove merkle module from stdlib (noir-lang/noir#7582)
fix: Display causes but not stack trace in CLI error report (noir-lang/noir#7584)
chore: bump `light-poseidon` (noir-lang/noir#7568)
chore: bump external pinned commits (noir-lang/noir#7581)
chore!: remove deprecated hash functions from stdlib (noir-lang/noir#7477)
fix(frontend)!: Restrict capturing mutable variable in lambdas (noir-lang/noir#7488)
feat: perform constant sha256 compressions at compile-time (noir-lang/noir#7566)
chore: bump external pinned commits (noir-lang/noir#7565)
chore(ssa): Turn the Brillig constraints check back on by default (noir-lang/noir#7404)
chore: bump external pinned commits (noir-lang/noir#7561)
chore: address some frontend tests TODOs (noir-lang/noir#7554)
fix: shift right overflow in ACIR with unknown var now returns zero (noir-lang/noir#7509)
chore(cli): Forward `nargo execute` to `noir_artifact_cli` (noir-lang/noir#7406)
feat: Support `<Type as Trait>::method` in expressions (noir-lang/noir#7551)
chore: remove FileDiagnostic (noir-lang/noir#7546)
chore: add some extra tests (noir-lang/noir#7544)
fix: fix a few cases where safety comment wasn't correctly identified (noir-lang/noir#7548)
chore!: remove U128 struct from stdlib (noir-lang/noir#7529)
feat: simplify simple conditionals for brillig (noir-lang/noir#7205)
chore: put RcTracker as part of the DIE context (noir-lang/noir#7309)
AztecBot added a commit to AztecProtocol/aztec-packages that referenced this pull request Mar 6, 2025
chore!: bump bb version to v0.77.0 (noir-lang/noir#7599)
chore: bump external pinned commits (noir-lang/noir#7601)
feat(cli): Log and replay oracle transcript (noir-lang/noir#7417)
chore: some SSA improvements (noir-lang/noir#7588)
chore(profiler): Add option to only get the total sample count for the `execution-opcodes` command (noir-lang/noir#7578)
feat(experimental): Issue errors for unreachable match branches (noir-lang/noir#7556)
fix: Log to `stderr` (noir-lang/noir#7585)
chore!: remove merkle module from stdlib (noir-lang/noir#7582)
fix: Display causes but not stack trace in CLI error report (noir-lang/noir#7584)
chore: bump `light-poseidon` (noir-lang/noir#7568)
chore: bump external pinned commits (noir-lang/noir#7581)
chore!: remove deprecated hash functions from stdlib (noir-lang/noir#7477)
fix(frontend)!: Restrict capturing mutable variable in lambdas (noir-lang/noir#7488)
feat: perform constant sha256 compressions at compile-time (noir-lang/noir#7566)
chore: bump external pinned commits (noir-lang/noir#7565)
chore(ssa): Turn the Brillig constraints check back on by default (noir-lang/noir#7404)
chore: bump external pinned commits (noir-lang/noir#7561)
chore: address some frontend tests TODOs (noir-lang/noir#7554)
fix: shift right overflow in ACIR with unknown var now returns zero (noir-lang/noir#7509)
chore(cli): Forward `nargo execute` to `noir_artifact_cli` (noir-lang/noir#7406)
feat: Support `<Type as Trait>::method` in expressions (noir-lang/noir#7551)
chore: remove FileDiagnostic (noir-lang/noir#7546)
chore: add some extra tests (noir-lang/noir#7544)
fix: fix a few cases where safety comment wasn't correctly identified (noir-lang/noir#7548)
chore!: remove U128 struct from stdlib (noir-lang/noir#7529)
feat: simplify simple conditionals for brillig (noir-lang/noir#7205)
chore: put RcTracker as part of the DIE context (noir-lang/noir#7309)
TomAFrench added a commit that referenced this pull request Mar 6, 2025
* master: (65 commits)
  feat: Compare bincode to CBOR, FlexBuffers and Protobuf - implement best (#7513)
  chore: track more critical libraries (#7604)
  chore!: bump bb version to v0.77.0 (#7599)
  chore: bump external pinned commits (#7601)
  feat(cli): Log and replay oracle transcript (#7417)
  chore: some SSA improvements (#7588)
  chore(profiler): Add option to only get the total sample count for the `execution-opcodes` command (#7578)
  feat(experimental): Issue errors for unreachable match branches (#7556)
  fix: Log to `stderr` (#7585)
  chore!: remove merkle module from stdlib (#7582)
  fix: Display causes but not stack trace in CLI error report (#7584)
  chore: bump `light-poseidon` (#7568)
  chore: bump external pinned commits (#7581)
  chore!: remove deprecated hash functions from stdlib (#7477)
  fix(frontend)!: Restrict capturing mutable variable in lambdas (#7488)
  feat: perform constant sha256 compressions at compile-time (#7566)
  chore: bump external pinned commits (#7565)
  chore(ssa): Turn the Brillig constraints check back on by default (#7404)
  chore: bump external pinned commits (#7561)
  chore: address some frontend tests TODOs (#7554)
  ...
rahul-kothari pushed a commit to AztecProtocol/aztec-packages that referenced this pull request Mar 7, 2025
🤖 I have created a new Aztec Packages release
---


##
[0.78.0](v0.77.1...v0.78.0)
(2025-03-07)


### ⚠ BREAKING CHANGES

* convert `TraitMethodNotInScope` to error
(noir-lang/noir#7427)
* bump bb version to v0.77.0
(noir-lang/noir#7599)
* remove merkle module from stdlib
(noir-lang/noir#7582)
* remove deprecated hash functions from stdlib
(noir-lang/noir#7477)
* **frontend:** Restrict capturing mutable variable in lambdas
(noir-lang/noir#7488)
* remove U128 struct from stdlib
(noir-lang/noir#7529)

### Features

* **barretenberg:** Graph methods for circuit analysis (part 2)
([#12130](#12130))
([ec4c0c4](ec4c0c4))
* **cli:** Log and replay oracle transcript
(noir-lang/noir#7417)
([f13b729](f13b729))
* Compare bincode to CBOR, FlexBuffers and Protobuf - implement best
(noir-lang/noir#7513)
([8eb727c](8eb727c))
* **experimental:** Enable ownership syntax
(noir-lang/noir#7603)
([1a3c112](1a3c112))
* **experimental:** Issue errors for unreachable match branches
(noir-lang/noir#7556)
([f13b729](f13b729))
* nullify just-added notes
([#12552](#12552))
([dcba7a4](dcba7a4))
* perform constant sha256 compressions at compile-time
(noir-lang/noir#7566)
([f13b729](f13b729))
* relate errors to macro built-ins errors
(noir-lang/noir#7609)
([fbaa634](fbaa634))
* simplify simple conditionals for brillig
(noir-lang/noir#7205)
([f13b729](f13b729))
* Support `&lt;Type as Trait&gt;::method` in expressions
(noir-lang/noir#7551)
([f13b729](f13b729))
* Sync from aztec-packages (noir-lang/noir#7606)
([8eb727c](8eb727c))
* teardown in call interface
([#12499](#12499))
([062df02](062df02))
* translation evaluations with zk
([#12222](#12222))
([568982d](568982d))


### Bug Fixes

* **avm:** use the correct number of rows in check_interaction
([#12519](#12519))
([b1284ef](b1284ef))
* aztec-up
([#12509](#12509))
([3ddb6de](3ddb6de))
* bbup
([#12555](#12555))
([e7b5353](e7b5353))
* Bitwise lookup
([#12471](#12471))
([a38f353](a38f353))
* **ci:** remove regex - transfer explicitly
([#12525](#12525))
([352bb1d](352bb1d))
* Cl/fix arm anvil
([#12565](#12565))
([e4bfbd1](e4bfbd1))
* compare Quoted by expanding interned values
(noir-lang/noir#7602)
([1a3c112](1a3c112))
* Display causes but not stack trace in CLI error report
(noir-lang/noir#7584)
([f13b729](f13b729))
* **experimental:** Fix execution of match expressions with multiple
branches (noir-lang/noir#7570)
([1a3c112](1a3c112))
* fix a few cases where safety comment wasn't correctly identified
(noir-lang/noir#7548)
([f13b729](f13b729))
* fix bbup and add CI
([#12541](#12541))
([1b2604c](1b2604c))
* Fix the config
([#12513](#12513))
([fb9fac6](fb9fac6))
* **frontend:** Restrict capturing mutable variable in lambdas
(noir-lang/noir#7488)
([f13b729](f13b729))
* FunctionDefinition::as_typed_expr didn't work well for trait imp…
(noir-lang/noir#7611)
([1a3c112](1a3c112))
* Log to `stderr` (noir-lang/noir#7585)
([f13b729](f13b729))
* **LSP:** references/rename only when underlying span has the correct…
(noir-lang/noir#7598)
([8eb727c](8eb727c))
* make vk metadata actual witnesses
([#12459](#12459))
([dada06f](dada06f))
* no fast deployments when the boot node needs to restart.
([#12557](#12557))
([866582e](866582e))
* **node:** drop log level of handler not registered
([#12523](#12523))
([cb7e42d](cb7e42d))
* override bb path in cli-wallet PXE config
([#12511](#12511))
([0c3024e](0c3024e))
* publish-bb-mac.yml version replace
([#12554](#12554))
([7e89dfb](7e89dfb))
* release bb-mac
([fac5fb5](fac5fb5))
* Revert "make vk metadata actual witnesses"
([#12534](#12534))
([ed46a3c](ed46a3c))
* shift right overflow in ACIR with unknown var now returns zero
(noir-lang/noir#7509)
([f13b729](f13b729))
* TokensPrettyPrinter was missing some spaces between tokens
(noir-lang/noir#7607)
([1a3c112](1a3c112))
* yarn-project e2e bench
([#12547](#12547))
([b40b904](b40b904))


### Miscellaneous

* add some extra tests (noir-lang/noir#7544)
([f13b729](f13b729))
* add underscore parameter documentation
(noir-lang/noir#7562)
([1a3c112](1a3c112))
* add yaml aliases in .test_patterns.yml
([#12516](#12516))
([3ee8d51](3ee8d51))
* address some frontend tests TODOs
(noir-lang/noir#7554)
([f13b729](f13b729))
* addressing remaining feedback in PR 12182
([#12494](#12494))
([f733879](f733879)),
closes
[#12193](#12193)
* bump `light-poseidon` (noir-lang/noir#7568)
([f13b729](f13b729))
* bump bb version to v0.77.0
(noir-lang/noir#7599)
([f13b729](f13b729))
* bump external pinned commits
(noir-lang/noir#7561)
([f13b729](f13b729))
* bump external pinned commits
(noir-lang/noir#7565)
([f13b729](f13b729))
* bump external pinned commits
(noir-lang/noir#7581)
([f13b729](f13b729))
* bump external pinned commits
(noir-lang/noir#7601)
([f13b729](f13b729))
* bump external pinned commits
(noir-lang/noir#7618)
([fbaa634](fbaa634))
* bump ring to address advisory
(noir-lang/noir#7619)
([fbaa634](fbaa634))
* Cleaner PXE
([#12515](#12515))
([a69f416](a69f416))
* cleanup committing and masking utility
([#12514](#12514))
([9f57048](9f57048))
* **cli:** exclude kind smoke test from flake list
([#12518](#12518))
([778bfa6](778bfa6))
* **cli:** Forward `nargo execute` to `noir_artifact_cli`
(noir-lang/noir#7406)
([f13b729](f13b729))
* convert `TraitMethodNotInScope` to error
(noir-lang/noir#7427)
([fbaa634](fbaa634))
* explode aliases when looking up owners in `.test_patterns.yml`
([#12526](#12526))
([2e0d791](2e0d791))
* fix trait import issues
([#12500](#12500))
([fd9f145](fd9f145))
* Fix yarn install immutable issues
([#12539](#12539))
([fb9ada3](fb9ada3)),
closes
[#12538](#12538)
* More config defaults and forward p2p ports
([#12529](#12529))
([2c45fb9](2c45fb9))
* **node:** return correct node version
([#12520](#12520))
([5502901](5502901))
* **profiler:** Add option to only get the total sample count for the
`execution-opcodes` command
(noir-lang/noir#7578)
([f13b729](f13b729))
* put RcTracker as part of the DIE context
(noir-lang/noir#7309)
([f13b729](f13b729))
* remove deprecated hash functions from stdlib
(noir-lang/noir#7477)
([f13b729](f13b729))
* remove FileDiagnostic (noir-lang/noir#7546)
([f13b729](f13b729))
* remove merkle module from stdlib
(noir-lang/noir#7582)
([f13b729](f13b729))
* Remove scope interpolation from env vars
([#12522](#12522))
([70942e9](70942e9))
* remove U128 struct from stdlib
(noir-lang/noir#7529)
([f13b729](f13b729))
* replace relative paths to noir-protocol-circuits
([f20c0dd](f20c0dd))
* replace relative paths to noir-protocol-circuits
([4365064](4365064))
* restore bb --version
([#12542](#12542))
([ab13d43](ab13d43))
* restore method syntax on `get_storage_slot` calls
([#12532](#12532))
([8e9f594](8e9f594))
* rm unused methods
([#12544](#12544))
([ed1dbdc](ed1dbdc))
* some SSA improvements (noir-lang/noir#7588)
([f13b729](f13b729))
* **spartan:** kind test speedup
([#12478](#12478))
([8ede7b1](8ede7b1))
* **ssa:** Turn the Brillig constraints check back on by default
(noir-lang/noir#7404)
([f13b729](f13b729))
* track more critical libraries
(noir-lang/noir#7604)
([f13b729](f13b729))
* update and lock AVM's lockfile
([#12533](#12533))
([2babc50](2babc50))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add an example to test the artifact CLI on CI Execute artifact with oracle transcript
3 participants