Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming errors for encoders #7093

Merged
merged 4 commits into from
Aug 8, 2024
Merged

Streaming errors for encoders #7093

merged 4 commits into from
Aug 8, 2024

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Aug 7, 2024

This is a simple change to make our encoding methods take iterators of results instead of iterators of unwrapped values.

In most (all?) real-world scenarios, you will have to deal with errors as you serialize LogMsgs.
The previous design forced you to collect all that data in order to first check for errors, and only then start encoding. Now you can just stream as needed.
In the rare case where nothing can fail, you're just one .map(Ok) away anyway.

As a side-effect, rerun rrd merge|compact now streams their output.

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!
  • If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

@teh-cmc teh-cmc added 🧑‍💻 dev experience developer experience (excluding CI) 🚀 performance Optimization, memory use, etc do-not-merge Do not merge this PR exclude from changelog PRs with this won't show up in CHANGELOG.md CLI Related to the Rerun CLI labels Aug 7, 2024
Copy link
Member

@jleibs jleibs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

@teh-cmc teh-cmc force-pushed the cmc/cli_imp_3_stdin branch from 7412909 to f712632 Compare August 8, 2024 07:50
Base automatically changed from cmc/cli_imp_3_stdin to main August 8, 2024 07:53
@teh-cmc teh-cmc removed the do-not-merge Do not merge this PR label Aug 8, 2024
@teh-cmc teh-cmc force-pushed the cmc/streaming_err_encoder branch from cb7fbda to 98dcaee Compare August 8, 2024 07:56
@teh-cmc teh-cmc merged commit dc9b50c into main Aug 8, 2024
13 of 19 checks passed
@teh-cmc teh-cmc deleted the cmc/streaming_err_encoder branch August 8, 2024 07:56
teh-cmc added a commit that referenced this pull request Aug 8, 2024
You can now do this:
```
cat docs/snippets/all/archetypes/*_rust.rrd | rerun rrd compact --max-rows 99999999 --max-bytes 999999999 | rerun rrd print
```

Also made sure that encoders and decoders accurately report the size of
the data they plow through.

- Part of #7048 
- DNM: requires #7093
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLI Related to the Rerun CLI 🧑‍💻 dev experience developer experience (excluding CI) exclude from changelog PRs with this won't show up in CHANGELOG.md 🚀 performance Optimization, memory use, etc
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants