Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improved CLI: stdin streaming support #7092

Merged
merged 4 commits into from
Aug 8, 2024
Merged

Improved CLI: stdin streaming support #7092

merged 4 commits into from
Aug 8, 2024

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Aug 7, 2024 •
edited by github-actions bot
Loading

You can now do this:

cat docs/snippets/all/archetypes/*_rust.rrd | rerun rrd print

and this:

cat docs/snippets/all/archetypes/*_rust.rrd | rrd merge -o /tmp/all_merged.rrd

and this

cat docs/snippets/all/archetypes/*_rust.rrd | rerun rrd compact --max-rows 99999999 --max-bytes 999999999 -o /tmp/all_compacted_max.rrd

Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested the web demo (if applicable):
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG
  • If applicable, add a new check to the release checklist!
  • If have noted any breaking changes to the log API in CHANGELOG.md and the migration guide

To run all checks from main, comment on the PR with @rerun-bot full-check.

@teh-cmc teh-cmc added 🧑‍💻 dev experience developer experience (excluding CI) 🚀 performance Optimization, memory use, etc do-not-merge Do not merge this PR include in changelog CLI Related to the Rerun CLI labels Aug 7, 2024
@teh-cmc teh-cmc mentioned this pull request Aug 7, 2024
Merged
6 tasks
@teh-cmc teh-cmc force-pushed the cmc/multiplexed_decodeer branch from 9533541 to 951ce6b Compare August 8, 2024 07:49
Base automatically changed from cmc/multiplexed_decodeer to main August 8, 2024 07:50
@teh-cmc teh-cmc force-pushed the cmc/cli_imp_3_stdin branch from 7412909 to f712632 Compare August 8, 2024 07:50
@teh-cmc teh-cmc removed the do-not-merge Do not merge this PR label Aug 8, 2024
@teh-cmc teh-cmc merged commit c81fe6a into main Aug 8, 2024
26 of 29 checks passed
@teh-cmc teh-cmc deleted the cmc/cli_imp_3_stdin branch August 8, 2024 07:53
teh-cmc added a commit that referenced this pull request Aug 8, 2024
@teh-cmc
Streaming errors for encoders (#7093)
dc9b50c 
This is a simple change to make our encoding methods take iterators of
results instead of iterators of unwrapped values.

In most (all?) real-world scenarios, you will have to deal with errors
as you serialize `LogMsg`s.
The previous design forced you to collect all that data in order to
first check for errors, and only then start encoding. Now you can just
stream as needed.
In the rare case where nothing can fail, you're just one `.map(Ok)` away
anyway.

As a side-effect, `rerun rrd merge|compact` now streams their output.

- DNM: requires #7092
- Related to #6984
@teh-cmc teh-cmc changed the title Improved CLI 3: stdin streaming support Improved CLI: stdin streaming support Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jleibs jleibs jleibs approved these changes

Assignees
No one assigned
Labels
CLI Related to the Rerun CLI 🧑‍💻 dev experience developer experience (excluding CI) include in changelog 🚀 performance Optimization, memory use, etc
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@teh-cmc @jleibs