Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support batch publishes #6549

Open
ripienaar opened this issue Feb 21, 2025 · 0 comments
Open

Support batch publishes #6549

ripienaar opened this issue Feb 21, 2025 · 0 comments
Labels
2.12 needs design Needs design or ADR proposal Enhancement idea or proposal

Comments

@ripienaar
Copy link
Contributor

ripienaar commented Feb 21, 2025

Proposed change

We need to support batch publishes for a few use cases:

  • Clients want a very high ingest rate on streams and so might want to send batches of messages that would result in a single ACK
  • We want to support almost transaction-like related writes where several publishes to a stream are treated as a group and that groups messages are all committed or not with no chance for half commits.

Use case

For typed KV we might expand a hash/map into many keys inside a bucket. Imagine a USER record holding name, address etc. The address would be 4 or 5 keys and when the address change all 5 keys MUST update or none of the 5 should update.

A batch feature will facilitate that.

Additional to the high throughput publish mode already mentioned, we should aim for orders of magnitude increase in write performance.

@ripienaar ripienaar added proposal Enhancement idea or proposal 2.12 needs design Needs design or ADR labels Feb 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.12 needs design Needs design or ADR proposal Enhancement idea or proposal
Projects
None yet
Development

No branches or pull requests

1 participant