kafka: produce request for mesh-filter #17818
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adamkotwasinski
force-pushed
the
kafka-mesh-produce
branch
from
9475061 to
0ab3c55
Compare
adamkotwasinski
force-pushed
the
kafka-mesh-produce
branch
from
0ab3c55 to
5eab981
Compare
Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>
adamkotwasinski
force-pushed
the
kafka-mesh-produce
branch
from
5eab981 to
b315a0b
Compare
adamkotwasinski
commented
Aug 24, 2021
| // Find a record that matches provided delivery confirmation coming from Kafka producer. | ||
| // If all the records got their delivery data filled in, we are done, and can notify the origin | ||
| // filter. | ||
| bool ProduceRequestHolder::accept(const DeliveryMemento& memento) { |
There was a problem hiding this comment.
[future]
this method gets invoked thru future RichKafkaProducer implementation going thru the dispatcher to handle librdkafka delivery confirmation
TLDR: librdkafka callback (dedicated thread) ~> dispatcher ~> RichKafkaProducer (envoy worker thread) -> PRH::accept
adamkotwasinski
commented
Aug 24, 2021
| // We use this to tell apart messages. | ||
| // Important: we do not free this memory, it's still part of the 'ProduceRequestHandler' object. | ||
| // Future work: adopt Kafka's opaque-pointer functionality so we use less memory instead of | ||
| // keeping whole payload until we receive a confirmation. |
There was a problem hiding this comment.
[future work] we'd be sending with this and getting back the same pointer in callback's argument
mattklein123
approved these changes
Aug 25, 2021
mpuncel
added a commit
to mpuncel/envoy
that referenced
this pull request
Aug 25, 2021
* main: config: fix dfp config validation (envoyproxy#17835) docs: updating where meetings are uploaded (envoyproxy#17832) h2: moving a comment (envoyproxy#17846) quiche: early fail listener config with both quic and connection_balencer (envoyproxy#17834) dns: configuring a basic key value store to persist to disk (envoyproxy#17745) quic: fix receiving STOP_SENDING (envoyproxy#17815) tooling: Add Github release manager (envoyproxy#17741) tooling: Use upstream pytest-patches (envoyproxy#17809) Remove `hidden_envoy_deprecated_use_http2` (envoyproxy#17805) kafka: produce request for mesh-filter (envoyproxy#17818) Signed-off-by: Michael Puncel <mpuncel@squareup.com>
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Commit Message: kafka: produce request for mesh-filter
Additional Description: add produce handler for mesh filter; produce handler needs to extract records out of produce requests' bytes, send the records upstream (using Kafka producer), and will wait until it gets notifications (
onDelivery) to finally find out that the processing has finished, and we can notify the filter (== send the response to downstream Kafka client).Risk Level: Low
Testing: unit tests
Docs Changes: N/A
Release Notes: N/A
Platform Specific Features: N/A