Kafka-mesh filter #11936

adamkotwasinski · 2020-07-07T21:22:33Z

Commit message: Kafka-mesh filter
Additional Description:
Kafka mesh filter.
Can be put between native Kafka client and intercept produce requests to forward them to upstream Kafka clusters as per its fowarding rules. Right now there's only one type of rule - prefix by topic name (so e.g. we can send messages to topics a* and have them in cluster 1, while others go to cluster 2). Only produce requests for now. Support for consumers would be more complex or require more limitations (as Kafka consumers keep state, so it's not like Redis - or actually it's like Redis pub/sub).
Doesn't work on Windows because of librdkafka compilation issues (and I don't have a Windows machine).
Risk Level: Low
Testing: Unit tests + integration test (2 kafka clusters are started + proxy in between + producer) + manual testing.
Docs Changes: New .rst & .proto files, as we have a new filter.
Release Notes: N/A

Without getting into too much details:

Kafka producers (Java, Python, etc.) send records to Envoy (the filter contains enough code to establish communication, first it needs to advertise itself it's a Kafka server through api-versions + metadata requests).
Kafka-mesh filter instance receives the records, parses them, forwards them to rules which decide where the records should go.
The records are forwarded to producers dedicated for each of clusters.
librdkafka producers send the records where they should have gone (librdkafka has its own threads and we use dispatcher.post to have the delivery confirmation back to normal worker thread).

Other PRs for kafka-mesh work (merged independently):
#17202
#17374
#17475
#17597
#17783
#17818
#17839
#17887

repokitteh-read-only · 2020-07-07T21:22:41Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy[\w/]*/(v1alpha\d?|v1|v2alpha\d?|v2))|(api/envoy/type/(matcher/)?\w+.proto).
CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #11936 was opened by adamkotwasinski.

see: more, trace.

stale · 2020-08-01T08:58:12Z

This pull request has been automatically marked as stale because it has not had activity in the last 7 days. It will be closed in 7 days if no further activity occurs. Please feel free to give a status update now, ping for review, or re-open when it's ready. Thank you for your contributions!

repokitteh-read-only · 2020-08-07T21:53:59Z

Retrying Azure Pipelines, to retry CircleCI checks, use /retest-circle.

🐱

Caused by: a #11936 (comment) was created by @adamkotwasinski.

see: more, trace.

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

adamkotwasinski · 2021-09-09T00:44:54Z

Ready for review - also ran some Java tests.

mattklein123

Cool, thanks. A few small comments.

/wait

mattklein123 · 2021-09-09T15:43:38Z

docs/root/configuration/listeners/network_filters/kafka_mesh_filter.rst

+clusters, depending on configured forwarding rules. Corresponding message versions from
+Kafka 2.4.0 are supported.
+
+.. attention::


Can you also add an attention/warning here that this is only available on contrib builds with a link? See what we do for squash and some of the other contrib filters and please unify as needed, thanks.

done - comments and missing link added; same for kafka-broker

mattklein123 · 2021-09-09T15:44:33Z

source/extensions/filters/network/well_known_names.h

+  // Kafka Mesh filter
+  const std::string KafkaMesh = "envoy.filters.network.kafka_mesh";


we are trying to get rid of this file. Can you just inline this?

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only · 2021-09-09T21:03:07Z

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #11936 (comment) was created by @adamkotwasinski.

see: more, trace.

mattklein123

Thanks!

repokitteh-read-only bot added v2-freeze api labels Jul 7, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch 12 times, most recently from 7e7ea5f to 2ecbd19 Compare July 10, 2020 20:54

stale bot added the stale stalebot believes this issue/PR has not been touched recently label Aug 1, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch from 8ea61ac to a35ea7c Compare August 7, 2020 20:20

stale bot removed the stale stalebot believes this issue/PR has not been touched recently label Aug 7, 2020

adamkotwasinski force-pushed the kafka-mesh-public branch 10 times, most recently from d6e2bd6 to e49dbde Compare August 17, 2020 20:46

Cleanup

73549a0

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only bot added waiting:any and removed waiting labels Aug 31, 2021

Cleanup

d3b549d

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only bot added waiting:any and removed waiting:any labels Sep 1, 2021

repokitteh-read-only bot removed the waiting:any label Sep 9, 2021

adamkotwasinski marked this pull request as ready for review September 9, 2021 00:45

adamkotwasinski requested a review from mattklein123 as a code owner September 9, 2021 00:45

adamkotwasinski changed the title ~~[DRAFT] Kafka-mesh filter~~ Kafka-mesh filter Sep 9, 2021

mattklein123 requested changes Sep 9, 2021

View reviewed changes

repokitteh-read-only bot added the waiting label Sep 9, 2021

adamkotwasinski added 2 commits September 9, 2021 11:09

Merge remote-tracking branch 'envoy/main' into kafka-mesh-public

7b4ec98

Docs fixes; inlined filter name

57ff9c8

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only bot added waiting:any and removed waiting labels Sep 9, 2021

Remove well_known_names reference and other includes

3b6b3cf

Signed-off-by: Adam Kotwasinski <adam.kotwasinski@gmail.com>

repokitteh-read-only bot added waiting:any and removed waiting:any labels Sep 9, 2021

adamkotwasinski requested a review from mattklein123 September 9, 2021 22:44

mattklein123 approved these changes Sep 10, 2021

View reviewed changes

mattklein123 merged commit 0c8942d into envoyproxy:main Sep 10, 2021

adamkotwasinski deleted the kafka-mesh-public branch September 10, 2021 19:34

adamkotwasinski mentioned this pull request Mar 9, 2023

Kafka protocol filter #2852

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka-mesh filter #11936

Kafka-mesh filter #11936

adamkotwasinski commented Jul 7, 2020 •

edited

Loading

repokitteh-read-only bot commented Jul 7, 2020

stale bot commented Aug 1, 2020

repokitteh-read-only bot commented Aug 7, 2020

adamkotwasinski commented Sep 9, 2021

mattklein123 left a comment

mattklein123 Sep 9, 2021

adamkotwasinski Sep 9, 2021

mattklein123 Sep 9, 2021

adamkotwasinski Sep 9, 2021

repokitteh-read-only bot commented Sep 9, 2021

mattklein123 left a comment

		// Kafka Mesh filter
		const std::string KafkaMesh = "envoy.filters.network.kafka_mesh";

Kafka-mesh filter #11936

Kafka-mesh filter #11936

Conversation

adamkotwasinski commented Jul 7, 2020 • edited Loading

repokitteh-read-only bot commented Jul 7, 2020

stale bot commented Aug 1, 2020

repokitteh-read-only bot commented Aug 7, 2020

adamkotwasinski commented Sep 9, 2021

mattklein123 left a comment

Choose a reason for hiding this comment

mattklein123 Sep 9, 2021

Choose a reason for hiding this comment

adamkotwasinski Sep 9, 2021

Choose a reason for hiding this comment

mattklein123 Sep 9, 2021

Choose a reason for hiding this comment

adamkotwasinski Sep 9, 2021

Choose a reason for hiding this comment

repokitteh-read-only bot commented Sep 9, 2021

mattklein123 left a comment

Choose a reason for hiding this comment

adamkotwasinski commented Jul 7, 2020 •

edited

Loading