Add basic support for converting Avro records to JSON for output #151

llofberg · 2018-08-19T17:43:25Z

This change is

Requires libavro and libserdes to compile.

Marcelovk · 2018-10-25T13:50:37Z

This is awesome, I was able to compile with it and is much better than using kafka-avro-console-consumer. The only thing that I missed is the documentation in the command line describing -a,-A and -s options

sknop

It works for me on Mac after a small modification of configure.kafkacat.

Great work!

sknop · 2018-10-30T21:54:31Z

configure.kafkacat

+    if [[ $WITH_AVRO == y ]] && \
+        mkl_lib_check --static=-lserdes "serdes" HAVE_SERDES disable CC "-lserdes -lavro" \
+        "#include  <stdlib.h>
+        #include  <libserdes/serdes-avro.h>


It looks like on Mac you need to include
#include <sys/types.h>
as well or the test compile will fail.

sknop

Observation: specifying -A or -a without -s "schema-registry-url" crashes kafkacat with segmentation fault. Should be caught and properly dealt with in the argument parsing.

ryansobol · 2018-11-09T14:45:11Z

Patiently awaiting this feature...

casidiablo · 2018-12-21T06:23:03Z

My C mojo is very rusty... it took me a while to compile it on MacOS. This is what I had to do:

brew install cmake
brew install automake
brew install pkgconfig
brew install jansson

./bootstrap.sh

homer6 · 2019-01-18T18:50:44Z

@edenhill @sknop What is still required for this to be merged into the mainline?

edenhill · 2019-01-18T19:41:34Z

Sorry for the delay, busy finalizing the librdkafka v1.0 release. Will tend to this in a week or two.

arifalkner · 2019-01-29T23:31:13Z

@casidiablo You also have to install avro-c to compile:

brew install avro-c

KarthikRangaraju · 2019-02-17T21:24:03Z

Looking forward to this feature to get merged 👍

arifalkner · 2019-03-25T21:31:44Z

@edenhill What's the ETA on getting this merged?

edenhill · 2019-03-25T22:48:55Z

librdkafka v1.0.0 is going out tonight, should be able to address kafkacat PRs next week and get a new kafkacat release out.

jataggart · 2019-04-23T15:05:42Z

this would be really great to have, any updates?

…to llofberg-output_avro_as_json

bigkraig · 2019-05-10T21:57:21Z

friendly ping @edenhill

avegancafe · 2019-06-04T21:56:02Z

Any update on this? I use kafkacat quite often and would love this support!

arifalkner · 2019-06-05T21:17:27Z

I understand if you are swamped with work and might also consider accepting other people to maintain this repo. I appreciate all the hard work you've put into this tool and I'm sure we'd love to help continue support your workload for continuing on adding new features

edenhill · 2019-06-17T17:46:18Z

Please try out this docker image to consume Avro-encoded messages:
docker run edenhill/kafkacat:sr-wip -b YOURBROKER -s YOUR_SR_URL -C -t YOUR_TOPIC -a value -Ee -J | jq .payload.some.avro.field

cecchisandrone · 2019-06-18T09:27:16Z

Please try out this docker image to consume Avro-encoded messages:
docker run edenhill/kafkacat:sr-wip -b YOURBROKER -s YOUR_SR_URL -C -t YOUR_TOPIC -a value -Ee -J | jq .payload.some.avro.field

I tried and I receive "payload_error": "Invalid CP1 magic byte 32, expected 0". there is no request to schema registry. I was wondering what is the subject convention to gather the schema. Is it modifiable?

edenhill · 2019-06-18T09:34:09Z

@cecchisandrone That means the message value/payload was not serialized by a schema-registry-aware producer.

The schema-registry-aware avro producer prefixes the serialized avro data with a short header that includes a magic value and the schema-id, this is what the consumer uses to look up the schema and deserialize the message.

Marcelovk · 2019-06-18T11:01:59Z

@edenhill That command works correcly for me. I also tried with -f "%k : %s\n" . This feature will be a great addition to kafkacat

cecchisandrone · 2019-06-18T16:03:07Z

Tested with a correct topic. It seems to work.

* refactoring and fixes of the original PR * added key_error, payload_error optional fields to JSON message envelope to indicate deserialization errors. * prepare for future Avro producer when we (or preferably avro-c) figure out how to convert JSON to Avro. * fix librdkafka version in bootstrap.sh * make Dockerfile rely more on bootstrap.sh * use edenhill fork of yajl to allow embedding the avro-to-json deserialized message key/value directly in the JSON envelope with -J.

edenhill · 2019-07-07T15:33:33Z

Added generic deserializers (for various ints, char, string, inpspired by Python's struct.unpack) and changed the command line arguments.
I believe this is a more flexible and future-proof path forward.

Please try out this latest change in with edenhill/kafkacat:serdes-wip (docker image).

edenhill · 2019-07-08T22:23:28Z

Thanks for your contribution and your patience!

cecchisandrone · 2019-07-10T15:40:28Z

Is there also a plan to support for AVRO serialization?

cecchisandrone · 2019-07-10T16:07:14Z

Also, I wanted to test this on latest master build on windows. Should I need to enable something in the build config? I receive this message after building kafkacat on windows:
% ERROR: This build of kafkacat lacks Avro/Schema-Registry support

edenhill · 2019-07-12T12:03:26Z

We'll add Avro serialization in a later version (incremental delivery, aight?!).

Windows support is further down the line since libserdes is not supported on Windows yet.

Add basic support for converting Avro records to JSON for output

222b4d1

llofberg mentioned this pull request Aug 19, 2018

Confluent's Avro / Schema Registry support? #119

Closed

Fix serdes initialization etc.

a95ef27

Requires libavro and libserdes to compile.

sknop reviewed Oct 30, 2018

View reviewed changes

sknop reviewed Nov 1, 2018

View reviewed changes

Adds usage doc and check for schema registry url

95d83b3

lenny added 6 commits November 10, 2018 15:01

Build with --enable-avro

0b43bb3

Add jansson to .travis

e3dc6c8

Fix typo

cc1ad37

Add builddeps for doozer

8cbceb4

Add more builddeps for doozer

32ca603

Add more builddeps for doozer

b0a8f01

edenhill mentioned this pull request Apr 8, 2019

kafkacat -F errors due to "No such configuration property" #170

Closed

Merge branch 'output_avro_as_json' of github.com:llofberg/kafkacat in…

1bf9ea3

…to llofberg-output_avro_as_json

edenhill mentioned this pull request Jun 18, 2019

Debian release #183

Closed

edenhill force-pushed the output_avro_as_json branch from 2546e5f to 3ba4ebc Compare July 5, 2019 16:26

edenhill force-pushed the output_avro_as_json branch 4 times, most recently from 7cbaaf4 to edc5d3e Compare July 8, 2019 21:49

Added generic deserialization using pack-str format

490b3c3

edenhill force-pushed the output_avro_as_json branch from edc5d3e to 490b3c3 Compare July 8, 2019 21:54

edenhill merged commit c0e4521 into edenhill:master Jul 8, 2019

jjlin mentioned this pull request Nov 8, 2019

Add support for Base64-encoded output of key and/or payload data #206

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add basic support for converting Avro records to JSON for output #151

Add basic support for converting Avro records to JSON for output #151

llofberg commented Aug 19, 2018 •

edited

Loading

Marcelovk commented Oct 25, 2018

sknop left a comment

sknop Oct 30, 2018

sknop left a comment •

edited

Loading

ryansobol commented Nov 9, 2018

casidiablo commented Dec 21, 2018

homer6 commented Jan 18, 2019

edenhill commented Jan 18, 2019

arifalkner commented Jan 29, 2019

KarthikRangaraju commented Feb 17, 2019

arifalkner commented Mar 25, 2019

edenhill commented Mar 25, 2019

jataggart commented Apr 23, 2019

bigkraig commented May 10, 2019

avegancafe commented Jun 4, 2019

arifalkner commented Jun 5, 2019

edenhill commented Jun 17, 2019

cecchisandrone commented Jun 18, 2019

edenhill commented Jun 18, 2019

Marcelovk commented Jun 18, 2019

cecchisandrone commented Jun 18, 2019

edenhill commented Jul 7, 2019

edenhill commented Jul 8, 2019

cecchisandrone commented Jul 10, 2019

cecchisandrone commented Jul 10, 2019

edenhill commented Jul 12, 2019

Add basic support for converting Avro records to JSON for output #151

Add basic support for converting Avro records to JSON for output #151

Conversation

llofberg commented Aug 19, 2018 • edited Loading

Marcelovk commented Oct 25, 2018

sknop left a comment

Choose a reason for hiding this comment

sknop Oct 30, 2018

Choose a reason for hiding this comment

sknop left a comment • edited Loading

Choose a reason for hiding this comment

ryansobol commented Nov 9, 2018

casidiablo commented Dec 21, 2018

homer6 commented Jan 18, 2019

edenhill commented Jan 18, 2019

arifalkner commented Jan 29, 2019

KarthikRangaraju commented Feb 17, 2019

arifalkner commented Mar 25, 2019

edenhill commented Mar 25, 2019

jataggart commented Apr 23, 2019

bigkraig commented May 10, 2019

avegancafe commented Jun 4, 2019

arifalkner commented Jun 5, 2019

edenhill commented Jun 17, 2019

cecchisandrone commented Jun 18, 2019

edenhill commented Jun 18, 2019

Marcelovk commented Jun 18, 2019

cecchisandrone commented Jun 18, 2019

edenhill commented Jul 7, 2019

edenhill commented Jul 8, 2019

cecchisandrone commented Jul 10, 2019

cecchisandrone commented Jul 10, 2019

edenhill commented Jul 12, 2019

llofberg commented Aug 19, 2018 •

edited

Loading

sknop left a comment •

edited

Loading