Schema Registry considers avro.java.string as part of the schema comparison #868

markush81 · 2018-08-15T13:47:05Z

Given

{
  "type": "record",
  "name": "Ping",
  "namespace": "de.markush.kafka",
  "fields": [
    {
      "name": "id",
      "type": "string",
      "logicalType": "uuid"
    },
    {
      "name": "created_at",
      "type": "string",
      "logicalType": "iso8601Timestamp"
    }
  ]
}

is published to Schema-Registry

when

Java Producer using generated POJOs with avro-maven-plugin tries to serialize this events with KafkaAvroSerialzerthe lookup for the schema fails because of the fact, that SCHEMA$ property in generated classes looks a little different

then

org.apache.kafka.common.errors.SerializationException: Error retrieving Avro schema: {"type":"record","name":"Ping","namespace":"de.markush.kafka","fields":[{"name":"id","type":{"type":"string","avro.java.string":"String"},"logicalType":"uuid"},{"name":"created_at","type":{"type":"string","avro.java.string":"String"},"logicalType":"iso8601Timestamp"}]}
Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Schema not found; error code: 40403
	at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:203) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:229) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:296) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.schemaregistry.client.rest.RestService.lookUpSubjectVersion(RestService.java:284) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getIdFromRegistry(CachedSchemaRegistryClient.java:132) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getId(CachedSchemaRegistryClient.java:264) ~[kafka-schema-registry-client-5.0.0.jar:na]
	at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer.serializeImpl(AbstractKafkaAvroSerializer.java:82) ~[kafka-avro-serializer-5.0.0.jar:na]
	at io.confluent.kafka.serializers.KafkaAvroSerializer.serialize(KafkaAvroSerializer.java:53) ~[kafka-avro-serializer-5.0.0.jar:na]
	at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:65) ~[kafka-clients-1.0.2.jar:na]
	at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:55) ~[kafka-clients-1.0.2.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:791) ~[kafka-clients-1.0.2.jar:na]
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:768) ~[kafka-clients-1.0.2.jar:na]
	at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:285) ~[spring-kafka-2.1.8.RELEASE.jar:2.1.8.RELEASE]
	at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:357) ~[spring-kafka-2.1.8.RELEASE.jar:2.1.8.RELEASE]
	at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:188) ~[spring-kafka-2.1.8.RELEASE.jar:2.1.8.RELEASE]
	at de.markush.kafka.schemaregistryexamples.producer.PingProducer.ping(PingProducer.java:29) ~[classes/:na]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_172]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_172]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_172]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_172]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.0.8.RELEASE.jar:5.0.8.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_172]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_172]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_172]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_172]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_172]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_172]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_172]

SCHEMA$

{  
   "type":"record",
   "name":"Ping",
   "namespace":"de.markush.kafka",
   "fields":[  
      {  
         "name":"id",
         "type":{  
            "type":"string",
            "avro.java.string":"String"
         },
         "logicalType":"uuid"
      },
      {  
         "name":"created_at",
         "type":{  
            "type":"string",
            "avro.java.string":"String"
         },
         "logicalType":"iso8601Timestamp"
      }
   ]
}

it has these avro.java.string properties.
(I still do not get why these are needed at all and because of this the schema is treated being not equal. Btw. consumers can handle this situation.)

Remark

I guess that #28 is related, but due to having the Map<MD5, SchemaIdAndSubjects> schemaHashToGuid AFAIK it is not enough to just change io.confluent.kafka.schemaregistry.avro.AvroUtils#parseSchema to a more or less schema equal form, especially not Parsing Canonical Form for Schemas since it lacks the at a minimum the default properties (What about order?). Already tried that ... but i assume it is more of a concept change, just storing canonical schemas into the registry from startup and not only change the lookup.

Anyone from Confluent can guide towards a solution? Concerning implementation i would be happy to help.

Version

Confluent Open Source: 5.0.0

The text was updated successfully, but these errors were encountered:

OneCricketeer · 2018-08-23T00:57:28Z

I have one question: Why did you manually register the schema instead of letting the Serializer do it?

Also, can you post the rest of the stack trace and mention version numbers of the Registry, please?

markush81 · 2018-08-23T04:35:40Z

I edited my previous input: added complete stacktrace and version.

To answer your question:

The setup ist, that there are multiple clients Java, .NET, ruby and even Node. So the process of defining schema and maintaining schema-registry is done centralised (yet manually, in process of writing some tool for it) one the one hand and language agnostic on the other. In case each producer is allowed to register hist own schemas, all consumers (example: Java is producing and all others are consuming) need to be able to live with that language specific properties.

OneCricketeer · 2018-08-23T05:09:05Z

Thanks! And yes, that makes sense. I come from an almost entirely Java producer environment, so this error is a new one to me, but it makes sense given that the MD5 computation is on the raw Schema string body, I believe, not the Parsing Canonical Form, as you mention

markush81 · 2018-08-23T05:24:41Z

Yes, the key point is to find the right canonical form, cause the one implemented by avro has at least one big issue, it does not consider default property, which is essential.

So there is sth. needed more like language-agnostic form, basically remove all custom properties.

Btw. do you know why avro.java.string is needed at all? AFAIK the consumer does not need it and i have no clue why a producer should need this as well.

OneCricketeer · 2018-08-23T12:20:00Z

Did you look at the PR mentioned on #28 it mentions defaults are preserved.

The java.string property can be set to String, or its parent class CharSequence. That way, the Avro schema can refer to any other sibling of a String type, and it's generally only used by code generators, as far as I know.

markush81 · 2018-08-23T15:23:30Z

The PR did a custom implementation to preserve them yes, but seems not work with 5.0.0 anymore.

Actually Code Generator does not need the property according to what i experienced so far, but it actually generates them and that causes the issue. It only makes sense if these are used by Serializer ... but until now i did not debug that deep, but will hopefully find time soon.

OneCricketeer · 2018-08-23T15:38:59Z

By code generator, I mean the Avro Maven Plugin's <stringType> tag.

https://github.com/apache/avro/blob/master/lang/java/maven-plugin/src/main/java/org/apache/avro/mojo/AbstractAvroMojo.java#L101-L106

That determines if Java classes are made with String x or CharSequence x

markush81 · 2018-08-23T18:02:51Z

sure i understand this part, but what i mean is: why does the generator add avro.java.string to the SCHEMA$ of each generated class. But i will find out.

OneCricketeer · 2018-08-25T02:44:25Z

In order to generate Java classes from the schema at a later point.
https://github.com/apache/avro/blob/master/lang/java/compiler/src/main/java/org/apache/avro/compiler/specific/SpecificCompiler.java#L575-L584

markush81 · 2018-08-25T06:04:28Z

I believe no.

My process is following:

Take a schema, without this property
Generate code from this schema
Look into the generated class and see the property SCHEMA$.

Only now this property occurs. So there is no reason for any later Generator or Compiler steps. Either the Serializer needs this property now when this generated class gets serialized or i do not see the reason.

markush81 · 2018-08-28T04:49:47Z

I think i now found out the purpose. It is used by the deserializer, but it seems that deserializer takes two things: reader- (<- the SCHEMA$ property with avro.java.string) and writer-schema (<- from schema registry, without avro.java.string) and somehow this does work. So i question (at least for my setup) the need of it. But i am pretty sure there are reasons for it.

Anyhow: the original problem still exists, how to let @confluentinc Schema Registry be language agnostic and still use it in such situation, that there is a Java client with generated code.

chrisdoberman · 2018-12-20T22:09:07Z

Yes, I am having the same issue. We have non-java consumers. The java producer, which uses the generated avro class with the SCHEMA$ field, writes this to the registry - which includes the avro.java.string type.

mbieser · 2019-02-22T23:41:18Z

+1
Having the java libraries accept language-independent schema would be helpful.

plinioj · 2019-06-18T13:16:39Z

+1

cbsmith · 2019-06-26T03:04:20Z

+1

clande · 2019-07-12T18:54:20Z

+1

alinazemian · 2019-08-10T08:57:07Z

+1

ghost · 2019-09-10T04:43:26Z

Is there any timeline for this issue fix? Thanks for your contributions.

anilkulkarni87 · 2019-10-29T17:16:30Z

What is the workaround for this?

zhuguangbin · 2019-11-06T09:32:41Z

I am having this problem. As a workaround, I manually register the schema with avro.java.string type restead of the original schema in avsc.

Or else, I set auto.register.schemas = true in my producer kafka avro serde config. But I don't recommend this way, because in developing stage, this may register wrong schema to production schema registry.

I think I will go this way:

write a schema in avsc
maven build to java jar. in test stage, use kafka-schema-registry-maven-plugin to test compatibility between local avsc schema and remote kafka schema registery.
if compatibly and confirm the schema is final, mannually register it to remote schema registy. (use a java tool, given jar artifact groupId/artifactId/version and full avro classname for topic, the tool extract this SpecificRecord's SCHEMA$ field, this is the right schema, POST this schema to registry. We must restrict the privilege who can use this tool to register schema )
consumer/producer upgrade artifact version to the right version for consuming/producing kafka topic

lukas-krecan · 2020-05-16T06:29:14Z

I have filed an issue to Avro project https://issues.apache.org/jira/browse/AVRO-2838

enima2684 · 2020-07-13T18:14:40Z

The same issue raises when producing avro data from Kafka Connect and consuming it with Kafka Streams
It would be really helpful to have a solution for that.

What workaround are you using while the solution is still in development ?

peknu · 2020-09-10T13:36:03Z

One way to resolve this is to replace all string types in the avsc file same type that the plugin will generate. For example:
Replace: { "name": "comment", "type": "string" }
With: { "name": "comment", "type": { "type": "string", "avro.java.string": "String" } }

Keep the avro-maven-plugin config:

<configuration>
    <stringType>String</stringType>
</configuration>

This way, the generated schema inserted into the jar file will be identical with the one used by the plugin to upload and verify schemas. I have only tested that the inserted schema is identical to the one in the avsc file but this should work.

rayokota · 2020-10-23T19:47:01Z

This is addressed in 5.5.1 and later by specifying auto.register.schemas=false and use.latest.version=true in the Avro serializer configs.

mnowaczyk · 2021-10-22T22:20:32Z

I'll just add here that Confluent recently introduced a flag in their version of avro serializer, which solves this problem. You need to put:
KafkaAvroSerializerConfig.AVRO_REMOVE_JAVA_PROPS_CONFIG = true
in the serde config, and it will send correct schemas to the registry, without "avro.java.string".

lukas-krecan · 2021-10-27T12:25:01Z

At the end, we ended-up using our serializer that does not require code generation at all https://github.com/productboardlabs/jackson-kafka-avro-serializer

slominskir · 2021-11-09T18:37:15Z

The new property:

KafkaAvroSerializerConfig.AVRO_REMOVE_JAVA_PROPS_CONFIG = true

seems like an ugly hack. I guess it it is easier for down-stream projects to bend over backwards than try to fix the root of the problem at the source? I submitted a patch to AVRO project months ago with no movement so I guess the answer is "yes": apache/avro#1235

markush81 changed the title ~~Schema Registry consideres avro.java.string as part of the schema comparison~~ Schema Registry considers avro.java.string as part of the schema comparison Aug 17, 2018

rayokota added the enhancement label Nov 9, 2018

Kabowyad mentioned this issue Mar 24, 2020

avro-maven-plugin with stringType config causes 40403 Schema not found #1352

Closed

rayokota closed this as completed Oct 23, 2020

jeeftor mentioned this issue Nov 5, 2020

Issues matching on "avro.java.string" : "String" (version 6.0.0) #1677

Closed

slominskir mentioned this issue Feb 10, 2021

Regression: list-active.py no longer shows value JeffersonLab/jaws-epics2kafka#1

Closed

rayokota mentioned this issue Feb 19, 2021

DGS-1333 Add config to remove Java props from Avro schemas #1774

Merged

slominskir mentioned this issue May 13, 2021

Java and Python AVRO compatibility - stringType JeffersonLab/jaws#21

Closed

rmarting mentioned this issue Jun 17, 2021

Avro Producer considers "avro.java.string" as part of the schema comparations Apicurio/apicurio-registry#1592

Closed

dynamike2010 mentioned this issue Apr 16, 2024

Error deserializing AVRO - schema evolution example databricks/iceberg-kafka-connect#238

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schema Registry considers avro.java.string as part of the schema comparison #868

Schema Registry considers avro.java.string as part of the schema comparison #868

markush81 commented Aug 15, 2018 •

edited

Loading

OneCricketeer commented Aug 23, 2018 •

edited

Loading

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018 •

edited

Loading

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 25, 2018

markush81 commented Aug 25, 2018 •

edited

Loading

markush81 commented Aug 28, 2018 •

edited

Loading

chrisdoberman commented Dec 20, 2018

mbieser commented Feb 22, 2019 •

edited

Loading

plinioj commented Jun 18, 2019

cbsmith commented Jun 26, 2019

clande commented Jul 12, 2019

alinazemian commented Aug 10, 2019

ghost commented Sep 10, 2019

anilkulkarni87 commented Oct 29, 2019

zhuguangbin commented Nov 6, 2019

lukas-krecan commented May 16, 2020

enima2684 commented Jul 13, 2020

peknu commented Sep 10, 2020 •

edited

Loading

rayokota commented Oct 23, 2020

mnowaczyk commented Oct 22, 2021

lukas-krecan commented Oct 27, 2021

slominskir commented Nov 9, 2021

Schema Registry considers avro.java.string as part of the schema comparison #868

Schema Registry considers avro.java.string as part of the schema comparison #868

Comments

markush81 commented Aug 15, 2018 • edited Loading

OneCricketeer commented Aug 23, 2018 • edited Loading

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018 • edited Loading

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 23, 2018

markush81 commented Aug 23, 2018

OneCricketeer commented Aug 25, 2018

markush81 commented Aug 25, 2018 • edited Loading

markush81 commented Aug 28, 2018 • edited Loading

chrisdoberman commented Dec 20, 2018

mbieser commented Feb 22, 2019 • edited Loading

plinioj commented Jun 18, 2019

cbsmith commented Jun 26, 2019

clande commented Jul 12, 2019

alinazemian commented Aug 10, 2019

ghost commented Sep 10, 2019

anilkulkarni87 commented Oct 29, 2019

zhuguangbin commented Nov 6, 2019

lukas-krecan commented May 16, 2020

enima2684 commented Jul 13, 2020

peknu commented Sep 10, 2020 • edited Loading

rayokota commented Oct 23, 2020

mnowaczyk commented Oct 22, 2021

lukas-krecan commented Oct 27, 2021

slominskir commented Nov 9, 2021

markush81 commented Aug 15, 2018 •

edited

Loading

OneCricketeer commented Aug 23, 2018 •

edited

Loading

markush81 commented Aug 23, 2018 •

edited

Loading

markush81 commented Aug 25, 2018 •

edited

Loading

markush81 commented Aug 28, 2018 •

edited

Loading

mbieser commented Feb 22, 2019 •

edited

Loading

peknu commented Sep 10, 2020 •

edited

Loading