kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcos Juarez <mjua...@gmail.com>
Subject Magic byte error when trying to consume Avro data with Kafka Connect
Date Thu, 06 Dec 2018 18:29:54 GMT
We're trying to use Kafka Connect to pull down data from Kafka, but we're
having issues with the Avro deserialization.

When we attempt to consume data using the kafka-avro-console-consumer, we
can consume it, and deserialize it correctly.  Our command is similar to
the following:

*./kafka-avro-console-consumer --topic my_topic --bootstrap-server
localhost:9092 --property schema.registry.url=http://localhost:8081
<http://localhost:8081>*

However, if we attempt to use Kafka Connect to consume the data (in our
case, with an S3SinkConnector), we always get the following error from the
connector:

"state": "FAILED",

"trace": "org.apache.kafka.connect.errors.DataException:
cf4_sdp_rest_test_v2

        at
io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:97)

        at
org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:467)

        at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:301)

        at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)

        at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)

        at
org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)

        at
org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

        at java.lang.Thread.run(Thread.java:748)

*Caused by: org.apache.kafka.common.errors.SerializationException: Error
deserializing Avro message for id -1*

*Caused by: org.apache.kafka.common.errors.SerializationException: Unknown
magic byte!*

We've tried several different configurations, but we are not able to get
Kafka Connect to properly consume these messages.  For reference, these
same messages work correctly with Apache Camus, and of course with the
command-line kafka-avro-console-consumer as described above. So we're
pretty confident that the messages are built correctly.

We have noticed that we get the same error if we attempt to print the
messages in the console consumer with the option *--property
print.key=false*.  However, we can't figure out a way to turn off key
deserialization (if that is what is causing this) on the kafka
connect/connector side.

We're using Kafka 1.1.1, and all the packages are Confluent platform 4.1.2.

Any help would be appreciated.

Thanks,

Marcos Juarez

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message