kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Plaatje <pplaa...@gmail.com>
Subject Re: Magic byte error when trying to consume Avro data with Kafka Connect
Date Thu, 06 Dec 2018 18:50:35 GMT
Hi Marco,

We've seen this issue when the actual message is not encoded with a schema.
In our case we had to refactor our producer slightly, as it wasn't
including the magic byte in the messages.

Cheers,
Patrick


On Thu, 6 Dec 2018, 19:30 Marcos Juarez, <mjuarez@gmail.com> wrote:

> We're trying to use Kafka Connect to pull down data from Kafka, but we're
> having issues with the Avro deserialization.
>
> When we attempt to consume data using the kafka-avro-console-consumer, we
> can consume it, and deserialize it correctly.  Our command is similar to
> the following:
>
> *./kafka-avro-console-consumer --topic my_topic --bootstrap-server
> localhost:9092 --property schema.registry.url=http://localhost:8081
> <http://localhost:8081>*
>
> However, if we attempt to use Kafka Connect to consume the data (in our
> case, with an S3SinkConnector), we always get the following error from the
> connector:
>
> "state": "FAILED",
>
> "trace": "org.apache.kafka.connect.errors.DataException:
> cf4_sdp_rest_test_v2
>
>         at
>
> io.confluent.connect.avro.AvroConverter.toConnectData(AvroConverter.java:97)
>
>         at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:467)
>
>         at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:301)
>
>         at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:205)
>
>         at
>
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:173)
>
>         at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:170)
>
>         at
> org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:214)
>
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>
>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>
>         at java.lang.Thread.run(Thread.java:748)
>
> *Caused by: org.apache.kafka.common.errors.SerializationException: Error
> deserializing Avro message for id -1*
>
> *Caused by: org.apache.kafka.common.errors.SerializationException: Unknown
> magic byte!*
>
> We've tried several different configurations, but we are not able to get
> Kafka Connect to properly consume these messages.  For reference, these
> same messages work correctly with Apache Camus, and of course with the
> command-line kafka-avro-console-consumer as described above. So we're
> pretty confident that the messages are built correctly.
>
> We have noticed that we get the same error if we attempt to print the
> messages in the console consumer with the option *--property
> print.key=false*.  However, we can't figure out a way to turn off key
> deserialization (if that is what is causing this) on the kafka
> connect/connector side.
>
> We're using Kafka 1.1.1, and all the packages are Confluent platform 4.1.2.
>
> Any help would be appreciated.
>
> Thanks,
>
> Marcos Juarez
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message