kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Fwd: MirrorMaker kafka.message.InvalidMessageException
Date Wed, 23 Sep 2015 07:23:29 GMT
Cheers!

Nobody has an Idea what to do with InvalidMessageExceptions?

Jörg


-------- Forwarded Message --------
Subject: 	MirrorMaker kafka.message.InvalidMessageException
Date: 	Fri, 18 Sep 2015 11:37:20 +0200
From: 	Jörg Wagner <joerg.wagner1@1und1.de>
Reply-To: 	users@kafka.apache.org
To: 	users@kafka.apache.org



Hey everyone!

One of my Mirrormakers is exiting with the following error:

[2015-09-18 11:27:35,591] FATAL [mirrormaker-consumer-0] Stream
unexpectedly exited. (kafka.tools.MirrorMaker$ConsumerThread)
kafka.message.InvalidMessageException: Message is corrupt (stored crc =
3256823012, computed crc = 4230971415)
         at kafka.message.Message.ensureValid(Message.scala:166)
         at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:101)
         at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
         at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
         at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
         at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:32)
         at
scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
         at kafka.consumer.KafkaStream.foreach(KafkaStream.scala:25)
         at
kafka.tools.MirrorMaker$ConsumerThread.run(MirrorMaker.scala:259)

The same topic has been mirrored to another DC without a problem.

A few questions arise:

  * Why could the other DC mirror the topic successfully?
  * Why is the MM not failing for the message, but carrying on?
  * What can I do now to carry on consuming with this MM?
  * How can I automate the answers to the above?

My MM currently also does not want to write a log, although I passed on:
KAFKA_LOG4J_OPTS="-Dlog4j.configuration=file:${KAFKA_CONFIG}/mirror/log4j.properties"
Any clues why I don't get any log output? (I got the error from running
MM interactive)

Thanks

Jörg




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message