kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Cai <h...@pinterest.com.INVALID>
Subject InvalidMessageException: Message is corrupt, Consumer stuck
Date Wed, 05 Aug 2015 05:23:53 GMT
Hi,

We are using the Kafka high-level consumer 8.1.1, somehow we got a
corrupted message in the topic.  We are not sure the root cause of this,
but the problem we are having now is the HL consumer is stuck in that
position:

kafka.message.InvalidMessageException: Message is corrupt (stored crc =
537685622, computed crc = 36513351)

        at kafka.message.Message.ensureValid(Message.scala:166)

        at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:102)

        at
kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)

        at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)

        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)


If we try to ignore that exception and iterate to the next message, the
iterator couldn't pass that error state:

java.lang.IllegalStateException: Iterator is in failed state

        at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:54)


By looking at the code, looks like you can only calling
IteratorTemplate.resetState() to clear the state, but this is an internal
method, is this the right way to workaround this problem?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message