kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manikumar <manikumar.re...@gmail.com>
Subject Re: mirror maker producer thread dies with error
Date Fri, 12 Jan 2018 02:54:15 GMT
"Memory records is not writable" error was fixed in 0.10.0.0 release
https://issues.apache.org/jira/browse/KAFKA-3594

On Fri, Jan 12, 2018 at 6:10 AM, Sunil Parmar <sunilosunil@gmail.com> wrote:

> We see multiple instance of this error
>
> 2017-12-23 05:30:53,722 WARN
> org.apache.kafka.clients.producer.internals.Sender: Got error produce
> response with correlation id 41987642 on topic-partition events-14,
> retrying (2147482899 attempts left). Error: NOT_LEADER_FOR_PARTITION
>
> Followed by this error (repeating multilpe times)
>
> 2017-12-23 05:31:09,165 INFO
> org.apache.kafka.clients.producer.KafkaProducer: Proceeding to force close
> the producer since pending requests could not be completed within timeout 0
> ms.
> 2017-12-23 05:31:09,165 ERROR
> org.apache.kafka.clients.producer.internals.ErrorLoggingCallback: Error
> when sending message to topic events with key: 36 bytes, value: 8794 bytes
> with error: Batch Expired
>
> Once in a while I also notice following error happens only once or two
> times between restarts of mm.
> 2017-09-10 20:02:47,723 FATAL kafka.tools.MirrorMaker$MirrorMakerThread:
> [mirrormaker-thread-14] Mirror maker thread failure due to
> java.lang.IllegalStateException: Memory records is not writable
>     at
> org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
>     at
> org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(
> RecordBatch.java:69)
>     at
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(
> RecordAccumulator.java:166)
>     at
> org.apache.kafka.clients.producer.KafkaProducer.send(
> KafkaProducer.java:438)
>     at
> kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:628)
>     at
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$
> run$6.apply(MirrorMaker.scala:423)
>     at
> kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$
> run$6.apply(MirrorMaker.scala:423)
>     at scala.collection.Iterator$class.foreach(Iterator.scala:742)
>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>     at kafka.tools.MirrorMaker$MirrorMakerThread.run(
> MirrorMaker.scala:423)
> 2017-09-10 20:02:47,728 INFO kafka.tools.MirrorMaker$MirrorMakerThread:
> [mirrormaker-thread-14] Flushing producer.
> 2017-09-10 20:02:47,780 WARN
> org.apache.kafka.clients.producer.internals.Sender: Got error produce
> response with correlation id 20892 on topic-partition
> thirdpartycalls_ingestion_am1-2, retrying (2147483645 attempts left).
> Error: NOT_LEADER_FOR_PARTITION
>
>
> followed by
>
> Clean shutdown of mirror maker
>
> We're using CDH, Kafka 0.9
>
> Can someone explains the reason/meaning of the errors we see ? This will
> help us debug this issue further. My understanding is that the producer
> threads are failing because for some reason partition leader is changing at
> run time. We also noticed that this happens on the perticular day of the
> week but I coudln't locate any weekly scheduler in any configuration.
>
> Any pointers will be helpful
>
> Thanks,
> Sunil Parmar
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message