kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sunil Parmar <sunilosu...@gmail.com>
Subject mirror maker producer thread dies with error
Date Fri, 12 Jan 2018 00:40:48 GMT
We see multiple instance of this error

2017-12-23 05:30:53,722 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 41987642 on topic-partition events-14,
retrying (2147482899 attempts left). Error: NOT_LEADER_FOR_PARTITION

Followed by this error (repeating multilpe times)

2017-12-23 05:31:09,165 INFO
org.apache.kafka.clients.producer.KafkaProducer: Proceeding to force close
the producer since pending requests could not be completed within timeout 0
ms.
2017-12-23 05:31:09,165 ERROR
org.apache.kafka.clients.producer.internals.ErrorLoggingCallback: Error
when sending message to topic events with key: 36 bytes, value: 8794 bytes
with error: Batch Expired

Once in a while I also notice following error happens only once or two
times between restarts of mm.
2017-09-10 20:02:47,723 FATAL kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Mirror maker thread failure due to
java.lang.IllegalStateException: Memory records is not writable
    at
org.apache.kafka.common.record.MemoryRecords.append(MemoryRecords.java:93)
    at
org.apache.kafka.clients.producer.internals.RecordBatch.tryAppend(RecordBatch.java:69)
    at
org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:166)
    at
org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:438)
    at
kafka.tools.MirrorMaker$MirrorMakerProducer.send(MirrorMaker.scala:628)
    at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
    at
kafka.tools.MirrorMaker$MirrorMakerThread$$anonfun$run$6.apply(MirrorMaker.scala:423)
    at scala.collection.Iterator$class.foreach(Iterator.scala:742)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:423)
2017-09-10 20:02:47,728 INFO kafka.tools.MirrorMaker$MirrorMakerThread:
[mirrormaker-thread-14] Flushing producer.
2017-09-10 20:02:47,780 WARN
org.apache.kafka.clients.producer.internals.Sender: Got error produce
response with correlation id 20892 on topic-partition
thirdpartycalls_ingestion_am1-2, retrying (2147483645 attempts left).
Error: NOT_LEADER_FOR_PARTITION


followed by

Clean shutdown of mirror maker

We're using CDH, Kafka 0.9

Can someone explains the reason/meaning of the errors we see ? This will
help us debug this issue further. My understanding is that the producer
threads are failing because for some reason partition leader is changing at
run time. We also noticed that this happens on the perticular day of the
week but I coudln't locate any weekly scheduler in any configuration.

Any pointers will be helpful

Thanks,
Sunil Parmar

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message