kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhjot Bharaj <prabhbha...@gmail.com>
Subject Slow ISR catch-up
Date Thu, 03 Sep 2015 14:56:12 GMT
Hi Folks,

Request your expertise on my doubt here.

*My setup:-*

5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB)
Using Kafka 0.8.2.1 with modified ProducerPerformance.scala
I've modified ProducerPerformance.scala to send custom ASCII data, instead
of Byte Array of Zeroes

*server.properties:-*

broker.id=0

log.cleaner.enable=false

log.dirs=/tmp/kafka-logs

log.retention.check.interval.ms=300000

log.retention.hours=168

log.segment.bytes=1073741824

num.io.threads=8

num.network.threads=3

num.partitions=1

num.recovery.threads.per.data.dir=1

*num.replica.fetchers=4*

port=9092

socket.receive.buffer.bytes=1048576

socket.request.max.bytes=104857600

socket.send.buffer.bytes=1048576

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000


*This is how I run the producer perf test:-*

kafka-producer-perf-test.sh --broker-list
a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092 --messages
100000 --message-size 500 --topics temp --show-detailed-stats  --threads 5
--request-num-acks -1 --batch-size 200 --request-timeout-ms 10000
--compression-codec 0

*Problem:-*

This test completes in under 15 seconds for me

But, after this test, if I try writing to another topic which has 2
partitions and 3 replicas, it is dead slow and the same script seems never
to finish because the slow ISR catch-up is still going on.

*My inference:-*
I have noticed that for a topic with 1 partition and 3 replicas, the ISR
shows only 1 broker id.

Topic:temp PartitionCount:1 ReplicationFactor:3 Configs:

Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5


I think it is because the data from the leader is not received in broker
ids 1 and 2
Also, I could confirm it from the data directory sizes for this topic.
Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB

*Doubts:-*
1. But, I was running the kafka-producer-perf-test.sh with acks=-1, which
means that all data must have been committed to all replicas. But, with the
replicas still at 7GB, it doesnt seem that acks=-1 is considered by the
producer.

Am I missing something ?

Regards,
Prabhjot

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message