kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhjot Bharaj <prabhbha...@gmail.com>
Subject Re: Slow ISR catch-up
Date Thu, 03 Sep 2015 19:52:27 GMT
Hi,

Can I use sync for acks = -1?

Regards,
Prabhjot
On Sep 3, 2015 11:49 PM, "Gwen Shapira" <gwen@confluent.io> wrote:

> The test uses the old producer (we should fix that), and since you don't
> specify --sync, it runs async.
> The old async producer simply sends data and doesn't wait for acks, so it
> is possible that the messages were never acked...
>
> On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj <prabhbharaj@gmail.com>
> wrote:
>
> > Hi Folks,
> >
> > Request your expertise on my doubt here.
> >
> > *My setup:-*
> >
> > 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB)
> > Using Kafka 0.8.2.1 with modified ProducerPerformance.scala
> > I've modified ProducerPerformance.scala to send custom ASCII data,
> instead
> > of Byte Array of Zeroes
> >
> > *server.properties:-*
> >
> > broker.id=0
> >
> > log.cleaner.enable=false
> >
> > log.dirs=/tmp/kafka-logs
> >
> > log.retention.check.interval.ms=300000
> >
> > log.retention.hours=168
> >
> > log.segment.bytes=1073741824
> >
> > num.io.threads=8
> >
> > num.network.threads=3
> >
> > num.partitions=1
> >
> > num.recovery.threads.per.data.dir=1
> >
> > *num.replica.fetchers=4*
> >
> > port=9092
> >
> > socket.receive.buffer.bytes=1048576
> >
> > socket.request.max.bytes=104857600
> >
> > socket.send.buffer.bytes=1048576
> >
> > zookeeper.connect=localhost:2181
> >
> > zookeeper.connection.timeout.ms=6000
> >
> >
> > *This is how I run the producer perf test:-*
> >
> > kafka-producer-perf-test.sh --broker-list
> > a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092
> --messages
> > 100000 --message-size 500 --topics temp --show-detailed-stats  --threads
> 5
> > --request-num-acks -1 --batch-size 200 --request-timeout-ms 10000
> > --compression-codec 0
> >
> > *Problem:-*
> >
> > This test completes in under 15 seconds for me
> >
> > But, after this test, if I try writing to another topic which has 2
> > partitions and 3 replicas, it is dead slow and the same script seems
> never
> > to finish because the slow ISR catch-up is still going on.
> >
> > *My inference:-*
> > I have noticed that for a topic with 1 partition and 3 replicas, the ISR
> > shows only 1 broker id.
> >
> > Topic:temp PartitionCount:1 ReplicationFactor:3 Configs:
> >
> > Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5
> >
> >
> > I think it is because the data from the leader is not received in broker
> > ids 1 and 2
> > Also, I could confirm it from the data directory sizes for this topic.
> > Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB
> >
> > *Doubts:-*
> > 1. But, I was running the kafka-producer-perf-test.sh with acks=-1, which
> > means that all data must have been committed to all replicas. But, with
> the
> > replicas still at 7GB, it doesnt seem that acks=-1 is considered by the
> > producer.
> >
> > Am I missing something ?
> >
> > Regards,
> > Prabhjot
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message