kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhjot Bharaj <prabhbha...@gmail.com>
Subject Re: Slow ISR catch-up
Date Sat, 05 Sep 2015 03:06:44 GMT
Hello friends,

Request your expertise on this problem I'm facing

Thanks
On Sep 4, 2015 8:09 PM, "Prabhjot Bharaj" <prabhbharaj@gmail.com> wrote:

> Hi,
>
> I am experiencing super slow throughput when using acks=-1
> Some further progress in continuation to the test in my previous email:-
>
> *Topic details -*
>
> Topic:temp PartitionCount:1 ReplicationFactor:3 Configs:
>
> Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5,2,1
> *This is the command I'm running - *
>
> kafka-producer-perf-test.sh --broker-list 96.7.250.122:9092,
> 96.17.183.53:9092,96.17.183.54:9092,96.7.250.117:9092,96.7.250.118:9092
> --messages 100000 --message-size 500 --topics temp --show-detailed-stats
> --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms
> 10000
>
> *Server.properties:-*
>
> broker.id=0
>
> port=9092
>
> *num.network.threads=6*
>
> num.io.threads=8
>
> *socket.send.buffer.bytes=10485760*
>
> *socket.receive.buffer.bytes=10485760*
>
> socket.request.max.bytes=104857600
>
> log.dirs=/tmp/kafka-logs
>
> num.partitions=1
>
> num.recovery.threads.per.data.dir=1
>
> log.retention.hours=168
>
> log.segment.bytes=1073741824
>
> log.retention.check.interval.ms=300000
>
> log.cleaner.enable=false
>
> zookeeper.connect=localhost:2181
>
> zookeeper.connection.timeout.ms=6000
>
> *num.replica.fetchers=6*
> *Observation:-*
>
> I have also noticed that if I use acks=1 (without --sync) and immediately
> use acks=-1 (without --sync), the test completes very quickly. Also, after
> running  this, if I describe the topic, it is still not in sync, which
> means acks=-1 is treated as 1 only
>
> Also, when running on a freshly created topic with just acks=-1, it takes
> 8 minutes to complete with --sync
>
> time kafka-producer-perf-test.sh --broker-list 96.7.250.122:9092,
> 96.17.183.53:9092,96.17.183.54:9092,96.7.250.117:9092,96.7.250.118:9092
> --messages 100000 --message-size 500 --topics temp --show-detailed-stats
> --threads 30 --request-num-acks -1 --batch-size 1000 --request-timeout-ms
> 10000 --compression-codec 2 --sync
>
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
>
> 2015-09-04 12:19:36:223, 2015-09-04 12:27:41:775, 2, 500, 1000, 47.68,
> 0.0982, 99990, 205.9306
>
> real 8m6.563s
>
> user 0m19.787s
>
> sys 0m5.601s
>
> If I use --sync, it is taking way longer.
>
> Where am I doing wrong?
>
> Thanks,
> Prabhjot
>
> On Fri, Sep 4, 2015 at 1:45 AM, Gwen Shapira <gwen@confluent.io> wrote:
>
>> Yes, this should work. Expect lower throughput though.
>>
>> On Thu, Sep 3, 2015 at 12:52 PM, Prabhjot Bharaj <prabhbharaj@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > Can I use sync for acks = -1?
>> >
>> > Regards,
>> > Prabhjot
>> > On Sep 3, 2015 11:49 PM, "Gwen Shapira" <gwen@confluent.io> wrote:
>> >
>> > > The test uses the old producer (we should fix that), and since you
>> don't
>> > > specify --sync, it runs async.
>> > > The old async producer simply sends data and doesn't wait for acks,
>> so it
>> > > is possible that the messages were never acked...
>> > >
>> > > On Thu, Sep 3, 2015 at 7:56 AM, Prabhjot Bharaj <
>> prabhbharaj@gmail.com>
>> > > wrote:
>> > >
>> > > > Hi Folks,
>> > > >
>> > > > Request your expertise on my doubt here.
>> > > >
>> > > > *My setup:-*
>> > > >
>> > > > 5 node kafka cluster (4 cores, 8GB RAM) on RAID-6 (500 GB)
>> > > > Using Kafka 0.8.2.1 with modified ProducerPerformance.scala
>> > > > I've modified ProducerPerformance.scala to send custom ASCII data,
>> > > instead
>> > > > of Byte Array of Zeroes
>> > > >
>> > > > *server.properties:-*
>> > > >
>> > > > broker.id=0
>> > > >
>> > > > log.cleaner.enable=false
>> > > >
>> > > > log.dirs=/tmp/kafka-logs
>> > > >
>> > > > log.retention.check.interval.ms=300000
>> > > >
>> > > > log.retention.hours=168
>> > > >
>> > > > log.segment.bytes=1073741824
>> > > >
>> > > > num.io.threads=8
>> > > >
>> > > > num.network.threads=3
>> > > >
>> > > > num.partitions=1
>> > > >
>> > > > num.recovery.threads.per.data.dir=1
>> > > >
>> > > > *num.replica.fetchers=4*
>> > > >
>> > > > port=9092
>> > > >
>> > > > socket.receive.buffer.bytes=1048576
>> > > >
>> > > > socket.request.max.bytes=104857600
>> > > >
>> > > > socket.send.buffer.bytes=1048576
>> > > >
>> > > > zookeeper.connect=localhost:2181
>> > > >
>> > > > zookeeper.connection.timeout.ms=6000
>> > > >
>> > > >
>> > > > *This is how I run the producer perf test:-*
>> > > >
>> > > > kafka-producer-perf-test.sh --broker-list
>> > > > a.a.a.a:9092,b.b.b.b:9092,c.c.c.c:9092,d.d.d.d:9092,e.e.e.e:9092
>> > > --messages
>> > > > 100000 --message-size 500 --topics temp --show-detailed-stats
>> > --threads
>> > > 5
>> > > > --request-num-acks -1 --batch-size 200 --request-timeout-ms 10000
>> > > > --compression-codec 0
>> > > >
>> > > > *Problem:-*
>> > > >
>> > > > This test completes in under 15 seconds for me
>> > > >
>> > > > But, after this test, if I try writing to another topic which has
2
>> > > > partitions and 3 replicas, it is dead slow and the same script seems
>> > > never
>> > > > to finish because the slow ISR catch-up is still going on.
>> > > >
>> > > > *My inference:-*
>> > > > I have noticed that for a topic with 1 partition and 3 replicas, the
>> > ISR
>> > > > shows only 1 broker id.
>> > > >
>> > > > Topic:temp PartitionCount:1 ReplicationFactor:3 Configs:
>> > > >
>> > > > Topic: temp Partition: 0 Leader: 5 Replicas: 5,1,2 Isr: 5
>> > > >
>> > > >
>> > > > I think it is because the data from the leader is not received in
>> > broker
>> > > > ids 1 and 2
>> > > > Also, I could confirm it from the data directory sizes for this
>> topic.
>> > > > Leader (5) has 20GB but replicas - 1 and 2 are still at 7GB
>> > > >
>> > > > *Doubts:-*
>> > > > 1. But, I was running the kafka-producer-perf-test.sh with acks=-1,
>> > which
>> > > > means that all data must have been committed to all replicas. But,
>> with
>> > > the
>> > > > replicas still at 7GB, it doesnt seem that acks=-1 is considered by
>> the
>> > > > producer.
>> > > >
>> > > > Am I missing something ?
>> > > >
>> > > > Regards,
>> > > > Prabhjot
>> > > >
>> > >
>> >
>>
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message