kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Prabhjot Bharaj <prabhbha...@gmail.com>
Subject Re: Producer becomes slow over time
Date Tue, 29 Sep 2015 13:17:23 GMT
Hi Erik,

I've not tested it on a producer that is not a part of the kafka cluster.


*Producer configuration on all machines:-*

root@x.x.x.x:/a/kafka/config# cat producer.properties  | egrep -v '^#|^$'

metadata.broker.list=localhost:9092

producer.type=sync

compression.codec=none

serializer.class=kafka.serializer.DefaultEncoder


*Server configuration on all machines:-*

root@y.y.y.y:/a/kafka/config# cat server.properties  | egrep -v '^#|^$'

broker.id=0

port=9092

num.network.threads=6

num.io.threads=8

socket.send.buffer.bytes=10485760

socket.receive.buffer.bytes=10485760

socket.request.max.bytes=104857600

log.dirs=/tmp/kafka-logs

num.partitions=1

num.recovery.threads.per.data.dir=1

log.retention.hours=168

log.segment.bytes=1073741824

log.retention.check.interval.ms=300000

log.cleaner.enable=false

zookeeper.connect=localhost:2181

zookeeper.connection.timeout.ms=6000

num.replica.fetchers=4

*Command used from both the machines (slow and fast producer):*

kafka-producer-perf-test.sh --broker-list
x.x.x.x:9092,y.y.y.y:9092,z.z.z.z:9092,a.a.a.a:9092,b.b.b.b:9092 --messages
1048576 --message-size 500 --topics part_1_repl_3_4 --show-detailed-stats
--threads 32 --request-num-acks 1 --batch-size 1000 --request-timeout-ms
10000 --compression-codec 2 --reporting-interval 1000

Regards,

Prabhjot

On Thu, Sep 24, 2015 at 6:36 PM, Helleren, Erik <Erik.Helleren@cmegroup.com>
wrote:

> What happens when the new producer that is getting 70 MB/s is started on a
> machine that is not part of the kafka cluster?
>
> Can you include your topic description/configuration, producer
> configuration, and broker configuration?
>
> On 9/24/15, 1:44 AM, "Prabhjot Bharaj" <prabhbharaj@gmail.com> wrote:
>
> >Hi,
> >
> >I would like to dig deep into this issue. I've changed log4j.properties
> >for
> >logging in ALL mode in all places. I am getting lost in the logs.
> >
> >Any pointers would be welcome
> >
> >Please let me know if you would need any information regarding this
> >
> >Thanks,
> >Prabhjot
> >
> >On Wed, Sep 23, 2015 at 6:46 PM, Prabhjot Bharaj <prabhbharaj@gmail.com>
> >wrote:
> >
> >> Hello Folks,
> >>
> >> I've noticed that 2 producer machines, that I had configured, have
> >>become
> >> very slow over time
> >> They are giving 17-19 MB/s
> >>
> >> But, a producer that I setup today is giving 70MB/s as the write
> >>throughput
> >>
> >> If I see the contents of bin, libs, config directories, nothing is
> >> different in the files on any of the producer machines.
> >>
> >> Producer is running on the kafka machines itself
> >>
> >> Request your expertise
> >>
> >> Regards,
> >> Prabhjot
> >>
> >>
> >>
> >
> >
> >--
> >---------------------------------------------------------
> >"There are only 10 types of people in the world: Those who understand
> >binary, and those who don't"
>
>


-- 
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message