kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Unable to scale kafka cluster
Date Thu, 22 May 2014 14:34:22 GMT
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIaddnewbrokersdynamicallytoacluster
?

Thanks,

Jun


On Thu, May 22, 2014 at 4:33 AM, Ertuğrul Yılmaz <
ertugrul.yilmaz@convertale.com> wrote:

> Hi Kafka User and Dev team,
>
> We want to use kafka to scale high traffic load. My plan is adding new
> kafka instances to the cluster regarding the current load. In order to
> achieve this, we have executed some producer tests and want to learn how we
> can scale kafka cluster. Our test environments and configs are listed
> below.
>
> During our tests, nMsg.sec value did not increase after adding a new
> broker. Even decreases.
>
> Waiting for your comments.
> Thanks.
>
> *Topic descriptions are;*
> Topic:tt3 PartitionCount:1 ReplicationFactor:1 Configs:
> Topic: tt3 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
>
> Topic:tt4 PartitionCount:2 ReplicationFactor:1 Configs:
> Topic: tt4 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
> Topic: tt4 Partition: 1 Leader: 2 Replicas: 2 Isr: 2
>
> *Test code scripts are;*
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
> 150000 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
> --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:52:04:523, 2014-05-22 08:52:30:668, 0, 1000, 200, 143.05,
> 5.4715, 150000, 5737.2347
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
> --messages 150000 --topics tt3 --threads 5 --message-size 1000 --batch-size
> 200 --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:50:25:753, 2014-05-22 08:50:53:584, 0, 1000, 200, 143.05,
> 5.1400, 150000, 5389.6734
>
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
> 150000 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
> --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:48:18:628, 2014-05-22 08:48:47:153, 0, 1000, 200, 143.05,
> 5.0149, 150000, 5258.5451
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
> --messages 150000 --topics tt4 --threads 5 --message-size 1000 --batch-size
> 200 --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:46:41:443, 2014-05-22 08:47:11:770, 0, 1000, 200, 143.05,
> 4.7170, 150000, 4946.0876
>
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
> 300000 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
> --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:42:03:119, 2014-05-22 08:42:45:885, 0, 1000, 200, 286.10,
> 6.6899, 300000, 7014.9184
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
> --messages 300000 --topics tt3 --threads 5 --message-size 1000 --batch-size
> 200 --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:33:20:526, 2014-05-22 08:34:03:131, 0, 1000, 200, 286.10,
> 6.7152, 300000, 7041.4271
>
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
> 300000 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
> --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:43:41:984, 2014-05-22 08:44:23:987, 0, 1000, 200, 286.10,
> 6.8115, 300000, 7142.3470
>
> bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
> --messages 300000 --topics tt4 --threads 5 --message-size 1000 --batch-size
> 200 --initial-message-id 500
> start.time, end.time, compression, message.size, batch.size,
> total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
> 2014-05-22 08:44:47:356, 2014-05-22 08:45:29:058, 0, 1000, 200, 286.10,
> 6.8606, 300000, 7193.8996
>
>
> *machine instance types*
> zookeeper amazon ec2 t1.micro
> kafka broker1   amazon ec2 m1.small
> kafka broker2   amazon ec2 m1.small
> producer amazon ec2 m3.medium
>
>
> *zookeeper1 conf*
> tickTime=2000
> initLimit=10
> syncLimit=5
> clientPort=2184
> server.1=localhost:2888:3888
> server.2= localhost:2889:3889
> server.3= localhost:2890:3890
> kafka broker 1 server.properties
> broker.id=1
> port=9092
> host.name=server1
> num.network.threads=8
> num.io.threads=8
> num.replica.fetchers=4
> replica.fetch.max.bytes=1048576
> replica.fetch.wait.max.ms=500
> replica.high.watermark.checkpoint.interval.ms=5000
> replica.socket.timeout.ms=30000
> replica.socket.receive.buffer.bytes=65536
> replica.lag.time.max.ms=10000
> replica.lag.max.messages=4000
> controller.socket.timeout.ms=30000
> controller.message.queue.size=10
> socket.send.buffer.bytes=1048576
> socket.receive.buffer.bytes=1048576
> socket.request.max.bytes=104857600
> queued.max.requests=16
> fetch.purgatory.purge.interval.requests=100
> producer.purgatory.purge.interval.requests=100
> log.dirs=/opt/kafka-logs
> num.partitions=8
> sage.max.bytes=1000000
> auto.create.topics.enable=true
> log.index.interval.bytes=4096
> log.index.size.max.bytes=10485760
> log.retention.hours=168
> log.flush.interval.ms=10000
> log.flush.interval.messages=20000
> log.flush.scheduler.interval.ms=2000
> log.roll.hours=168
> log.cleanup.interval.mins=30
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=60000
> log.cleaner.enable=false
> zookeeper.connect=zookeeper:2184,zookeeper:2185,zookeeper:2186
> zookeeper.connection.timeout.ms=1000000
> zookeeper.sync.time.ms=2000
>
>
>
> --
>
> Saygılarımla
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message