kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ertuğrul Yılmaz <ertugrul.yil...@convertale.com>
Subject Unable to scale kafka cluster
Date Thu, 22 May 2014 11:33:25 GMT
Hi Kafka User and Dev team,

We want to use kafka to scale high traffic load. My plan is adding new
kafka instances to the cluster regarding the current load. In order to
achieve this, we have executed some producer tests and want to learn how we
can scale kafka cluster. Our test environments and configs are listed below.

During our tests, nMsg.sec value did not increase after adding a new
broker. Even decreases.

Waiting for your comments.
Thanks.

*Topic descriptions are;*
Topic:tt3 PartitionCount:1 ReplicationFactor:1 Configs:
Topic: tt3 Partition: 0 Leader: 1 Replicas: 1 Isr: 1

Topic:tt4 PartitionCount:2 ReplicationFactor:1 Configs:
Topic: tt4 Partition: 0 Leader: 1 Replicas: 1 Isr: 1
Topic: tt4 Partition: 1 Leader: 2 Replicas: 2 Isr: 2

*Test code scripts are;*
bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
150000 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:52:04:523, 2014-05-22 08:52:30:668, 0, 1000, 200, 143.05,
5.4715, 150000, 5737.2347

bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 150000 --topics tt3 --threads 5 --message-size 1000 --batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:50:25:753, 2014-05-22 08:50:53:584, 0, 1000, 200, 143.05,
5.1400, 150000, 5389.6734


bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
150000 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:48:18:628, 2014-05-22 08:48:47:153, 0, 1000, 200, 143.05,
5.0149, 150000, 5258.5451

bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 150000 --topics tt4 --threads 5 --message-size 1000 --batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:46:41:443, 2014-05-22 08:47:11:770, 0, 1000, 200, 143.05,
4.7170, 150000, 4946.0876


bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
300000 --topics tt3 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:42:03:119, 2014-05-22 08:42:45:885, 0, 1000, 200, 286.10,
6.6899, 300000, 7014.9184

bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 300000 --topics tt3 --threads 5 --message-size 1000 --batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:33:20:526, 2014-05-22 08:34:03:131, 0, 1000, 200, 286.10,
6.7152, 300000, 7041.4271


bin/kafka-producer-perf-test.sh --broker-list=server1:9092 --messages
300000 --topics tt4 --threads 5 --message-size 1000 --batch-size 200
--initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:43:41:984, 2014-05-22 08:44:23:987, 0, 1000, 200, 286.10,
6.8115, 300000, 7142.3470

bin/kafka-producer-perf-test.sh --broker-list=server1:9092,server2:9092
--messages 300000 --topics tt4 --threads 5 --message-size 1000 --batch-size
200 --initial-message-id 500
start.time, end.time, compression, message.size, batch.size,
total.data.sent.in.MB, MB.sec, total.data.sent.in.nMsg, nMsg.sec
2014-05-22 08:44:47:356, 2014-05-22 08:45:29:058, 0, 1000, 200, 286.10,
6.8606, 300000, 7193.8996


*machine instance types*
zookeeper amazon ec2 t1.micro
kafka broker1   amazon ec2 m1.small
kafka broker2   amazon ec2 m1.small
producer amazon ec2 m3.medium


*zookeeper1 conf*
tickTime=2000
initLimit=10
syncLimit=5
clientPort=2184
server.1=localhost:2888:3888
server.2= localhost:2889:3889
server.3= localhost:2890:3890
kafka broker 1 server.properties
broker.id=1
port=9092
host.name=server1
num.network.threads=8
num.io.threads=8
num.replica.fetchers=4
replica.fetch.max.bytes=1048576
replica.fetch.wait.max.ms=500
replica.high.watermark.checkpoint.interval.ms=5000
replica.socket.timeout.ms=30000
replica.socket.receive.buffer.bytes=65536
replica.lag.time.max.ms=10000
replica.lag.max.messages=4000
controller.socket.timeout.ms=30000
controller.message.queue.size=10
socket.send.buffer.bytes=1048576
socket.receive.buffer.bytes=1048576
socket.request.max.bytes=104857600
queued.max.requests=16
fetch.purgatory.purge.interval.requests=100
producer.purgatory.purge.interval.requests=100
log.dirs=/opt/kafka-logs
num.partitions=8
sage.max.bytes=1000000
auto.create.topics.enable=true
log.index.interval.bytes=4096
log.index.size.max.bytes=10485760
log.retention.hours=168
log.flush.interval.ms=10000
log.flush.interval.messages=20000
log.flush.scheduler.interval.ms=2000
log.roll.hours=168
log.cleanup.interval.mins=30
log.segment.bytes=1073741824
log.retention.check.interval.ms=60000
log.cleaner.enable=false
zookeeper.connect=zookeeper:2184,zookeeper:2185,zookeeper:2186
zookeeper.connection.timeout.ms=1000000
zookeeper.sync.time.ms=2000



-- 

Saygılarımla

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message