kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Wang <chen.apache.s...@gmail.com>
Subject Consumer lag keep increasing
Date Wed, 05 Nov 2014 22:01:45 GMT
Hey Guys,
I have a really simply storm topology with a kafka spout, reading from
kafka through high level consumer. Since the topic has 30 partitions, we
have 30 threads in the spout reading from it. However, it seems that the
lag keeps increasing even the thread only read the message and do nothing.
The largest message size  are around 30KB, and the incoming rate can be as
hight as 14k/seconds. There are 3 brokers on some high config bare metal
machines. The client side config is like this:

kafka.config.fetch.message.max.bytes    3145728
kafka.config.group.id   spout_readonly
kafka.config.rebalance.backoff.ms       6000
kafka.config.rebalance.max.retries      6
kafka.config.zookeeper.connect  dare-broker00.sv.walmartlabs.com:2181,
dare-broker01.sv.walmartlabs.com:2181,dare-broker02.sv.walmartlabs.com:2181
kafka.config.zookeeper.session.timeout.ms       60000

what could possibly cause this huge lag? Will broker be a bottle neck, or
some config need to be adjusted? The server side config is like this:

replica.fetch.max.bytes=2097152
message.max.bytes=2097152
num.network.threads=4
num.io.threads=4

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=4194304

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=2097152

# The maximum size of a request that the socket server will accept
(protection against OOM)
socket.request.max.bytes=104857600

Any help appreciated!
Chen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message