kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chen Wang <chen.apache.s...@gmail.com>
Subject Consumer lag keep increasing
Date Wed, 05 Nov 2014 22:01:45 GMT
Hey Guys,
I have a really simply storm topology with a kafka spout, reading from
kafka through high level consumer. Since the topic has 30 partitions, we
have 30 threads in the spout reading from it. However, it seems that the
lag keeps increasing even the thread only read the message and do nothing.
The largest message size  are around 30KB, and the incoming rate can be as
hight as 14k/seconds. There are 3 brokers on some high config bare metal
machines. The client side config is like this:

kafka.config.fetch.message.max.bytes    3145728
kafka.config.group.id   spout_readonly
kafka.config.rebalance.backoff.ms       6000
kafka.config.rebalance.max.retries      6
kafka.config.zookeeper.connect  dare-broker00.sv.walmartlabs.com:2181,
kafka.config.zookeeper.session.timeout.ms       60000

what could possibly cause this huge lag? Will broker be a bottle neck, or
some config need to be adjusted? The server side config is like this:


# The send buffer (SO_SNDBUF) used by the socket server

# The receive buffer (SO_RCVBUF) used by the socket server

# The maximum size of a request that the socket server will accept
(protection against OOM)

Any help appreciated!

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message