kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sybrandy, Casey" <Casey.Sybra...@Six3Systems.com>
Subject Consumer pauses when running many threads
Date Thu, 01 Aug 2013 14:30:16 GMT

We're seeing an issue running 0.7.0 where one or more of our consumers are pausing after about
an hour when we have a lot of threads configured.  Our setup is as follows:

*         4 brokers configured for 32 threads and 32 partitions on each broker.

*         2 consumers each processing 40 streams (24 and 16).

*         Zookeeper server is a CDH version that's at least 3.3.4.

We were also seeing this with 3 consumers running 18 threads each.  As you can tell, the hardware
is quite beefy and the brokers are described as being "bored."

Outside of upgrading to 0.7.2, which we are planning on doing but can't yet, what else can
we look into to try to resolve this or at least determine what's happening?



  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message