kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Consumer pauses when running many threads
Date Thu, 01 Aug 2013 15:12:38 GMT
Have you looked at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Myconsumerseemstohavestopped%2Cwhy%3F?

Thanks,

Jun


On Thu, Aug 1, 2013 at 7:30 AM, Sybrandy, Casey <
Casey.Sybrandy@six3systems.com> wrote:

> Hello,
>
> We're seeing an issue running 0.7.0 where one or more of our consumers are
> pausing after about an hour when we have a lot of threads configured.  Our
> setup is as follows:
>
>
> *         4 brokers configured for 32 threads and 32 partitions on each
> broker.
>
> *         2 consumers each processing 40 streams (24 and 16).
>
> *         Zookeeper server is a CDH version that's at least 3.3.4.
>
> We were also seeing this with 3 consumers running 18 threads each.  As you
> can tell, the hardware is quite beefy and the brokers are described as
> being "bored."
>
> Outside of upgrading to 0.7.2, which we are planning on doing but can't
> yet, what else can we look into to try to resolve this or at least
> determine what's happening?
>
> Thanks.
>
> Casey
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message