kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Auradkar <aaurad...@linkedin.com.INVALID>
Subject RE: Unclaimed partitions
Date Thu, 30 Apr 2015 03:30:10 GMT
Hey Dave,

It's hard to say why this is happening without more information. Even if there are no errors
in the log, is there anything to indicate that the rebalance process on those hosts even started?
Does this happen occasionally or every time you start the consumer group? Can you paste the
output of ConsumerOffsetChecker and describe topic?

From: Dave Hamilton [dhamilton@nanigans.com]
Sent: Wednesday, April 29, 2015 6:46 PM
To: users@kafka.apache.org; users@kafka.apache.org
Subject: Re: Unclaimed partitions

Hi, would anyone be able to help me with this issue? Thanks.

- Dave

On Tue, Apr 28, 2015 at 1:32 PM -0700, "Dave Hamilton" <dhamilton@nanigans.com<mailto:dhamilton@nanigans.com>>

1. We’re using version
2. No failures in the consumer logs
3. We’re using the ConsumerOffsetChecker to see what partitions are assigned to the consumer
group and what their offsets are. 8 of the 12 process each have been assigned two partitions
and they’re keeping up with the topic. The other 4 do not get assigned partitions and no
consumers in the group are consuming those 8 partitions.

Thanks for your help,

On 4/28/15, 1:40 PM, "Aditya Auradkar" <aauradkar@linkedin.com.INVALID> wrote:

>Couple of questions:
>- What version of the consumer API are you using?
>- Are you seeing any rebalance failures in the consumer logs?
>- How do you determine that some partitions are unassigned? Just confirming that you have
partitions that are not being consumed from as opposed to consumer threads that aren't assigned
any partitions.
>From: Dave Hamilton [dhamilton@nanigans.com]
>Sent: Tuesday, April 28, 2015 10:19 AM
>To: users@kafka.apache.org
>Subject: Re: Unclaimed partitions
>I’m sorry, I forgot to specify that these processes are in the same consumer group.
>On 4/28/15, 1:15 PM, "Aditya Auradkar" <aauradkar@linkedin.com.INVALID> wrote:
>>Hi Dave,
>>The simple consumer doesn't do any state management across consumer instances. So
I'm not sure how you are assigning partitions in your application code. Did you mean to say
that you are using the high level consumer API?
>>From: Dave Hamilton [dhamilton@nanigans.com]
>>Sent: Tuesday, April 28, 2015 7:58 AM
>>To: users@kafka.apache.org
>>Subject: Unclaimed partitions
>>Hi, I am trying to consume a 24-partition topic across 12 processes. Each process
is using the simple consumer API, and each is being assigned two consumer threads. I have
noticed when starting these processes that sometimes some of my processes are not being assigned
any partitions, and no rebalance seems to ever be triggered, leaving some of the partitions
>>When I first tried deploying this yesterday, I noticed 8 of the 24 partitions, for
4 of the consumer processes, went unclaimed. Redeploying shortly later corrected this issue.
I tried deploying again today, and now I see a different set of 4 processes not getting assigned
partitions. The processes otherwise appear to be running normally, they are currently running
in production and we are working to get the consumers quietly running before enabling them
to do any work. I’m not sure if we might be looking at some sort of timing issue.
>>Does anyone know what might be causing the issues we’re observing?

View raw message