kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aditya Auradkar <aaurad...@linkedin.com.INVALID>
Subject RE: Unclaimed partitions
Date Tue, 28 Apr 2015 17:15:30 GMT
Hi Dave,

The simple consumer doesn't do any state management across consumer instances. So I'm not
sure how you are assigning partitions in your application code. Did you mean to say that you
are using the high level consumer API?


From: Dave Hamilton [dhamilton@nanigans.com]
Sent: Tuesday, April 28, 2015 7:58 AM
To: users@kafka.apache.org
Subject: Unclaimed partitions

Hi, I am trying to consume a 24-partition topic across 12 processes. Each process is using
the simple consumer API, and each is being assigned two consumer threads. I have noticed when
starting these processes that sometimes some of my processes are not being assigned any partitions,
and no rebalance seems to ever be triggered, leaving some of the partitions unclaimed.

When I first tried deploying this yesterday, I noticed 8 of the 24 partitions, for 4 of the
consumer processes, went unclaimed. Redeploying shortly later corrected this issue. I tried
deploying again today, and now I see a different set of 4 processes not getting assigned partitions.
The processes otherwise appear to be running normally, they are currently running in production
and we are working to get the consumers quietly running before enabling them to do any work.
I’m not sure if we might be looking at some sort of timing issue.

Does anyone know what might be causing the issues we’re observing?


View raw message