kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Kafka streaming partition assignment
Date Mon, 14 May 2018 03:48:37 GMT
It depends on your version. The behavior is known and we put one
improvement into 1.1 release: https://github.com/apache/kafka/pull/4410

Thus, it's "by design" (for 1.0 and older) but we we want to improve it.
Cf: https://issues.apache.org/jira/browse/KAFKA-4969

-Matthias

On 5/13/18 7:52 PM, Liam Clarke wrote:
> Hi all,
> 
> We are running a KStreaming app with a basic topology of
> 
> consume from topic A -> transform and write through topic B (making the app
> a consumer of topic B also) ->  finally write to topic C
> 
> We are running it with two instances of the application. Topic A has 100
> partitions, topics B and C both have 10 partitions.
> 
> What we've noticed in the consumer group info that is confusing us is that
> instance 1 of the app will own 45 partitions of topic A and all 10
> partitions of topic B, whereas the 2nd instance will own 55 partitions of
> topic A and 0 of topic B.
> 
> Our expectation is that we'd see an even split of partitions across both
> instances - is this a realistic expectation? Or is this working as intended?
> 
> Regards,
> 
> Liam Clarke
> 


Mime
View raw message