kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Liam Clarke <liam.cla...@adscale.co.nz>
Subject Kafka streaming partition assignment
Date Mon, 14 May 2018 02:52:01 GMT
Hi all,

We are running a KStreaming app with a basic topology of

consume from topic A -> transform and write through topic B (making the app
a consumer of topic B also) ->  finally write to topic C

We are running it with two instances of the application. Topic A has 100
partitions, topics B and C both have 10 partitions.

What we've noticed in the consumer group info that is confusing us is that
instance 1 of the app will own 45 partitions of topic A and all 10
partitions of topic B, whereas the 2nd instance will own 55 partitions of
topic A and 0 of topic B.

Our expectation is that we'd see an even split of partitions across both
instances - is this a realistic expectation? Or is this working as intended?

Regards,

Liam Clarke

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message