kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joel Koshy <jjkosh...@gmail.com>
Subject Re: consumer partition rebalancing
Date Tue, 10 Sep 2013 23:36:15 GMT
>         * Assuming 1) theconsumerdoesn'tcontrolthe partition allocation within a topic
and 2) theconstraintthat a single consumer C(i) within a consumer group C(g) must be the only
reader of that partition:
>         * Are there ways to scale partition consumption if C(i) cannot keep up?

To some degree - by either reducing the number of streams that C(i) is
configured with; or by removing it altogether.

>         * A way to designate particular consumers should consume particular partitions
based on consumer capability (similar to container allocation in YARN)?
>         * A way to designate sticky consumer / partition allocation so that a new consumer
C(j) or new broker B(k) or new topic T(l) does not cause C(i) to lose particular partitions
during rebalancing?

Right now, no. I think what you may be effectively suggesting is to
support some form of pluggable logic for partition ownership which
should work as long as it is consistent across all consumer instances.
However, a designated assignment list could be achieved by using
SimpleConsumer - although you would then need to write a little code
to react to leader changes. (We currently don't have a
intermediate-level consumer - i.e., a consumer that can be designated
to consume from a given partition and automatically react to leader
changes.)

Joel

Mime
View raw message