kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Partitioner interface
Date Tue, 04 Oct 2011 16:00:54 GMT
Mathias,

Yes, this is currently a limitation. In this design, we try not to expose
too much physical information to the user.

In 0.8, we plan to make partitions logical. A partition won't be tied to a
broker. If the number of the partitions grows, the existing partitions still
have the same partition ids. Hopefully, this will solve  most of your
problem.

Thanks,

Jun

On Tue, Oct 4, 2011 at 7:37 AM, Mathias Herberts <mathias.herberts@gmail.com
> wrote:

> Hi,
>
> the current Partitioner interface is the following:
>
> interface Partitioner<T> {
>   int partition(T key, int numPartitions);
> }
>
> this allows for partition selection based solely on message (or
> message set) key and number of partitions.
>
> Message order guarantees provided by Kafka are per partition, a
> partition being a pair (brokerid, partitionid).
>
> With the current Partitioner interface, the partition selection is a
> blind process. Some use cases might better benefit from a partition
> selection process which exposes the brokerid/partitionid info.
>
> Imagine a Kafka cluster that currently has 2 brokers, each with one
> partition. Messages are dispatched to both partitions with a custom
> Partitioner, sending messages witht the same key to the same
> partition.
>
> Now suppose we add a broker with one partition. With the current
> Partitioner interface, you cannot ensure that messages with a given
> key that were sent to a given partition will still be sent to the same
> partition because you have no info on the brokerid/partitionid that a
> given partition index will select. In some use cases this can be bad
> as adding a broker will induce the same kind of disturbance as a
> broker failure.
>
> Could the Partitioner interface be modified to include the sorted list
> of broker/partition as an additional parameter?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message