kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Taylor Gautier <tgaut...@tagged.com>
Subject Re: Partitioner interface
Date Tue, 04 Oct 2011 15:19:54 GMT
A possibly related question - is it possible to use the interface to
select a partition based on topic name?



On Oct 4, 2011, at 7:37 AM, Mathias Herberts <mathias.herberts@gmail.com> wrote:

> Hi,
>
> the current Partitioner interface is the following:
>
> interface Partitioner<T> {
>   int partition(T key, int numPartitions);
> }
>
> this allows for partition selection based solely on message (or
> message set) key and number of partitions.
>
> Message order guarantees provided by Kafka are per partition, a
> partition being a pair (brokerid, partitionid).
>
> With the current Partitioner interface, the partition selection is a
> blind process. Some use cases might better benefit from a partition
> selection process which exposes the brokerid/partitionid info.
>
> Imagine a Kafka cluster that currently has 2 brokers, each with one
> partition. Messages are dispatched to both partitions with a custom
> Partitioner, sending messages witht the same key to the same
> partition.
>
> Now suppose we add a broker with one partition. With the current
> Partitioner interface, you cannot ensure that messages with a given
> key that were sent to a given partition will still be sent to the same
> partition because you have no info on the brokerid/partitionid that a
> given partition index will select. In some use cases this can be bad
> as adding a broker will induce the same kind of disturbance as a
> broker failure.
>
> Could the Partitioner interface be modified to include the sorted list
> of broker/partition as an additional parameter?

Mime
View raw message