kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mathias Herberts <mathias.herbe...@gmail.com>
Subject Partitioner interface
Date Tue, 04 Oct 2011 14:37:03 GMT

the current Partitioner interface is the following:

interface Partitioner<T> {
   int partition(T key, int numPartitions);

this allows for partition selection based solely on message (or
message set) key and number of partitions.

Message order guarantees provided by Kafka are per partition, a
partition being a pair (brokerid, partitionid).

With the current Partitioner interface, the partition selection is a
blind process. Some use cases might better benefit from a partition
selection process which exposes the brokerid/partitionid info.

Imagine a Kafka cluster that currently has 2 brokers, each with one
partition. Messages are dispatched to both partitions with a custom
Partitioner, sending messages witht the same key to the same

Now suppose we add a broker with one partition. With the current
Partitioner interface, you cannot ensure that messages with a given
key that were sent to a given partition will still be sent to the same
partition because you have no info on the brokerid/partitionid that a
given partition index will select. In some use cases this can be bad
as adding a broker will induce the same kind of disturbance as a
broker failure.

Could the Partitioner interface be modified to include the sorted list
of broker/partition as an additional parameter?

View raw message