kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Partitioner interface
Date Tue, 04 Oct 2011 16:01:52 GMT
Taylor,

You can put anything you want in the key. Perhaps you can pass in the topic
name as part of the key?

Thanks,

Jun


On Tue, Oct 4, 2011 at 8:19 AM, Taylor Gautier <tgautier@tagged.com> wrote:

> A possibly related question - is it possible to use the interface to
> select a partition based on topic name?
>
>
>
> On Oct 4, 2011, at 7:37 AM, Mathias Herberts <mathias.herberts@gmail.com>
> wrote:
>
> > Hi,
> >
> > the current Partitioner interface is the following:
> >
> > interface Partitioner<T> {
> >   int partition(T key, int numPartitions);
> > }
> >
> > this allows for partition selection based solely on message (or
> > message set) key and number of partitions.
> >
> > Message order guarantees provided by Kafka are per partition, a
> > partition being a pair (brokerid, partitionid).
> >
> > With the current Partitioner interface, the partition selection is a
> > blind process. Some use cases might better benefit from a partition
> > selection process which exposes the brokerid/partitionid info.
> >
> > Imagine a Kafka cluster that currently has 2 brokers, each with one
> > partition. Messages are dispatched to both partitions with a custom
> > Partitioner, sending messages witht the same key to the same
> > partition.
> >
> > Now suppose we add a broker with one partition. With the current
> > Partitioner interface, you cannot ensure that messages with a given
> > key that were sent to a given partition will still be sent to the same
> > partition because you have no info on the brokerid/partitionid that a
> > given partition index will select. In some use cases this can be bad
> > as adding a broker will induce the same kind of disturbance as a
> > broker failure.
> >
> > Could the Partitioner interface be modified to include the sorted list
> > of broker/partition as an additional parameter?
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message