kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Partition size for topic
Date Sat, 02 Apr 2016 05:07:01 GMT
Oleg,

Normally the number of partitions doesn't change (or infrequently, at
least) so regardless of how you got the number of partitions there
shouldn't be an inconsistency. Are you actually seeing an inconsistency
causing this exception? And is the number of partitions not changing? Is it
possible a simple off-by-one error is the issue, where you're specifying
`numPartitions` instead of a value in [0, `numPartitions`-1]?

-Ewen

On Mon, Mar 28, 2016 at 2:20 PM, Oleg Zhurakousky <
ozhurakousky@hortonworks.com> wrote:

> Hi
>
> It seems there are several ways to get to the same number in Kafka API.
> In Kafka Partitioner which is invoked by KafkaProducer we have this
>
> public int partition(ProducerRecord<byte[], byte[]> record, Cluster
> cluster) {
>         List<PartitionInfo> partitions =
> cluster.partitionsForTopic(record.topic());
>         int numPartitions = partitions.size();
> . . .
>
> And then we have KafkaProducer.partitionsFor(topicName);
>
> It appears that the two may result in a different number causing failure
> during internal validation in Partitioner
>
> // they have given us a partition, use it
>             if (record.partition() < 0 || record.partition() >=
> numPartitions)
>                 throw new IllegalArgumentException("Invalid partition
> given with record: " + record.partition()
>                                                    + " is not in the range
> [0..."
>                                                    + numPartitions
>                                                    + "].”);
>
>
> Basically we have RoundRobin partitioner that uses
> KafkaProducer.partitionsFor(topicName) to calculate the cycle, but getting
> the above error.
> Could someone please explain the difference between the two methods to get
> partitions size?
>
> Cheers
> Oleg
>
>
>
>
>
>
>
>
>


-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message