kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Process for changing producer partition assignment strategy
Date Fri, 15 Nov 2019 08:21:41 GMT
In Kafka Streams the producer config `PARTITIONER_CLASS_CONFIG` does not
take effect, because Kafka Streams computes and set partition numbers
explicitly and thus the producer does never use the partitioner to
compute a partition, but accepts whatever Kafka Streams specifies on
each `ProducerRecord`.

If you want to change the partitioning strategy, you need to implement a
custom `StreamPartitioner` and pass it into the corresponding methods.
For example, `to()` or `through()`.

Hope this helps.


-Matthias

On 11/14/19 9:51 AM, Mikkel Gadegaard wrote:
> I've set up a POC using KafkaStreams with microservices consuming and
> producing from/to topics. In the beginning I hadn't thought about
> partition strategy, and so I was using the DefaultPartitioner for producer
> partition assignments. My messages have keys (I use these for
> forking/joining), and the keys are time based UUIDs, this causes some
> rather uneven distribution on my topics. I looked around google and
> stumbled on KIP-369 (Alternative Partitioner to Support "Always
> Round-Robin" Selection) and figured that would be what I needed, so since
> 2.4 isn't out yet I borrowed the class from the PR on github, added it to
> my project and added the property to my config, like so:
> 
> streamProperties.put(ProducerConfig.PARTITIONER_CLASS_CONFIG,
> RoundRobinPartitioner.class.getCanonicalName());
> 
> 
> And the round robin strategy works on a newly added topic, spreading
> messages evenly over 4 partitions. But, and I'm finally getting to my
> question, it doesn't seem to have any effect on existing topics, in other
> words, it seems to be continuing to use the DefaultPartitioner for topics
> created before I added the RoundRobinPartioner class to my
> project/properties.
> 
> Is it me that just hasn't understood that it is impossible to change
> strategy for an existing partition or do I have to do something specific
> apart from re-deploying the Microservice containing the producer?
> 
> Thanks
> Mikkel
> 
> --
> 


Mime
View raw message