kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Snehal Nagmote <nagmote.sne...@gmail.com>
Subject Re: Question on partitions while consuming multiple topics
Date Wed, 06 Jul 2016 06:44:50 GMT
Just an update, as I was reading about Kafka Streams, this functionality is
by default supported with Kafka Streams Library.
Following links are really helpful

http://docs.confluent.io/3.0.0/streams/developer-guide.html#partition-grouper
https://github.com/apache/kafka/blob/0.10.0/streams/src/main/java/org/apache/kafka/streams/processor/DefaultPartitionGrouper.java

(Kafka Stream is supported with 0.10.0)

Thanks,
Snehal

On 5 July 2016 at 11:47, Snehal Nagmote <nagmote.snehal@gmail.com> wrote:

> Hello Yardena ,
>
> You may want to take a look at manual assignment for partitions section
> mentioned here ,
>
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
>  .
>
> However I have not tried using this for multiple topics , but looking at
> api , it should be doable.
>
> You have to use same partitioning method which was used by producer to
> determine the correct partition for consumer process for multiple topics.
>
> Note that , you would lose the ordering guarantee with this approach since
> Kafka guarantees ordering within partition for a single topic ,
>
> Thanks,
> Snehal
>
>
> On 4 July 2016 at 07:50, Yardena Meymann <yardenam@feature.fm> wrote:
>
>> Hi,
>>
>> We have several topics, same number of partitions for each, same key used
>> for all topics.
>> We also have several processes consuming the topics (one consumer group).
>> What we wish would happen is that messages with the same key would end up
>> consumed by the same process, regardless of the topic.
>> Can it be achieved with Kafka? What is needed for that?
>>
>> Thanks in advance,
>>   Yardena
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message