kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Snehal Nagmote <nagmote.sne...@gmail.com>
Subject Re: Question on partitions while consuming multiple topics
Date Tue, 05 Jul 2016 18:47:07 GMT
Hello Yardena ,

You may want to take a look at manual assignment for partitions section
mentioned here ,
http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
 .

However I have not tried using this for multiple topics , but looking at
api , it should be doable.

You have to use same partitioning method which was used by producer to
determine the correct partition for consumer process for multiple topics.

Note that , you would lose the ordering guarantee with this approach since
Kafka guarantees ordering within partition for a single topic ,

Thanks,
Snehal


On 4 July 2016 at 07:50, Yardena Meymann <yardenam@feature.fm> wrote:

> Hi,
>
> We have several topics, same number of partitions for each, same key used
> for all topics.
> We also have several processes consuming the topics (one consumer group).
> What we wish would happen is that messages with the same key would end up
> consumed by the same process, regardless of the topic.
> Can it be achieved with Kafka? What is needed for that?
>
> Thanks in advance,
>   Yardena
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message