kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andras Beni <andrasb...@cloudera.com>
Subject Re: Does Kafka consumer api provide a way for you to search for partition number of a topic with key?
Date Mon, 22 Jan 2018 13:37:28 GMT
Hi Brilly,

Do I understand right, that you want your consumers to consume only from a
given partition / partitions based on a preconfigured value?
You need to copy the mechanism the default partitioner (if this is used on
producer side) uses to determine partition.
(
https://github.com/apache/kafka/blob/d9b784e1470714c8b04e7c3d74f626a96ca1591e/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L68
 )
You can do this the following way

import org.apache.kafka.common.utils.Utils;

String topicToConsume = ...;

Object theKeyYouWantToConsume = ...;

byte[] theKeyAsByteArray = yourKeySerializer.serialize(topicToConsume,
theKeyYouWantToConsume);

int partition = Utils.toPositive(Utils.murmur2(theKeyAsByteArray)) %
partitionCount;

consumer.assign(Collections.singletonList(new TopicPartition(topicToConsume,
partition))));

However, still this way you will receive messages with different keys that
map to the same partition.
I recommend you create separate topics for separate kinds of messages and
rely on default partition assignment instead of producing all messages to
the same topic and assigning partitions manually. And if you still need to
consume all messages as if they were in one stream, you still can subscribe
to multiple topics with a pattern.

HTH,
Andras



On Mon, Jan 22, 2018 at 3:08 AM, TSANG, Brilly <brilly.tsang@hk.daiwacm.com>
wrote:

> Hi folks,
>
>
> I'm working with a topic that have many messages. Kafka is scale
> horizontally. As a result, when they spread out to multiple processes, only
> 1 will work with the specific key. The rest are not related and should stop
> the processing.
>
> Is there a way from the client API to hash the partition number from the
> key? With the partition number, we can stop the other processor from
> wasting cycle and start processing other events
> Regards,
> Brilly
>
>
>
> ________________________________
>
> ************************************************************
> ********************
> DISCLAIMER:
> This email and any attachment(s) are intended solely for the person(s)
> named above, and are or may contain information of a proprietary or
> confidential nature. If you are not the intended recipient(s), you should
> delete this message immediately. Any use, disclosure or distribution of
> this message without our prior consent is strictly prohibited.
> This message may be subject to errors, incomplete or late delivery,
> interruption, interception, modification, or may contain viruses. Neither
> Daiwa Capital Markets Hong Kong Limited, its subsidiaries, affiliates nor
> their officers or employees represent or warrant the accuracy or
> completeness, nor accept any responsibility or liability whatsoever for any
> use of or reliance upon, this email or any of the contents hereof. The
> contents of this message are for information purposes only, and subject to
> change without notice.
> This message is not and is not intended to be an offer or solicitation to
> buy or sell any securities or financial products, nor does any
> recommendation, opinion or advice necessarily reflect those of Daiwa
> Capital Markets Hong Kong Limited, its subsidiaries or affiliates.
> ************************************************************
> ********************
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message