kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brilly Tsang <brilly.ts...@gmail.com>
Subject Re: Does Kafka consumer api provide a way for you to search for partition number of a topic with key?
Date Mon, 22 Jan 2018 14:30:32 GMT
Many thanks Andras!!! That's exactly what I am looking for. 

Regards,
Brilly

⁣Sent from TypeApp ​

On Jan 22, 2018, 21:37, at 21:37, Andras Beni <andrasbeni@cloudera.com> wrote:
>Hi Brilly,
>
>Do I understand right, that you want your consumers to consume only
>from a
>given partition / partitions based on a preconfigured value?
>You need to copy the mechanism the default partitioner (if this is used
>on
>producer side) uses to determine partition.
>(
>https://github.com/apache/kafka/blob/d9b784e1470714c8b04e7c3d74f626a96ca1591e/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L68
> )
>You can do this the following way
>
>import org.apache.kafka.common.utils.Utils;
>
>String topicToConsume = ...;
>
>Object theKeyYouWantToConsume = ...;
>
>byte[] theKeyAsByteArray = yourKeySerializer.serialize(topicToConsume,
>theKeyYouWantToConsume);
>
>int partition = Utils.toPositive(Utils.murmur2(theKeyAsByteArray)) %
>partitionCount;
>
>consumer.assign(Collections.singletonList(new
>TopicPartition(topicToConsume,
>partition))));
>
>However, still this way you will receive messages with different keys
>that
>map to the same partition.
>I recommend you create separate topics for separate kinds of messages
>and
>rely on default partition assignment instead of producing all messages
>to
>the same topic and assigning partitions manually. And if you still need
>to
>consume all messages as if they were in one stream, you still can
>subscribe
>to multiple topics with a pattern.
>
>HTH,
>Andras
>
>
>
>On Mon, Jan 22, 2018 at 3:08 AM, TSANG, Brilly
><brilly.tsang@hk.daiwacm.com>
>wrote:
>
>> Hi folks,
>>
>>
>> I'm working with a topic that have many messages. Kafka is scale
>> horizontally. As a result, when they spread out to multiple
>processes, only
>> 1 will work with the specific key. The rest are not related and
>should stop
>> the processing.
>>
>> Is there a way from the client API to hash the partition number from
>the
>> key? With the partition number, we can stop the other processor from
>> wasting cycle and start processing other events
>> Regards,
>> Brilly
>>
>>
>>
>> ________________________________
>>
>> ************************************************************
>> ********************
>> DISCLAIMER:
>> This email and any attachment(s) are intended solely for the
>person(s)
>> named above, and are or may contain information of a proprietary or
>> confidential nature. If you are not the intended recipient(s), you
>should
>> delete this message immediately. Any use, disclosure or distribution
>of
>> this message without our prior consent is strictly prohibited.
>> This message may be subject to errors, incomplete or late delivery,
>> interruption, interception, modification, or may contain viruses.
>Neither
>> Daiwa Capital Markets Hong Kong Limited, its subsidiaries, affiliates
>nor
>> their officers or employees represent or warrant the accuracy or
>> completeness, nor accept any responsibility or liability whatsoever
>for any
>> use of or reliance upon, this email or any of the contents hereof.
>The
>> contents of this message are for information purposes only, and
>subject to
>> change without notice.
>> This message is not and is not intended to be an offer or
>solicitation to
>> buy or sell any securities or financial products, nor does any
>> recommendation, opinion or advice necessarily reflect those of Daiwa
>> Capital Markets Hong Kong Limited, its subsidiaries or affiliates.
>> ************************************************************
>> ********************
>>

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message