kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brilly Tsang <brilly.ts...@gmail.com>
Subject Re: Does Kafka consumer api provide a way for you to search for partition number of a topic with key?
Date Mon, 22 Jan 2018 14:30:32 GMT
Many thanks Andras!!! That's exactly what I am looking for. 


⁣Sent from TypeApp ​

On Jan 22, 2018, 21:37, at 21:37, Andras Beni <andrasbeni@cloudera.com> wrote:
>Hi Brilly,
>Do I understand right, that you want your consumers to consume only
>from a
>given partition / partitions based on a preconfigured value?
>You need to copy the mechanism the default partitioner (if this is used
>producer side) uses to determine partition.
> )
>You can do this the following way
>import org.apache.kafka.common.utils.Utils;
>String topicToConsume = ...;
>Object theKeyYouWantToConsume = ...;
>byte[] theKeyAsByteArray = yourKeySerializer.serialize(topicToConsume,
>int partition = Utils.toPositive(Utils.murmur2(theKeyAsByteArray)) %
>However, still this way you will receive messages with different keys
>map to the same partition.
>I recommend you create separate topics for separate kinds of messages
>rely on default partition assignment instead of producing all messages
>the same topic and assigning partitions manually. And if you still need
>consume all messages as if they were in one stream, you still can
>to multiple topics with a pattern.
>On Mon, Jan 22, 2018 at 3:08 AM, TSANG, Brilly
>> Hi folks,
>> I'm working with a topic that have many messages. Kafka is scale
>> horizontally. As a result, when they spread out to multiple
>processes, only
>> 1 will work with the specific key. The rest are not related and
>should stop
>> the processing.
>> Is there a way from the client API to hash the partition number from
>> key? With the partition number, we can stop the other processor from
>> wasting cycle and start processing other events
>> Regards,
>> Brilly
>> ________________________________
>> ************************************************************
>> ********************
>> This email and any attachment(s) are intended solely for the
>> named above, and are or may contain information of a proprietary or
>> confidential nature. If you are not the intended recipient(s), you
>> delete this message immediately. Any use, disclosure or distribution
>> this message without our prior consent is strictly prohibited.
>> This message may be subject to errors, incomplete or late delivery,
>> interruption, interception, modification, or may contain viruses.
>> Daiwa Capital Markets Hong Kong Limited, its subsidiaries, affiliates
>> their officers or employees represent or warrant the accuracy or
>> completeness, nor accept any responsibility or liability whatsoever
>for any
>> use of or reliance upon, this email or any of the contents hereof.
>> contents of this message are for information purposes only, and
>subject to
>> change without notice.
>> This message is not and is not intended to be an offer or
>solicitation to
>> buy or sell any securities or financial products, nor does any
>> recommendation, opinion or advice necessarily reflect those of Daiwa
>> Capital Markets Hong Kong Limited, its subsidiaries or affiliates.
>> ************************************************************
>> ********************

  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message