kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Kafka Streams dynamic partitioning
Date Tue, 04 Oct 2016 23:00:54 GMT
By default the partitioner will use murmur hash on the key and mode on
current num.partitions to determine which partitions to go to, so records
with the same key will be assigned to the same partition. Would that be OK
for your case?


Guozhang


On Tue, Oct 4, 2016 at 3:00 PM, Adrienne Kole <adriennekole1@gmail.com>
wrote:

> Hi,
>
> From Streams documentation, I can see that each Streams instance is
> processing data independently (from other instances), reads from topic
> partition(s) and writes to specified topic.
>
>
> So here, the partitions of topic should be determined beforehand and should
> remain static.
> In my usecase I want to create partitioned/keyed (time) windows and
> aggregate them.
> I can partition the incoming data to specified topic's partitions and each
> Stream instance can do windowed aggregations.
>
> However, if I don't know the number of possible keys (to partition), then
> what should I do?
>
> Thanks
> Adrienne
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message