kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Finding StreamsMetadata with value-dependent partitioning
Date Fri, 02 Jun 2017 21:11:31 GMT
I am not sure if I understand the use case correctly. Could you give
some more context?

> backing store whose partitioning is value dependent

In infer that you are using a custom store and not default RocksDB? If
yes, what do you use? What does "value dependent" mean in this context?

Right now, I am wondering, why you not just set a new key to get your
data grouped by the field you are interesting in? Also, if you don't
partitioned your data by key, you might break your streams application
with regard to fault-tolerance -- or does your custom store not rely on
changelog backup for fault-tolerance?


On 6/2/17 10:34 AM, Steven Schlansker wrote:
> I have a KTable and backing store whose partitioning is value dependent.
> I want certain groups of messages to be ordered and that grouping is determined
> by one field (D) of the (possibly large) value.
> When I lookup by only K, obviously you don't know the partition it should be on.
> So I will build a GlobalKTable of K -> D.  This gives me enough information
> to determine the partition.
> Unfortunately, the KafkaStreams metadata API doesn't fit this use case well.
> It allows you to either get all metadata, or by key -- but if you lookup by key
> it just substitutes a null value (causing a downstream NPE)
> I can iterate over all metadata and compute the mapping of K -> K,D -> P
> and then iterate over all metadata looking for P.  It's not difficult but ends
> up being a bit of somewhat ugly code that feels like I shouldn't have to write it.
> Am I missing something here?  Is there a better way that I've missed?  Thanks!

View raw message