kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ward <tim.w...@origamienergy.com>
Subject Restructure data in GlobakKTable
Date Fri, 09 Aug 2019 13:32:05 GMT
I can arrange for there to be a topic which creates a GlobalKTable which satisfies one of the
lookup needs for my Streams application. Basically this is keyed on "widget ID" and the value
includes a "foobar ID". So this lets me do a join on "widget ID" and find out the "foobar
ID", and that works fine.

**BUT** at some points in my application I also need to do a lookup the other way around,
from "foobar ID" to "widget ID" (yes, this will produce multiple values, but given the application
"multiple" is usually one and is unlikely ever to be more than a handful).

If I could get a changelog stream off the GlobalKTable I could use this to create and maintain
a new table which does lookups the other way, but I can't see any way of getting a stream
off the GlobalKTable?

I could, I suppose, do either of

(1) Arrange for the source of the input topic to create two topics, with the data keyed appropriately,
and create two GlobalKTables from the two topics. But this is very clunky, as it requires
the other application to have to know about and supply this application's internal data structure

(2) Build a separate topology reading from the original topic, restructuring the data as needed,
and building a state store. But how do I ensure that this topology will re-read the (compacted)
input topic from the start every time? (I'm assuming StreamsBuilder.GlobalTable() knows how
to do this, but I don't know how to do this if using StreamsBuilder.stream().)

What am I missing please?

Tim Ward

This email is from Origami Energy Limited. The contents of this email and any attachment are
confidential to the intended recipient(s). If you are not an intended recipient: (i) do not
use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message