kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: Restructure data in GlobakKTable
Date Wed, 21 Aug 2019 06:09:21 GMT
Sounds reasonable.

Note that a `GlobalKTable` is designed to read data from an topic only
without further processing. That's why you don't get any "output
stream". For fault-tolerance, the input topic is used, and hence, there
is also no "changelog topic".

I agree that option (1) is not ideal with regard to separation of
concerns / decoupling.

For option (2), why do you need the application to re-read the topic?
StateStores are by default fully fault-tolerant and hence, if you so a
`builder.stream....` and create a StateStore/KTable, on restart the
state will still be there.

Or maybe I miss understand your question.


On 8/9/19 6:32 AM, Tim Ward wrote:
> I can arrange for there to be a topic which creates a GlobalKTable which satisfies one
of the lookup needs for my Streams application. Basically this is keyed on "widget ID" and
the value includes a "foobar ID". So this lets me do a join on "widget ID" and find out the
"foobar ID", and that works fine.
> **BUT** at some points in my application I also need to do a lookup the other way around,
from "foobar ID" to "widget ID" (yes, this will produce multiple values, but given the application
"multiple" is usually one and is unlikely ever to be more than a handful).
> If I could get a changelog stream off the GlobalKTable I could use this to create and
maintain a new table which does lookups the other way, but I can't see any way of getting
a stream off the GlobalKTable?
> I could, I suppose, do either of
> (1) Arrange for the source of the input topic to create two topics, with the data keyed
appropriately, and create two GlobalKTables from the two topics. But this is very clunky,
as it requires the other application to have to know about and supply this application's internal
data structure needs.
> (2) Build a separate topology reading from the original topic, restructuring the data
as needed, and building a state store. But how do I ensure that this topology will re-read
the (compacted) input topic from the start every time? (I'm assuming StreamsBuilder.GlobalTable()
knows how to do this, but I don't know how to do this if using StreamsBuilder.stream().)
> What am I missing please?
> Tim Ward
> This email is from Origami Energy Limited. The contents of this email and any attachment
are confidential to the intended recipient(s). If you are not an intended recipient: (i) do
not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

View raw message