kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Ward <tim.w...@origamienergy.com.INVALID>
Subject RE: Restructure data in GlobakKTable
Date Wed, 21 Aug 2019 08:39:07 GMT
Thanks. Yes, you're right re (2), I'm new to this stuff.

I've currently got code which scans the GlobalKTable in a punctuator and inverts it to another
data structure, which I cache (a minute seems suitable for my application), as the only "obvious"
way of getting data out of a GlobalKTable (other than the join which is what it's there for
in the first place). (This only seems practical because the amount of data I'm handling here
is quite small.)

But now the requirements have changed and I'm struggling with handling deletions in the new
requirements ... such is life ...

Tim Ward

-----Original Message-----
From: Matthias J. Sax <matthias@confluent.io>
Sent: 21 August 2019 07:09
To: users@kafka.apache.org
Subject: Re: Restructure data in GlobakKTable

Sounds reasonable.

Note that a `GlobalKTable` is designed to read data from an topic only
without further processing. That's why you don't get any "output
stream". For fault-tolerance, the input topic is used, and hence, there
is also no "changelog topic".

I agree that option (1) is not ideal with regard to separation of
concerns / decoupling.

For option (2), why do you need the application to re-read the topic?
StateStores are by default fully fault-tolerant and hence, if you so a
`builder.stream....` and create a StateStore/KTable, on restart the
state will still be there.

Or maybe I miss understand your question.


-Matthias

On 8/9/19 6:32 AM, Tim Ward wrote:
> I can arrange for there to be a topic which creates a GlobalKTable which satisfies one
of the lookup needs for my Streams application. Basically this is keyed on "widget ID" and
the value includes a "foobar ID". So this lets me do a join on "widget ID" and find out the
"foobar ID", and that works fine.
>
> **BUT** at some points in my application I also need to do a lookup the other way around,
from "foobar ID" to "widget ID" (yes, this will produce multiple values, but given the application
"multiple" is usually one and is unlikely ever to be more than a handful).
>
> If I could get a changelog stream off the GlobalKTable I could use this to create and
maintain a new table which does lookups the other way, but I can't see any way of getting
a stream off the GlobalKTable?
>
> I could, I suppose, do either of
>
> (1) Arrange for the source of the input topic to create two topics, with the data keyed
appropriately, and create two GlobalKTables from the two topics. But this is very clunky,
as it requires the other application to have to know about and supply this application's internal
data structure needs.
>
> (2) Build a separate topology reading from the original topic, restructuring the data
as needed, and building a state store. But how do I ensure that this topology will re-read
the (compacted) input topic from the start every time? (I'm assuming StreamsBuilder.GlobalTable()
knows how to do this, but I don't know how to do this if using StreamsBuilder.stream().)
>
> What am I missing please?
>
> Tim Ward
>
> This email is from Origami Energy Limited. The contents of this email and any attachment
are confidential to the intended recipient(s). If you are not an intended recipient: (i) do
not use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.
>

This email is from Origami Energy Limited. The contents of this email and any attachment are
confidential to the intended recipient(s). If you are not an intended recipient: (i) do not
use, disclose, distribute, copy or publish this email or its contents; (ii) please contact
Origami Energy Limited immediately; and then (iii) delete this email. For more information,
our privacy policy is available here: https://origamienergy.com/privacy-policy/. Origami Energy
Limited (company number 8619644) is a company registered in England with its registered office
at Ashcombe Court, Woolsack Way, Godalming, GU7 1LQ.

Mime
View raw message