kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrian Jardan <andrianjar...@gmail.com>
Subject Kafka Streams & Distributed state question
Date Wed, 13 Feb 2019 11:16:19 GMT
Hello dear Kafka community,

We are planning to build a Kafka Streams application that will build a pretty big state (~100Gb)
in real time from various sources.

We later on need to query this state as fast as possible, and the main idea is to use the
built-in in-memory RockDB and get the data via,
some sort of REST API we will build on top. The question is how do we identify where the data
we need is, since the state will obviously
not fit in the memory of a single instance and we need to scale somehow in case this state
keeps growing, and it will…

Is there a way to identify where the data we care about resides, on what kafka streams app
instance ?

I tried to find the answer in the documentation, but was not able to figure it our unfortunately.

Thank you in advance !

Andrian Jardan
Infrastructure and DevOps expert
cell: +49 174 2815994
Skype: macrosdnb

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message