kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hans Jespersen <h...@confluent.io>
Subject Re: Capturing and storing these Kafka events for query.
Date Fri, 12 Jan 2018 04:34:30 GMT

Another approach would be to create the query first (in something like KSQL) and then send
the Kafka data through the pre-existing streaming query. In this case the results would be
going into various result topics.

Tools like KSQL also let you query historical data but you need to be sure that this won't
create a performance impact if you have a lot of data and are continually query it over and
over from the beginning to the end. It can work if the data set is small but as Manoj already
said so well, it might be better to materialize your data stream into a repository designed
with indexes beyond those in Kafka for faster or more complex interactive queries. 


> On Jan 11, 2018, at 1:33 PM, Manoj Khangaonkar <khangaonkar@gmail.com> wrote:
> Hi,
> If I understood the question correctly , then the better approach is to
> consume events from topic and store in
> your favorite database. Then query the database as needed.
> Querying the topic for messages in kafka is not recommended as that will be
> a linear search.
> regards
> On Thu, Jan 11, 2018 at 8:55 AM, Maria Pilar <pilifer78@gmail.com> wrote:
>> Hi all,
>> I have a requirement to be able to capture and store events for query,
>> and I'm trying to choose the best option for that:
>> 1) Capture the events from a separate topic, store events a state, in
>> order to convert a stream to a table, that means materializing the
>> stream.
>> The option for it, that I'm thinking is to use an external state that
>> is maintained in an external datastore, often a NoSQL (e.g Cassandra).
>> That solution provides the advantage unlimited size and it can be
>> accessed from multiple instances of the application or from different
>> applications.
>> 2) Other options could be query direct in local o internal state or in
>> memory, however, to be able to query directly onto a topic, I would
>> need to search a record by partition and offset, I don't know exactly
>> how to implement that option if it's possible.
>> Cheers.
> -- 
> http://khangaonkar.blogspot.com/

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message