kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manoj Khangaonkar <khangaon...@gmail.com>
Subject Re: Capturing and storing these Kafka events for query.
Date Thu, 11 Jan 2018 21:33:29 GMT

If I understood the question correctly , then the better approach is to
consume events from topic and store in
your favorite database. Then query the database as needed.

Querying the topic for messages in kafka is not recommended as that will be
a linear search.


On Thu, Jan 11, 2018 at 8:55 AM, Maria Pilar <pilifer78@gmail.com> wrote:

> Hi all,
> I have a requirement to be able to capture and store events for query,
> and I'm trying to choose the best option for that:
> 1) Capture the events from a separate topic, store events a state, in
> order to convert a stream to a table, that means materializing the
> stream.
> The option for it, that I'm thinking is to use an external state that
> is maintained in an external datastore, often a NoSQL (e.g Cassandra).
> That solution provides the advantage unlimited size and it can be
> accessed from multiple instances of the application or from different
> applications.
> 2) Other options could be query direct in local o internal state or in
> memory, however, to be able to query directly onto a topic, I would
> need to search a record by partition and offset, I don't know exactly
> how to implement that option if it's possible.
> Cheers.


  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message