kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eno Thereska <eno.there...@gmail.com>
Subject Re: Writing data from kafka-streams to remote database
Date Sun, 05 Mar 2017 14:38:43 GMT
Hi Shimi,

Could you tell us more about your scenario? Kafka Streams uses embedded databases (RocksDb)
to store it's state, so often you don't need to write anything to an external database and
you can query your streams state directly from streams. Have a look at this blog if that matches
your scenario: https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/
<https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-apache-kafka/>.

Cheers
Eno

> On 5 Mar 2017, at 10:48, Shimi Kiviti <shimi.k@gmail.com> wrote:
> 
> Hi Everyone,
> 
> I was wondering about writing data to remote database.
> I see 4 possible options:
> 
>   1. Read from a topic and write to the database.
>   2. Use kafka connect
>   3. Write from anywhere in kafka streams.
>   4. Register a CachedStateStore FlushListener that will send a batch of
>   records when the store flush the records.
> 
> Advantages of #4:
> 
>   - The code is decoupled from the processing code and easier to refactor
>   in the future.
>   - Remove the need for additional kafka topic.
> 
> 
> Thanks,
> 
> Shimi


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message