kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anatoly Pulyaevskiy <anatoly.pulyaevs...@gmail.com>
Subject KafkaStreams StateStore as EventStore (Event Sourcing)
Date Fri, 16 Dec 2016 02:10:50 GMT
Hi everyone,

I've been reading a lot about new features in Kafka Streams and everything
looks very promising. There is even an article on Kafka and Event Sourcing:
https://www.confluent.io/blog/event-sourcing-cqrs-stream-processing-apache-kafka-whats-connection/

There are a couple of things that I'm concerned about though. For Event
Sourcing it is assumed that there is a way to fetch all events for a
particular object and replay them in order to get "latest snapshot" of that
object.

It seems like (and the article says so) that StateStore in KafkaStreams can
be used to achieve that.

My first question is would it scale well for millions of objects?
I understand that StateStore is backed by a compacted Kafka topic so in an
event of failure KafkaStreams will recover to the latest state by reading
all messages from that topic. But my suspicion is that for millions of
objects this may take a while (it would need to read the whole partition
for each object), is this a correct assumption?

My second question is would it make more sense to use an external DB in
such case or is there a "best practice" around implementing Event Sourcing
and using Kafka's internal StateStore as EventStore?

Thanks,
Anatoly

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message