kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Daniel Schierbeck <da...@zendesk.com.INVALID>
Subject Re: Kafka for event sourcing architecture
Date Tue, 17 May 2016 17:32:07 GMT
I'm not sure why Kafka at least in theory cannot be used for infinite
retention – any replicated database system would need to have a new node
ingest all the data from failed node from its replicas. Surely this is no
different in S3 itself. Why is this harder to do in Kafka than in other
systems? The benefit of having just a single message log system would be
rather big.

On Tue, May 17, 2016 at 4:44 AM Tom Crayford <tcrayford@heroku.com> wrote:

> Hi Oli,
>
> Inline.
>
> On Tuesday, 17 May 2016, Olivier Lalonde <olalonde@gmail.com> wrote:
>
> > Hi all,
> >
> > I am considering adopting an "event sourcing" architecture for a system I
> > am developing and Kafka seems like a good choice of store for events.
> >
> > For those who aren't aware, this architecture style consists in storing
> all
> > state changes of the system as an ordered log of events and building
> > derivative views as needed for easier querying (using a SQL database for
> > example). Those views must be completely derived from the event log alone
> > so that the log effectively becomes a "single source of truth".
> >
> > I was wondering if anyone else is using Kafka for that purpose and more
> > specifically:
> >
> > 1) Can Kafka store messages permanently?
>
>
> No. Whilst you can tweak config and such to get a very long retention
> period, this doesn't work well with Kafka at all. Keeping data around
> forever has severe impacts on the operability of your cluster. For example,
> if a machine fails, a replacement would have to catch up with vast
> quantities of data from its replicas. Currently we (Heroku Kafka) restrict
> our customers to a maximum of 14 days of retention, because of all the
> operational headaches of more retention than that. Of course on your own
> cluster you *can* set it as high as you like, this is just an anecdotal
> experience thing from a team that runs thousands of clusters - infinite
> retention is an operational disaster waiting to happen.
>
> Whilst Kafka does have a replay mechanism, that should mostly be though of
> as a mechanism for handling other system failures. E.g. If the database you
> store indexed views is is down, Kafka's replay and retention mechanisms
> mean you're not losing data whilst restoring the availability of that
> database.
>
> What we typically suggest customers do when they ask about this use case is
> to use Kafka as a messaging system, but use e.g. S3 as the long term store.
> Kafka can help with batching writes up to S3 (see e.g. Pintrest's secor
> project), and act as a very high throughput, durable, replicated messaging
> layer for communication. In this paradigm, when you want to replay, you do
> so out of S3 until you've consumed the last offset there, then start
> replaying out of and catching up with the small amount of remaining data in
> Kafka. Of course the replay logic there has to be hand rolled, as Kafka and
> its clients have no knowledge of external stores.
>
> Another potential thing to look at is Kafka's compacted topic mechanism.
> With compacted topics, Kafka keeps the latest element for a given key,
> making it act a little more like a database table. Note that you still have
> to consume by offset here - there's no "get the value for key Y
> operation". However, this assumes that your keyspace is still tractably
> small, and that you're ok with keeping only the latest value. Compaction
> completely overrides time based retention, so you have to "delete" keys or
> have a bounded keyspace if you want to retain operational sanity with
> Kafka. I'd recommend reading the docs on compacted topics, they cover the
> use cases quite well.
>
>
>
> >
> > 2) Let's say I throw away my derived view and want to re-build it from
> > scratch, is it possible to consume messages from a topic from its very
> > first message and once it has caught up, listen for new messages like it
> > would normally do?
>
>
> That's entirely possible, you can catch up from the first retained message
> and then continue from there very easily. However, see above about infinite
> retention.
>
>
>
> >
> > 2) Does it support transactions? Let's say I want to push 3 messages
> > atomically but the producer process crashes after sending only 2
> messages,
> > is it possible to "rollback" the first 2 messages (e.g. "all or nothing"
> > semantics)?
>
>
> No. Kafka at the moment only supports "at least once" semantics, and there
> are no cross broker transactions of any kind. Implementing such a thing
> would likely have huge negative impacts on the current performance
> characteristics of Kafka, which would be a issue for many users.
>
>
> >
> > 3) Does it support request/response style semantics or can they be
> > simulated? My system's primary interface with the outside world is an
> HTTP
> > API so it would be nice if I could publish an event and wait for all the
> > internal services which need to process the event to be "done"
> > processing before returning a response.
>
>
>
> In theory that's possible - the producer can return the offset of the
> message produced, and you could check the latest offset of each consumer in
> your web request handler.
>
> However, doing so is not going to work that well, unless you're ok with
> your web requests taking on the order of seconds to tens of seconds to
> fulfill. Kafka can do low latency messaging reasonably well, but
> coordinating the offsets of many consumers would likely have a huge latency
> impact. Writing the code for it and getting it handling failure correctly
> would likely be a lot of work (there's nothing in any of the client
> libraries like this, because it is not a desirable or supported use case).
>
> Instead I'd like to query *why* you need those semantics? What's the issue
> with just producing a message and telling the user HTTP 200 and later
> consuming it.
>
>
>
> >
> > PS: I'm a Node.js/Go developer so when possible please avoid Java centric
> > terminology.
>
>
> Please to note that the node and go clients are notably less mature than
> the JVM clients, and that running Kafka in production means knowing enough
> about the JVM and Zookeeper to handle that.
>
> Thanks!
> Tom Crayford
> Heroku Kafka
>
> >
> > Thanks!
> >
> > - Oli
>
> >
> > --
> > - Oli
> >
> > Olivier Lalonde
> > http://www.syskall.com
> <http://www.syskall.com> <-- connect with me!
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message