kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Posta <christian.po...@gmail.com>
Subject Re: Kafka for event sourcing architecture
Date Wed, 18 May 2016 00:01:10 GMT
Please create a JIRA with your thoughts. I'd be happy to help out with
something like that.

On Tue, May 17, 2016 at 4:57 PM, Radoslaw Gruchalski <radek@gruchalski.com>
wrote:

> Not as far as I'm aware. I'd be happy to contribute if there is a desire
> to have such feature. From experience with other projects, I know that
> without the initial pitch / discussion, it could be difficult to get such
> feature in. I can create a jira in the morning, no electricity again
> tonight :-/
>
> Get Outlook for iOS
>
>
>
>
> On Tue, May 17, 2016 at 4:53 PM -0700, "Christian Posta" <
> christian.posta@gmail.com> wrote:
>
>
>
>
>
>
>
>
>
>
> +1 to your solution of log.cleanup.policy. Other brokers (ie, ActiveMQ)
> have a feature like that.
> Is there a JIRA for this?
>
> On Tue, May 17, 2016 at 4:48 PM, Radoslaw Gruchalski
> wrote:
>
> > I have described a cold storage solution for Kafka:
> >
> https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv
> .
> > Also described it here a couple of times. Thd potential solution seems
> > rather straightforward.
> > Get Outlook for iOS
> >
> >     _____________________________
> > From: Luke Steensen
> > Sent: Tuesday, May 17, 2016 11:22 pm
> > Subject: Re: Kafka for event sourcing architecture
> > To:
> >
> >
> > It's harder in Kafka because the unit of replication is an entire
> > partition, not a single key/value pair. Partitions are large and
> constantly
> > growing, where key/value pairs are typically much smaller and don't
> change
> > in size. There would theoretically be no difference if you had one
> > partition per key, but that's not practical. Instead, you end up trying
> to
> > pick a number of partitions big enough that they'll each be a reasonable
> > size for the foreseeable future but not so big that the cluster overhead
> is
> > untenable. Even then the clock is ticking towards the day your biggest
> > partition approaches the limit of storage available on a single machine.
> >
> > It's frustrating because, as you say, there would be enormous benefits to
> > being able to access all data through the same system. Unfortunately, it
> > seems too far away from Kafka's original use case to be practical.
> >
> >
> > On Tue, May 17, 2016 at 12:32 PM, Daniel Schierbeck <
> > dasch@zendesk.com.invalid> wrote:
> >
> > > I'm not sure why Kafka at least in theory cannot be used for infinite
> > > retention – any replicated database system would need to have a new
> node
> > > ingest all the data from failed node from its replicas. Surely this is
> no
> > > different in S3 itself. Why is this harder to do in Kafka than in other
> > > systems? The benefit of having just a single message log system would
> be
> > > rather big.
> > >
> > > On Tue, May 17, 2016 at 4:44 AM Tom Crayford
> > wrote:
> > >
> > > > Hi Oli,
> > > >
> > > > Inline.
> > > >
> > > > On Tuesday, 17 May 2016, Olivier Lalonde  wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I am considering adopting an "event sourcing" architecture for a
> > > system I
> > > > > am developing and Kafka seems like a good choice of store for
> events.
> > > > >
> > > > > For those who aren't aware, this architecture style consists in
> > storing
> > > > all
> > > > > state changes of the system as an ordered log of events and
> building
> > > > > derivative views as needed for easier querying (using a SQL
> database
> > > for
> > > > > example). Those views must be completely derived from the event log
> > > alone
> > > > > so that the log effectively becomes a "single source of truth".
> > > > >
> > > > > I was wondering if anyone else is using Kafka for that purpose and
> > more
> > > > > specifically:
> > > > >
> > > > > 1) Can Kafka store messages permanently?
> > > >
> > > >
> > > > No. Whilst you can tweak config and such to get a very long retention
> > > > period, this doesn't work well with Kafka at all. Keeping data around
> > > > forever has severe impacts on the operability of your cluster. For
> > > example,
> > > > if a machine fails, a replacement would have to catch up with vast
> > > > quantities of data from its replicas. Currently we (Heroku Kafka)
> > > restrict
> > > > our customers to a maximum of 14 days of retention, because of all
> the
> > > > operational headaches of more retention than that. Of course on your
> > own
> > > > cluster you *can* set it as high as you like, this is just an
> anecdotal
> > > > experience thing from a team that runs thousands of clusters -
> infinite
> > > > retention is an operational disaster waiting to happen.
> > > >
> > > > Whilst Kafka does have a replay mechanism, that should mostly be
> though
> > > of
> > > > as a mechanism for handling other system failures. E.g. If the
> database
> > > you
> > > > store indexed views is is down, Kafka's replay and retention
> mechanisms
> > > > mean you're not losing data whilst restoring the availability of that
> > > > database.
> > > >
> > > > What we typically suggest customers do when they ask about this use
> > case
> > > is
> > > > to use Kafka as a messaging system, but use e.g. S3 as the long term
> > > store.
> > > > Kafka can help with batching writes up to S3 (see e.g. Pintrest's
> secor
> > > > project), and act as a very high throughput, durable, replicated
> > > messaging
> > > > layer for communication. In this paradigm, when you want to replay,
> you
> > > do
> > > > so out of S3 until you've consumed the last offset there, then start
> > > > replaying out of and catching up with the small amount of remaining
> > data
> > > in
> > > > Kafka. Of course the replay logic there has to be hand rolled, as
> Kafka
> > > and
> > > > its clients have no knowledge of external stores.
> > > >
> > > > Another potential thing to look at is Kafka's compacted topic
> > mechanism.
> > > > With compacted topics, Kafka keeps the latest element for a given
> key,
> > > > making it act a little more like a database table. Note that you
> still
> > > have
> > > > to consume by offset here - there's no "get the value for key Y
> > > > operation". However, this assumes that your keyspace is still
> tractably
> > > > small, and that you're ok with keeping only the latest value.
> > Compaction
> > > > completely overrides time based retention, so you have to "delete"
> keys
> > > or
> > > > have a bounded keyspace if you want to retain operational sanity with
> > > > Kafka. I'd recommend reading the docs on compacted topics, they cover
> > the
> > > > use cases quite well.
> > > >
> > > >
> > > >
> > > > >
> > > > > 2) Let's say I throw away my derived view and want to re-build it
> > from
> > > > > scratch, is it possible to consume messages from a topic from its
> > very
> > > > > first message and once it has caught up, listen for new messages
> like
> > > it
> > > > > would normally do?
> > > >
> > > >
> > > > That's entirely possible, you can catch up from the first retained
> > > message
> > > > and then continue from there very easily. However, see above about
> > > infinite
> > > > retention.
> > > >
> > > >
> > > >
> > > > >
> > > > > 2) Does it support transactions? Let's say I want to push 3
> messages
> > > > > atomically but the producer process crashes after sending only 2
> > > > messages,
> > > > > is it possible to "rollback" the first 2 messages (e.g. "all or
> > > nothing"
> > > > > semantics)?
> > > >
> > > >
> > > > No. Kafka at the moment only supports "at least once" semantics, and
> > > there
> > > > are no cross broker transactions of any kind. Implementing such a
> thing
> > > > would likely have huge negative impacts on the current performance
> > > > characteristics of Kafka, which would be a issue for many users.
> > > >
> > > >
> > > > >
> > > > > 3) Does it support request/response style semantics or can they be
> > > > > simulated? My system's primary interface with the outside world is
> an
> > > > HTTP
> > > > > API so it would be nice if I could publish an event and wait for
> all
> > > the
> > > > > internal services which need to process the event to be "done"
> > > > > processing before returning a response.
> > > >
> > > >
> > > >
> > > > In theory that's possible - the producer can return the offset of the
> > > > message produced, and you could check the latest offset of each
> > consumer
> > > in
> > > > your web request handler.
> > > >
> > > > However, doing so is not going to work that well, unless you're ok
> with
> > > > your web requests taking on the order of seconds to tens of seconds
> to
> > > > fulfill. Kafka can do low latency messaging reasonably well, but
> > > > coordinating the offsets of many consumers would likely have a huge
> > > latency
> > > > impact. Writing the code for it and getting it handling failure
> > correctly
> > > > would likely be a lot of work (there's nothing in any of the client
> > > > libraries like this, because it is not a desirable or supported use
> > > case).
> > > >
> > > > Instead I'd like to query *why* you need those semantics? What's the
> > > issue
> > > > with just producing a message and telling the user HTTP 200 and later
> > > > consuming it.
> > > >
> > > >
> > > >
> > > > >
> > > > > PS: I'm a Node.js/Go developer so when possible please avoid Java
> > > centric
> > > > > terminology.
> > > >
> > > >
> > > > Please to note that the node and go clients are notably less mature
> > than
> > > > the JVM clients, and that running Kafka in production means knowing
> > > enough
> > > > about the JVM and Zookeeper to handle that.
> > > >
> > > > Thanks!
> > > > Tom Crayford
> > > > Heroku Kafka
> > > >
> > > > >
> > > > > Thanks!
> > > > >
> > > > > - Oli
> > > >
> > > > >
> > > > > --
> > > > > - Oli
> > > > >
> > > > > Olivier Lalonde
> > > > > http://www.syskall.com
> > > >  <-- connect with me!
> > > > >
> > > >
> > >
> >
> >
> >
> >
> >
>
>
>
> --
> *Christian Posta*
> twitter: @christianposta
> http://www.christianposta.com/blog
> http://fabric8.io
>
>
>
>
>
>


-- 
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message