kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ewen Cheslack-Postava <e...@confluent.io>
Subject Re: Kafka as an event store for Event Sourcing
Date Sat, 13 Jun 2015 18:59:03 GMT
Daniel: By random read, I meant not reading the data sequentially as is the
norm in Kafka, not necessarily a random disk seek. That in-memory data
structure is what enables the random read. You're either going to need the
disk seek if the data isn't in the fs cache or you're trading memory to
avoid it. If it's a full index containing keys and values then you're
potentially committing to a much larger JVM memory footprint (and all the
GC issues that come with it) since you'd be storing that data in the JVM
heap. If you're only storing the keys + offset info, then you potentially
introduce random disk seeks on any CAS operation (and making page caching
harder for the OS, etc.).


On Sat, Jun 13, 2015 at 11:33 AM, Daniel Schierbeck <
daniel.schierbeck@gmail.com> wrote:

> Ewen: would single-key CAS necessitate random reads? My idea was to have
> the broker maintain an in-memory table that could be rebuilt from the log
> or a snapshot.
> On lør. 13. jun. 2015 at 20.26 Ewen Cheslack-Postava <ewen@confluent.io>
> wrote:
>
> > Jay - I think you need broker support if you want CAS to work with
> > compacted topics. With the approach you described you can't turn on
> > compaction since that would make it last-writer-wins, and using any
> > non-infinite retention policy would require some external process to
> > monitor keys that might expire and refresh them by rewriting the data.
> >
> > That said, I think any addition like this warrants a lot of discussion
> > about potential use cases since there are a lot of ways you could go
> adding
> > support for something like this. I think this is an obvious next
> > incremental step, but someone is bound to have a use case that would
> > require multi-key CAS and would be costly to build atop single key CAS.
> Or,
> > since the compare requires a random read anyway, why not throw in
> > read-by-key rather than sequential log reads, which would allow for
> > minitransactions a la Sinfonia?
> >
> > I'm not convinced trying to make Kafka support traditional key-value
> store
> > functionality is a good idea. Compacted topics made it possible to use
> it a
> > bit more in that way, but didn't change the public interface, only the
> way
> > storage was implemented, and importantly all the potential additional
> > performance costs & data structures are isolated to background threads.
> >
> > -Ewen
> >
> > On Sat, Jun 13, 2015 at 9:59 AM, Daniel Schierbeck <
> > daniel.schierbeck@gmail.com> wrote:
> >
> > > @Jay:
> > >
> > > Regarding your first proposal: wouldn't that mean that a producer
> > wouldn't
> > > know whether a write succeeded? In the case of event sourcing, a failed
> > CAS
> > > may require re-validating the input with the new state. Simply
> discarding
> > > the write would be wrong.
> > >
> > > As for the second idea: how would a client of the writer service know
> > which
> > > writer is the leader? For example, how would a load balancer know which
> > web
> > > app process to route requests to? Ideally, all processes would be able
> to
> > > handle requests.
> > >
> > > Using conditional writes would allow any producer to write and provide
> > > synchronous feedback to the producers.
> > > On fre. 12. jun. 2015 at 18.41 Jay Kreps <jay@confluent.io> wrote:
> > >
> > > > I have been thinking a little about this. I don't think CAS actually
> > > > requires any particular broker support. Rather the two writers just
> > write
> > > > messages with some deterministic check-and-set criteria and all the
> > > > replicas read from the log and check this criteria before applying
> the
> > > > write. This mechanism has the downside that it creates additional
> > writes
> > > > when there is a conflict and requires waiting on the full roundtrip
> > > (write
> > > > and then read) but it has the advantage that it is very flexible as
> to
> > > the
> > > > criteria you use.
> > > >
> > > > An alternative strategy for accomplishing the same thing a bit more
> > > > efficiently is to elect leaders amongst the writers themselves. This
> > > would
> > > > require broker support for single writer to avoid the possibility of
> > > split
> > > > brain. I like this approach better because the leader for a partition
> > can
> > > > then do anything they want on their local data to make the decision
> of
> > > what
> > > > is committed, however the downside is that the mechanism is more
> > > involved.
> > > >
> > > > -Jay
> > > >
> > > > On Fri, Jun 12, 2015 at 6:43 AM, Ben Kirwin <ben@kirw.in> wrote:
> > > >
> > > > > Gwen: Right now I'm just looking for feedback -- but yes, if folks
> > are
> > > > > interested, I do plan to do that implementation work.
> > > > >
> > > > > Daniel: Yes, that's exactly right. I haven't thought much about
> > > > > per-key... it does sound useful, but the implementation seems a bit
> > > > > more involved. Want to add it to the ticket?
> > > > >
> > > > > On Fri, Jun 12, 2015 at 7:49 AM, Daniel Schierbeck
> > > > > <daniel.schierbeck@gmail.com> wrote:
> > > > > > Ben: your solutions seems to focus on partition-wide CAS. Have
> you
> > > > > > considered per-key CAS? That would make the feature more useful
> in
> > my
> > > > > > opinion, as you'd greatly reduce the contention.
> > > > > >
> > > > > > On Fri, Jun 12, 2015 at 6:54 AM Gwen Shapira <
> > gshapira@cloudera.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Ben,
> > > > > >>
> > > > > >> Thanks for creating the ticket. Having check-and-set capability
> > will
> > > > be
> > > > > >> sweet :)
> > > > > >> Are you planning to implement this yourself? Or is it just
an
> idea
> > > for
> > > > > >> the community?
> > > > > >>
> > > > > >> Gwen
> > > > > >>
> > > > > >> On Thu, Jun 11, 2015 at 8:01 PM, Ben Kirwin <ben@kirw.in>
> wrote:
> > > > > >> > As it happens, I submitted a ticket for this feature
a couple
> > days
> > > > > ago:
> > > > > >> >
> > > > > >> > https://issues.apache.org/jira/browse/KAFKA-2260
> > > > > >> >
> > > > > >> > Couldn't find any existing proposals for similar things,
but
> > it's
> > > > > >> > certainly possible they're out there...
> > > > > >> >
> > > > > >> > On the other hand, I think you can solve your particular
issue
> > by
> > > > > >> > reframing the problem: treating the messages as 'requests'
or
> > > > > >> > 'commands' instead of statements of fact. In your
> flight-booking
> > > > > >> > example, the log would correctly reflect that two different
> > people
> > > > > >> > tried to book the same flight; the stream consumer
would be
> > > > > >> > responsible for finalizing one booking, and notifying
the
> other
> > > > client
> > > > > >> > that their request had failed. (In-browser or by email.)
> > > > > >> >
> > > > > >> > On Wed, Jun 10, 2015 at 5:04 AM, Daniel Schierbeck
> > > > > >> > <daniel.schierbeck@gmail.com> wrote:
> > > > > >> >> I've been working on an application which uses
Event
> Sourcing,
> > > and
> > > > > I'd
> > > > > >> like
> > > > > >> >> to use Kafka as opposed to, say, a SQL database
to store
> > events.
> > > > This
> > > > > >> would
> > > > > >> >> allow me to easily integrate other systems by having
them
> read
> > > off
> > > > > the
> > > > > >> >> Kafka topics.
> > > > > >> >>
> > > > > >> >> I do have one concern, though: the consistency
of the data
> can
> > > only
> > > > > be
> > > > > >> >> guaranteed if a command handler has a complete
picture of all
> > > past
> > > > > >> events
> > > > > >> >> pertaining to some entity.
> > > > > >> >>
> > > > > >> >> As an example, consider an airline seat reservation
system.
> > Each
> > > > > >> >> reservation command issued by a user is rejected
if the seat
> > has
> > > > > already
> > > > > >> >> been taken. If the seat is available, a record
describing the
> > > event
> > > > > is
> > > > > >> >> appended to the log. This works great when there's
only one
> > > > producer,
> > > > > >> but
> > > > > >> >> in order to scale I may need multiple producer
processes.
> This
> > > > > >> introduces a
> > > > > >> >> race condition: two command handlers may simultaneously
> > receive a
> > > > > >> command
> > > > > >> >> to reserver the same seat. The event log indicates
that the
> > seat
> > > is
> > > > > >> >> available, so each handler will append a reservation
event –
> > thus
> > > > > >> >> double-booking that seat!
> > > > > >> >>
> > > > > >> >> I see three ways around that issue:
> > > > > >> >> 1. Don't use Kafka for this.
> > > > > >> >> 2. Force a singler producer for a given flight.
This will
> > impact
> > > > > >> >> availability and make routing more complex.
> > > > > >> >> 3. Have a way to do optimistic locking in Kafka.
> > > > > >> >>
> > > > > >> >> The latter idea would work either on a per-key
basis or
> > globally
> > > > for
> > > > > a
> > > > > >> >> partition: when appending to a partition, the producer
would
> > > > > indicate in
> > > > > >> >> its request that the request should be rejected
unless the
> > > current
> > > > > >> offset
> > > > > >> >> of the partition is equal to x. For the per-key
setup, Kafka
> > > > brokers
> > > > > >> would
> > > > > >> >> track the offset of the latest message for each
unique key,
> if
> > so
> > > > > >> >> configured. This would allow the request to specify
that it
> > > should
> > > > be
> > > > > >> >> rejected if the offset for key k is not equal to
x.
> > > > > >> >>
> > > > > >> >> This way, only one of the command handlers would
succeed in
> > > writing
> > > > > to
> > > > > >> >> Kafka, thus ensuring consistency.
> > > > > >> >>
> > > > > >> >> There are different levels of complexity associated
with
> > > > implementing
> > > > > >> this
> > > > > >> >> in Kafka depending on whether the feature would
work
> > > per-partition
> > > > or
> > > > > >> >> per-key:
> > > > > >> >> * For the per-partition optimistic locking, the
broker would
> > just
> > > > > need
> > > > > >> to
> > > > > >> >> keep track of the high water mark for each partition
and
> reject
> > > > > >> conditional
> > > > > >> >> requests when the offset doesn't match.
> > > > > >> >> * For per-key locking, the broker would need to
maintain an
> > > > in-memory
> > > > > >> table
> > > > > >> >> mapping keys to the offset of the last message
with that key.
> > > This
> > > > > >> should
> > > > > >> >> be fairly easy to maintain and recreate from the
log if
> > > necessary.
> > > > It
> > > > > >> could
> > > > > >> >> also be saved to disk as a snapshot from time to
time in
> order
> > to
> > > > cut
> > > > > >> down
> > > > > >> >> the time needed to recreate the table on restart.
There's a
> > small
> > > > > >> >> performance penalty associated with this, but it
could be
> > opt-in
> > > > for
> > > > > a
> > > > > >> >> topic.
> > > > > >> >>
> > > > > >> >> Am I the only one thinking about using Kafka like
this? Would
> > > this
> > > > > be a
> > > > > >> >> nice feature to have?
> > > > > >>
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>



-- 
Thanks,
Ewen

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message