kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <crypt...@gmail.com>
Subject Re: Replication Follow-up
Date Thu, 29 Dec 2011 15:35:39 GMT
I started using MapR last week in pilot mode and I have been considering
using it's HA NFS as the underlying file system for my Kafka brokers (on a
10Gb network) until replication is live.  For my use case it is all about
data reliability and high availability so having kafka guarantee this if
paramount.

Your solution sounds interesting for another use case we have where we (in
the consumer) will get an error (for lots of reasons) and then have to
reprocess what we consumed... right now we don't do this very well and
something I have on the to-be-do list and what you have might be useful for
that use case (except i would not send it to another kafka server just back
into the same one probably on another topic but not always, I would be
using zookeeper too) so maybe we can collaborate on the code
for separate use cases?


On Thu, Dec 29, 2011 at 10:24 AM, Matt Abrams <abramsm@clearspring.com>wrote:

> Joe -
>
> I am interested in replication as well.  We are going live with our
> implementation in the next few days.
>
> In our setup events are fed into one of two datacenters (depending on
> which is closest to the client).   We wanted to be able to process all
> of the data in a single DC regardless of which DC the event originally
> arrived in.  So we wrote small project to make this possible.
>
> The project uses a fairly simple process that consumes data from one
> Kafka broker and then repeats the data to another Kafka broker (in our
> case the 'other' broker is in the remote data center).  ZooKeeper is
> used to identify peers and detect failures, new nodes, etc.  This
> isn't really replication but can be used repeat all data from one
> broker onto another and works for our use case.  If there is general
> interest in this project we can work on open sourcing it.
>
> Matt
>
> On Thu, Dec 29, 2011 at 10:09 AM, Joe Stein <cryptcom@gmail.com> wrote:
> > Hello, hope everyone's holiday break was/is going well? (If you are lucky
> > enough to have one).
> >
> > My busy season (mobile advertising) will start to ease up next week and I
> > will get to have some down time.
> >
> > We use Kafka in production now and are looking to up it's usage but
> really
> > need/want replication and outside of work it is an interesting problem to
> > work on.
> >
> > So, I wanted to follow-up on the replication work and what I can do to
> > help?  I was not sure where this had left off and if the work had already
> > started or what.
> >
> > If not, is it good to have maybe have a conference call or something to
> go
> > through it and divy things up?  If it has started let me know where I can
> > jump in please.
> >
> > Thanks!!!
> >
> > /*
> > Joe Stein
> > http://www.linkedin.com/in/charmalloc
> > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> > */
>



-- 

/*
Joe Stein
http://www.linkedin.com/in/charmalloc
Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
*/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message