kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Kafka/ZK Cluster Example
Date Thu, 12 Jan 2012 17:15:34 GMT
Felix,

We use RAID too. One potential problem with RAID is that if you replace a
broken disk, RAID goes into rebuild mode. This could significantly slow
down I/O and make a broker not fully functional for new requests. Adding
more mirrors doesn't alleviate this problem.

Jun

On Wed, Jan 11, 2012 at 3:50 PM, Felix GV <felix@mate1inc.com> wrote:

> We've been thinking about this stuff a lot recently, at work.
>
> We've had some HD failures in our Kafka cluster. I don't know all the
> details, but from what I heard, the HDs were mirrored in RAID but several
> of them failed in a close time interval and the array did not have time to
> fully rebuild itself, so we lost all of that data from the Kafka cluster.
> Thankfully, the data was being consumed in near real time, so we only
> really lost a small unconsumed window of data.
>
> Now, we're wondering what we could improve to prevent this scenario in the
> future. I investigated Kafka mirroring but since it relies on consuming
> data, the probability to lose the unconsumed window is still there. If we
> had consumers that were more batch oriented (like hadoop) rather than
> real-time, the benefits of a mirrored Kafka cluster would be greater, but
> for our use cases, where data is consumed near real-time, we would still
> lose as much data as before. Am I right?
>
> KAFKA-50, with sync replication would have solved our problem, but until
> that's done, what are our options?
>
> I came to the conclusion that simply adding more mirrored copies in our
> RAID arrays would be the most cost-effective way to give us both more
> availability and more redundancy. This doesn't deal with the scenario where
> a machine fails and becomes unavailable, in which case the data on it would
> be temporarily unavailable but not lost (although, again, there could be a
> small window of uncommited data). However, in terms of protection against
> data loss from HD failures, it seems like the best option for now, no?
>
> It doesn't feel right to just throw more hardware at problems hehe... but I
> guess sometimes it's the only choice :) ...
>
> Please tell me if that makes sense!
>
> --
> Felix
>
>
>
> On Wed, Jan 11, 2012 at 6:16 PM, Felix GV <felix@mate1inc.com> wrote:
>
> > As I understand it, you cannot use a mirrored Kafka cluster as a hot
> > fail-over.
> >
> > You could probably use it as a manual fail-over, but I don't know the
> > complexity involved in doing that.
> >
> > Also, if your source cluster fails while producers were putting data into
> > it, there will be an "unconsumed window" of data that is lost. This
> > corresponds to the data that the embedded consumer in the mirrored
> cluster
> > did not have time to consume from the source cluster.
> >
> > All in all, the mirrored cluster is akin to asynchronous replication,
> > without any hot fail-over capability. Thus, it provides data redundancy
> > (outside of the unconsumed window described above) but no extra
> > availability (unless you count manual interventions).
> >
> > KAFKA-50 <https://issues.apache.org/jira/browse/KAFKA-50>, on the other
> > hand, will provide both asynchronous AND synchronous replication
> (although
> > the latter will incur a latency penalty) and will be able to use the
> > replicas (data redundancy) as hot-fail overs.
> >
> > Depending on your personal definition of "highly reliable" (whether it
> > includes data redundancy and/or availability), I think that should
> probably
> > answer your question...?
> >
> > To all the Kafka experts: please correct me if the above explanations are
> > incorrect :) !
> >
> > --
> > Felix
> >
> >
> >
> >
> > On Wed, Jan 11, 2012 at 5:53 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> It's just that the mirroring logic depends on ZK to be available most of
> >> the time.
> >>
> >> Jun
> >>
> >> On Wed, Jan 11, 2012 at 2:35 PM, Christian Carollo <ccarollo@gmail.com
> >> >wrote:
> >>
> >> > I see.  But if I used that configuration and then did the mirroring
> you
> >> > suggested would that be enough, in your opinion, to be considered
> highly
> >> > reliable?
> >> >
> >> > Christian
> >> >
> >> >
> >> > On Jan 11, 2012, at 2:32 PM, Jun Rao wrote:
> >> >
> >> > >> For example, can I have one ZK instance and one broker on one
> machine
> >> > and
> >> > > that is enough to define a ZK cluster and a Kafka Cluster?
> >> > >
> >> > > Yes, although you don't get the reliability of ZK now.
> >> > >
> >> > > Jun
> >> > >
> >> > >
> >> > > On Wed, Jan 11, 2012 at 2:06 PM, Christian Carollo <
> >> ccarollo@gmail.com
> >> > >wrote:
> >> > >
> >> > >> Jun,
> >> > >>
> >> > >> I don't think I ask my question the right way.
> >> > >>
> >> > >> What I am trying to understand is what are the minimum constituent
> >> parts
> >> > >> of a kafka cluster?
> >> > >>
> >> > >> Based on your last email, I am now wondering what are the minimum
> >> > >> constituent parts of a ZK cluster as well as a Kafka cluster?
> >> > >>
> >> > >> For example, can I have one ZK instance and one broker on one
> machine
> >> > and
> >> > >> that is enough to define a ZK cluster and a Kafka Cluster?
> >> > >>
> >> > >> Thanks,
> >> > >> Christian
> >> > >>
> >> > >>
> >> > >> On Jan 11, 2012, at 1:50 PM, Jun Rao <junrao@gmail.com>
wrote:
> >> > >>
> >> > >>> Chrsitan,
> >> > >>>
> >> > >>> A Kafka cluster containers a ZK cluster and a list of brokers.
> When
> >> a
> >> > >>> consumer subscribes to a topic in a kafka cluster, it consumes
> data
> >> > >> stored
> >> > >>> in all brokers in that cluster.
> >> > >>>
> >> > >>> Thanks,
> >> > >>>
> >> > >>> Jun
> >> > >>>
> >> > >>> On Tue, Jan 10, 2012 at 11:28 PM, Christian Carollo <
> >> > ccarollo@gmail.com
> >> > >>> wrote:
> >> > >>>
> >> > >>>> Thank you Jun that is quite helpful.  I have a question
about
> Kafka
> >> > >>>> Clusters.  What are the minimum number and types of services
that
> >> must
> >> > >> be
> >> > >>>> running to make up a Kafka Cluster?
> >> > >>>>
> >> > >>>> I ask this because the diagrams (in the Kafka Mirroring
document)
> >> > allude
> >> > >>>> to a multiple broker environment, however, since each
broker does
> >> not
> >> > >>>> appear to provide redundancy (as of today) to any of the
other
> >> brokers
> >> > >> in a
> >> > >>>> given zookeeper service, it seems like a Kafka Cluster
is nothing
> >> more
> >> > >> than
> >> > >>>> a grouping of a single zookeeper instance with a single
Kafka
> >> broker,
> >> > is
> >> > >>>> this the correct understanding?
> >> > >>>>
> >> > >>>> Thanks,
> >> > >>>> Christian
> >> > >>>>
> >> > >>>> On Jan 10, 2012, at 8:47 AM, Jun Rao wrote:
> >> > >>>>
> >> > >>>>> With 0.7, you can set up inter-cluster replication
(
> >> > >>>>>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring
> >> ).
> >> > >>>>>
> >> > >>>>> For the future 0.8 release, we are working on intra-cluster
> >> > replication
> >> > >>>>> support and details can be found at
> >> > >>>>> https://issues.apache.org/jira/browse/KAFKA-50
> >> > >>>>>
> >> > >>>>> Thanks,
> >> > >>>>>
> >> > >>>>> Jun
> >> > >>>>>
> >> > >>>>> On Mon, Jan 9, 2012 at 9:52 PM, Christian Carollo
<
> >> > ccarollo@gmail.com
> >> > >>>>> wrote:
> >> > >>>>>
> >> > >>>>>> I am looking to implement Kafka in a production
environment,
> >> > however,
> >> > >> I
> >> > >>>>>> haven't found in documentation or examples that
> >> > >>>>>> discuss how to build a redundant implementation.
 Is there any
> >> > >>>>>> documentation out their (blogs, articles, etc.)
that describes
> >> > >>>>>> how we can implement such a system with Kafka
0.6 or 0.7.
> >> > >>>>>>
> >> > >>>>>> Also, is there a timeframe the community is shooting
for, to
> >> release
> >> > >>>> 0.8 w/
> >> > >>>>>> replication?
> >> > >>>>>>
> >> > >>>>>> Thanks
> >> > >>>>>> Christian
> >> > >>>>>>
> >> > >>>>
> >> > >>>>
> >> > >>
> >> >
> >> >
> >>
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message