kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: Kafka/ZK Cluster Example
Date Thu, 12 Jan 2012 17:10:19 GMT
For 1), roughly speaking, hosts in a ZK cluster replicate among themselves
synchronously. So having multiple ZK hosts improves reliability. ZK
tolerates k failures with 2k+1 hosts.

For 2), that's exactly how our ZK-based producer works.

Thanks,

Jun

On Wed, Jan 11, 2012 at 3:35 PM, Christian Carollo <ccarollo@gmail.com>wrote:

> This leads to two more questions…
>
> 1) Maybe I am not understanding what a ZK cluster typically looks like or
> is made up of.  If I have more than one ZK service/instance running on a
> single node that doesn't sound like it is more reliable when there is a
> server failure.
>
> On the other hand, if I have one ZK on one node and another on another
> node, even as a hot standby via mirroring, that seems like a more reliable
> solution.  I think I must be missing something, am I?
>
> 2) Can the client producer interrogate the ZK service and determine if it
> is available and/or if one or more brokers are available?  And if so get
> there connection information from ZK so that the producer can intelligently
> send messages to the right brokers?  If this is possible the client
> producer could handle failure cases and either contact a different
> (hot-standby) ZK or Broker?
>
> Thanks
> Christian
>
> On Jan 11, 2012, at 3:16 PM, Felix GV wrote:
>
> > As I understand it, you cannot use a mirrored Kafka cluster as a hot
> > fail-over.
> >
> > You could probably use it as a manual fail-over, but I don't know the
> > complexity involved in doing that.
> >
> > Also, if your source cluster fails while producers were putting data into
> > it, there will be an "unconsumed window" of data that is lost. This
> > corresponds to the data that the embedded consumer in the mirrored
> cluster
> > did not have time to consume from the source cluster.
> >
> > All in all, the mirrored cluster is akin to asynchronous replication,
> > without any hot fail-over capability. Thus, it provides data redundancy
> > (outside of the unconsumed window described above) but no extra
> > availability (unless you count manual interventions).
> >
> > KAFKA-50 <https://issues.apache.org/jira/browse/KAFKA-50>, on the other
> > hand, will provide both asynchronous AND synchronous replication
> (although
> > the latter will incur a latency penalty) and will be able to use the
> > replicas (data redundancy) as hot-fail overs.
> >
> > Depending on your personal definition of "highly reliable" (whether it
> > includes data redundancy and/or availability), I think that should
> probably
> > answer your question...?
> >
> > To all the Kafka experts: please correct me if the above explanations are
> > incorrect :) !
> >
> > --
> > Felix
> >
> >
> >
> > On Wed, Jan 11, 2012 at 5:53 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> It's just that the mirroring logic depends on ZK to be available most of
> >> the time.
> >>
> >> Jun
> >>
> >> On Wed, Jan 11, 2012 at 2:35 PM, Christian Carollo <ccarollo@gmail.com
> >>> wrote:
> >>
> >>> I see.  But if I used that configuration and then did the mirroring you
> >>> suggested would that be enough, in your opinion, to be considered
> highly
> >>> reliable?
> >>>
> >>> Christian
> >>>
> >>>
> >>> On Jan 11, 2012, at 2:32 PM, Jun Rao wrote:
> >>>
> >>>>> For example, can I have one ZK instance and one broker on one machine
> >>> and
> >>>> that is enough to define a ZK cluster and a Kafka Cluster?
> >>>>
> >>>> Yes, although you don't get the reliability of ZK now.
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>> On Wed, Jan 11, 2012 at 2:06 PM, Christian Carollo <
> ccarollo@gmail.com
> >>>> wrote:
> >>>>
> >>>>> Jun,
> >>>>>
> >>>>> I don't think I ask my question the right way.
> >>>>>
> >>>>> What I am trying to understand is what are the minimum constituent
> >> parts
> >>>>> of a kafka cluster?
> >>>>>
> >>>>> Based on your last email, I am now wondering what are the minimum
> >>>>> constituent parts of a ZK cluster as well as a Kafka cluster?
> >>>>>
> >>>>> For example, can I have one ZK instance and one broker on one machine
> >>> and
> >>>>> that is enough to define a ZK cluster and a Kafka Cluster?
> >>>>>
> >>>>> Thanks,
> >>>>> Christian
> >>>>>
> >>>>>
> >>>>> On Jan 11, 2012, at 1:50 PM, Jun Rao <junrao@gmail.com> wrote:
> >>>>>
> >>>>>> Chrsitan,
> >>>>>>
> >>>>>> A Kafka cluster containers a ZK cluster and a list of brokers.
When
> a
> >>>>>> consumer subscribes to a topic in a kafka cluster, it consumes
data
> >>>>> stored
> >>>>>> in all brokers in that cluster.
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Jun
> >>>>>>
> >>>>>> On Tue, Jan 10, 2012 at 11:28 PM, Christian Carollo <
> >>> ccarollo@gmail.com
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thank you Jun that is quite helpful.  I have a question
about Kafka
> >>>>>>> Clusters.  What are the minimum number and types of services
that
> >> must
> >>>>> be
> >>>>>>> running to make up a Kafka Cluster?
> >>>>>>>
> >>>>>>> I ask this because the diagrams (in the Kafka Mirroring
document)
> >>> allude
> >>>>>>> to a multiple broker environment, however, since each broker
does
> >> not
> >>>>>>> appear to provide redundancy (as of today) to any of the
other
> >> brokers
> >>>>> in a
> >>>>>>> given zookeeper service, it seems like a Kafka Cluster is
nothing
> >> more
> >>>>> than
> >>>>>>> a grouping of a single zookeeper instance with a single
Kafka
> >> broker,
> >>> is
> >>>>>>> this the correct understanding?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Christian
> >>>>>>>
> >>>>>>> On Jan 10, 2012, at 8:47 AM, Jun Rao wrote:
> >>>>>>>
> >>>>>>>> With 0.7, you can set up inter-cluster replication (
> >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring
> >> ).
> >>>>>>>>
> >>>>>>>> For the future 0.8 release, we are working on intra-cluster
> >>> replication
> >>>>>>>> support and details can be found at
> >>>>>>>> https://issues.apache.org/jira/browse/KAFKA-50
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> Jun
> >>>>>>>>
> >>>>>>>> On Mon, Jan 9, 2012 at 9:52 PM, Christian Carollo <
> >>> ccarollo@gmail.com
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I am looking to implement Kafka in a production
environment,
> >>> however,
> >>>>> I
> >>>>>>>>> haven't found in documentation or examples that
> >>>>>>>>> discuss how to build a redundant implementation.
 Is there any
> >>>>>>>>> documentation out their (blogs, articles, etc.)
that describes
> >>>>>>>>> how we can implement such a system with Kafka 0.6
or 0.7.
> >>>>>>>>>
> >>>>>>>>> Also, is there a timeframe the community is shooting
for, to
> >> release
> >>>>>>> 0.8 w/
> >>>>>>>>> replication?
> >>>>>>>>>
> >>>>>>>>> Thanks
> >>>>>>>>> Christian
> >>>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message