kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Christian Carollo <ccaro...@gmail.com>
Subject Re: Kafka/ZK Cluster Example
Date Wed, 11 Jan 2012 23:35:07 GMT
This leads to two more questions…

1) Maybe I am not understanding what a ZK cluster typically looks like or is made up of. 
If I have more than one ZK service/instance running on a single node that doesn't sound like
it is more reliable when there is a server failure.

On the other hand, if I have one ZK on one node and another on another node, even as a hot
standby via mirroring, that seems like a more reliable solution.  I think I must be missing
something, am I?

2) Can the client producer interrogate the ZK service and determine if it is available and/or
if one or more brokers are available?  And if so get there connection information from ZK
so that the producer can intelligently send messages to the right brokers?  If this is possible
the client producer could handle failure cases and either contact a different (hot-standby)
ZK or Broker?

Thanks
Christian

On Jan 11, 2012, at 3:16 PM, Felix GV wrote:

> As I understand it, you cannot use a mirrored Kafka cluster as a hot
> fail-over.
> 
> You could probably use it as a manual fail-over, but I don't know the
> complexity involved in doing that.
> 
> Also, if your source cluster fails while producers were putting data into
> it, there will be an "unconsumed window" of data that is lost. This
> corresponds to the data that the embedded consumer in the mirrored cluster
> did not have time to consume from the source cluster.
> 
> All in all, the mirrored cluster is akin to asynchronous replication,
> without any hot fail-over capability. Thus, it provides data redundancy
> (outside of the unconsumed window described above) but no extra
> availability (unless you count manual interventions).
> 
> KAFKA-50 <https://issues.apache.org/jira/browse/KAFKA-50>, on the other
> hand, will provide both asynchronous AND synchronous replication (although
> the latter will incur a latency penalty) and will be able to use the
> replicas (data redundancy) as hot-fail overs.
> 
> Depending on your personal definition of "highly reliable" (whether it
> includes data redundancy and/or availability), I think that should probably
> answer your question...?
> 
> To all the Kafka experts: please correct me if the above explanations are
> incorrect :) !
> 
> --
> Felix
> 
> 
> 
> On Wed, Jan 11, 2012 at 5:53 PM, Jun Rao <junrao@gmail.com> wrote:
> 
>> It's just that the mirroring logic depends on ZK to be available most of
>> the time.
>> 
>> Jun
>> 
>> On Wed, Jan 11, 2012 at 2:35 PM, Christian Carollo <ccarollo@gmail.com
>>> wrote:
>> 
>>> I see.  But if I used that configuration and then did the mirroring you
>>> suggested would that be enough, in your opinion, to be considered highly
>>> reliable?
>>> 
>>> Christian
>>> 
>>> 
>>> On Jan 11, 2012, at 2:32 PM, Jun Rao wrote:
>>> 
>>>>> For example, can I have one ZK instance and one broker on one machine
>>> and
>>>> that is enough to define a ZK cluster and a Kafka Cluster?
>>>> 
>>>> Yes, although you don't get the reliability of ZK now.
>>>> 
>>>> Jun
>>>> 
>>>> 
>>>> On Wed, Jan 11, 2012 at 2:06 PM, Christian Carollo <ccarollo@gmail.com
>>>> wrote:
>>>> 
>>>>> Jun,
>>>>> 
>>>>> I don't think I ask my question the right way.
>>>>> 
>>>>> What I am trying to understand is what are the minimum constituent
>> parts
>>>>> of a kafka cluster?
>>>>> 
>>>>> Based on your last email, I am now wondering what are the minimum
>>>>> constituent parts of a ZK cluster as well as a Kafka cluster?
>>>>> 
>>>>> For example, can I have one ZK instance and one broker on one machine
>>> and
>>>>> that is enough to define a ZK cluster and a Kafka Cluster?
>>>>> 
>>>>> Thanks,
>>>>> Christian
>>>>> 
>>>>> 
>>>>> On Jan 11, 2012, at 1:50 PM, Jun Rao <junrao@gmail.com> wrote:
>>>>> 
>>>>>> Chrsitan,
>>>>>> 
>>>>>> A Kafka cluster containers a ZK cluster and a list of brokers. When
a
>>>>>> consumer subscribes to a topic in a kafka cluster, it consumes data
>>>>> stored
>>>>>> in all brokers in that cluster.
>>>>>> 
>>>>>> Thanks,
>>>>>> 
>>>>>> Jun
>>>>>> 
>>>>>> On Tue, Jan 10, 2012 at 11:28 PM, Christian Carollo <
>>> ccarollo@gmail.com
>>>>>> wrote:
>>>>>> 
>>>>>>> Thank you Jun that is quite helpful.  I have a question about
Kafka
>>>>>>> Clusters.  What are the minimum number and types of services
that
>> must
>>>>> be
>>>>>>> running to make up a Kafka Cluster?
>>>>>>> 
>>>>>>> I ask this because the diagrams (in the Kafka Mirroring document)
>>> allude
>>>>>>> to a multiple broker environment, however, since each broker
does
>> not
>>>>>>> appear to provide redundancy (as of today) to any of the other
>> brokers
>>>>> in a
>>>>>>> given zookeeper service, it seems like a Kafka Cluster is nothing
>> more
>>>>> than
>>>>>>> a grouping of a single zookeeper instance with a single Kafka
>> broker,
>>> is
>>>>>>> this the correct understanding?
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Christian
>>>>>>> 
>>>>>>> On Jan 10, 2012, at 8:47 AM, Jun Rao wrote:
>>>>>>> 
>>>>>>>> With 0.7, you can set up inter-cluster replication (
>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring
>> ).
>>>>>>>> 
>>>>>>>> For the future 0.8 release, we are working on intra-cluster
>>> replication
>>>>>>>> support and details can be found at
>>>>>>>> https://issues.apache.org/jira/browse/KAFKA-50
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> 
>>>>>>>> Jun
>>>>>>>> 
>>>>>>>> On Mon, Jan 9, 2012 at 9:52 PM, Christian Carollo <
>>> ccarollo@gmail.com
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> I am looking to implement Kafka in a production environment,
>>> however,
>>>>> I
>>>>>>>>> haven't found in documentation or examples that
>>>>>>>>> discuss how to build a redundant implementation.  Is
there any
>>>>>>>>> documentation out their (blogs, articles, etc.) that
describes
>>>>>>>>> how we can implement such a system with Kafka 0.6 or
0.7.
>>>>>>>>> 
>>>>>>>>> Also, is there a timeframe the community is shooting
for, to
>> release
>>>>>>> 0.8 w/
>>>>>>>>> replication?
>>>>>>>>> 
>>>>>>>>> Thanks
>>>>>>>>> Christian
>>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>> 
>>> 
>>> 
>> 


Mime
View raw message