kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Kreps <jay.kr...@gmail.com>
Subject Re: Kafka mirroring and zookeeper
Date Wed, 25 Apr 2012 22:32:37 GMT
Short answer is yes, both async (acks=0 or 1) and sync replication
(acks > 1) will be both be supported.

-Jay

On Wed, Apr 25, 2012 at 11:22 AM, Jun Rao <junrao@gmail.com> wrote:
> Felix,
>
> Initially, we thought we could keep the option of not sending acks from the
> broker to the producer. However, this seems hard since in the new wire
> protocol, we need to send at least the error code to the producer (e.g., a
> request is sent to the wrong broker or wrong partition).
>
> So, what we allow in the current design is the following. The producer can
> specify the # of acks in the request. By default (acks = -1), the broker
> will wait for the message to be written to all replicas that are still
> synced up with the leader before acking the producer. Otherwise (acks >=0),
> the broker will ack the producer after the message is written to acks
> replicas. Currently, acks=0 is treated the same as acks=1.
>
> Thanks,
>
> Jun
>
> On Wed, Apr 25, 2012 at 10:39 AM, Felix GV <felix@mate1inc.com> wrote:
>
>> Just curious, but if I remember correctly from the time I read KAFKA-50 and
>> the related JIRA issues, you guys plan to implement sync AND async
>> replication, right?
>>
>> --
>> Felix
>>
>>
>>
>> On Tue, Apr 24, 2012 at 4:42 PM, Jay Kreps <jay.kreps@gmail.com> wrote:
>>
>> > Right now we do sloppy failover. That is when a broker goes down
>> > traffic is redirected to the remaining machines, but any unconsumed
>> > messages are stuck on that server until it comes back, if it is
>> > permanently gone the messages are lost. This is acceptable for us in
>> > the near-term since our pipeline is pretty real-time so this window
>> > between production and consumption is pretty small. The complete
>> > solution is the intra-cluster replication in KAFA-50 which is coming
>> > along fairly nicely now that we are working on it.
>> >
>> > -Jay
>> >
>> > On Tue, Apr 24, 2012 at 12:21 PM, Oliver Krohne
>> > <oliver.krohne@googlemail.com> wrote:
>> > > Hi,
>> > >
>> > > indeed I thought could be used as failover approach.
>> > >
>> > > We use raid for local redundancy but it does not protect us in case of
>> a
>> > machine failure, so I am looking for a way to achieve a master/slave
>> setup
>> > until KAFKA-50 has been implemented.
>> > >
>> > > I think we can solve it for now by having multiple broker so that the
>> > application can continue sending messages if one broker goes down. My
>> main
>> > concern is to not introduce a new single point of failure which can stop
>> > the application. However as some consumer are not developed by us and it
>> is
>> > not clear how they store the offset in zookeeper we need to find out how
>> we
>> > can manage the consumer in case a broker will never return after a
>> failure.
>> > It will be acceptable to lose a couple of messages if a broker dies and
>> the
>> > consumers have not consumed all messages at the point of failure.
>> > >
>> > > Thanks,
>> > > Oliver
>> > >
>> > >
>> > >
>> > >
>> > > Am 23.04.2012 um 19:58 schrieb Jay Kreps:
>> > >
>> > >> I think the confusion comes from the fact that we are using mirroring
>> to
>> > >> handle geographic distribution not failover. If I understand correctly
>> > what
>> > >> Oliver is asking for is something to give fault tolerance not
>> something
>> > for
>> > >> distribution. I don't think that is really what the mirroring does
out
>> > of
>> > >> the box, though technically i suppose you could just reset the offsets
>> > and
>> > >> point the consumer at the new cluster and have it start from "now".
>> > >>
>> > >> I think it would be helpful to document our use case in the mirroring
>> > docs
>> > >> since this is not the first time someone has asked about this.
>> > >>
>> > >> -Jay
>> > >>
>> > >> On Mon, Apr 23, 2012 at 10:38 AM, Joel Koshy <jjkoshy.w@gmail.com>
>> > wrote:
>> > >>
>> > >>> Hi Oliver,
>> > >>>
>> > >>> I was reading the mirroring guide and I wonder if it is required
that
>> > the
>> > >>>> mirror runs it's own zookeeper?
>> > >>>>
>> > >>>> We have a zookeeper cluster running which is used by different
>> > >>>> applications, so can we use that zookeeper cluster for the
kafka
>> > source
>> > >>> and
>> > >>>> kafka mirror?
>> > >>>>
>> > >>>
>> > >>> You could have a single zookeeper cluster and use different
>> namespaces
>> > for
>> > >>> the source/target mirror. However, I don't think it is recommended
to
>> > use a
>> > >>> remote zookeeper (if you have a cross-DC set up) since that would
>> > >>> potentially mean very high ZK latencies on one of your clusters.
>> > >>>
>> > >>>
>> > >>>> What is the procedure if the kafka source server fails to switch
the
>> > >>>> applications to use the mirrored instance?
>> > >>>>
>> > >>>
>> > >>> I don't quite follow this question - can you clarify? The mirror
>> > cluster is
>> > >>> pretty much a separate instance. There is no built-in automatic
>> > fail-over
>> > >>> if your source cluster goes down.
>> > >>>
>> > >>>
>> > >>>> Are there any backup best practices if we would not use mirroring?
>> > >>>>
>> > >>>
>> > >>> You can use RAID arrays for (local) data redundancy. You may also
be
>> > >>> interested in the (intra-DC) replication feature (KAFKA-50) that
is
>> > >>> currently being developed. I believe some folks on this list have
>> also
>> > used
>> > >>> plain rsync's as an alternative to mirroring.
>> > >>>
>> > >>> Thanks,
>> > >>>
>> > >>> Joel
>> > >>>
>> > >
>> >
>>

Mime
View raw message