kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stevo Slavić <ssla...@gmail.com>
Subject Re: Kafka settings for (more) reliable/durable messaging
Date Tue, 07 Jul 2015 15:01:54 GMT
Hello Gwen,

Thanks for fast response!

Btw, congrats on officially becoming a Kafka committer and thanks, among
other things, for great "Intro to Kafka" video
http://shop.oreilly.com/product/0636920038603.do !

Have to read more docs and/or source. I thought this scenario is possible
because replica can fall behind (replica.lag.max.messages) and still be
considered ISR. Then I assumed also write can be ACKed by any ISR, and then
why not by one which has fallen more behind.

Kind regards,
Stevo Slavic.

On Tue, Jul 7, 2015 at 4:47 PM, Gwen Shapira <gshapira@cloudera.com> wrote:

> I am not sure "different replica" can ACK the second back of messages
> while not having the first - from what I can see, it will need to be
> up-to-date on the latest messages (i.e. correct HWM) in order to ACK.
>
> On Tue, Jul 7, 2015 at 7:13 AM, Stevo Slavić <sslavic@gmail.com> wrote:
> > Hello Apache Kafka community,
> >
> > Documentation for min.insync.replicas in
> > http://kafka.apache.org/documentation.html#brokerconfigs states:
> >
> > "When used together, min.insync.replicas and request.required.acks allow
> > you to enforce greater durability guarantees. A typical scenario would be
> > to create a topic with a replication factor of 3, set min.insync.replicas
> > to 2, and produce with request.required.acks of -1. This will ensure that
> > the producer raises an exception if a majority of replicas do not
> receive a
> > write."
> >
> > Correct me if wrong (doc reference?), I assume min.insync.replicas
> includes
> > lead, so with min.insync.replicas=2, lead and one more replica besides
> lead
> > will have to ACK writes.
> >
> > In such setup, with minimalistic 3 brokers cluster, given that
> > - all 3 replicas are insync
> > - a batch of messages is written and ends up on lead and one replica ACKs
> > - another batch of messages ends up on lead and different replica ACKs
> >
> > Is it possible that when lead crashes, while replicas didn't catch up,
> > (part of) one batch of messages could be lost (since one replica becomes
> a
> > new lead, and it's only serving all reads and requests, and replication
> is
> > one way)?
> >
> > Kind regards,
> > Stevo Slavic.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message