kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Guarantees
Date Sat, 21 Jul 2012 00:53:58 GMT
Once the network has recovered, you should see new messages.

Thanks,
Neha

On Fri, Jul 20, 2012 at 4:49 PM, graham sanderson <graham@vast.com> wrote:

> Thanks Neha, I really meant, yes I may lose some messages in the
> meanwhile, but should I expect new messages after everything gets back to
> normal to be delivered (unless my code throws an exception and kills a
> worker thread, which it wasn't)
>
> On Jul 20, 2012, at 6:00 PM, Neha Narkhede wrote:
>
> > Graham,
> >
> > It really depends on what sort of network outage. The producer, whether
> zk
> > or not, can be configured to retry couple of times. If it runs out of
> > retries during this outage, it will drop the messages and they will be
> > lost.
> >
> > Thanks,
> > Neha
> >
> > On Sat, Jul 14, 2012 at 11:32 AM, Jun Rao <junrao@gmail.com> wrote:
> >
> >> The pipeline is supposed to recover from the network outage. There
> could be
> >> bugs, especially in the ZK-based producer since it's relatively new.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Thu, Jul 12, 2012 at 7:08 PM, graham sanderson <graham@vast.com>
> wrote:
> >>
> >>> Hi, so I happened to be going to demo a prototype built with kafka in a
> >>> borrowed large room which I discovered had insufficient/flaky wireless.
> >> Was
> >>> using zookeeper config, and getting lots of timeouts etc. Since this
> was
> >>> the first time I had used kafka and I hadn't done any off path testing,
> >> my
> >>> first course of action was to find a hard wire, which I did and all the
> >>> timeouts disappeared. The demo was great. Note that even with the flaky
> >>> wireless, messages generally still seemed to be getting delivered, but
> >> not
> >>> always as far as I could tell (or perhaps with high latency - was more
> >>> focused on having a working demo than debugging)
> >>>
> >>> I'm using 0.7 atm, though I'm not sure if that matters.
> >>>
> >>> My somewhat question is, given a simple scenario using kafka/zookeeper
> >>> (prior to all the exciting fault tolerance work going on right now):
> >>>
> >>> 1) Lets say I have zookeeper server, kafka server, producer, and
> consumer
> >>> running on a perfect network. And I successfully send a message from
> >>> producer to consumer
> >>> 2) All JVMs stay up, however I lose network connectivity between some
> or
> >>> all of them for some time
> >>> 3) The network becomes perfect again.
> >>> 4) I wait for some time for everyone to reconnect/re-negociate to their
> >>> best ability
> >>>
> >>> Following that, should I expect a new message from the producer to
> reach
> >>> the consumer, or can the system get into a broken state?… I swear I saw
> >>> such a message not delivered, but I can't say for sure… I can certainly
> >>> investigate further by trying to reproduce again and wading thru the
> many
> >>> logged errors, but if someone already knows the answer that'd be
> awesome!
> >>>
> >>> Thanks,
> >>>
> >>> Graham.
> >>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message