kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gwen Shapira <g...@confluent.io>
Subject Re: at-least-once delivery
Date Tue, 02 Feb 2016 18:30:27 GMT
MAX_INT is a good value if you want to just block until the buffer has some
space (and never get an exception).

On Tue, Feb 2, 2016 at 8:08 AM, Franco Giacosa <fgiacosa@gmail.com> wrote:

> Thanks for the information James, the slides are really good.
>
> One question, in the new producer the property block.on.buffer.full (in the
> slides put this value in TRUE is a good practice, I image that this will
> avoid a buffer overflow) is deprecated, and instead the use of
> max.block.ms,
> which block for X amount of ms, can someone tell me what a good value will
> be for this property in order to mimic the behaviour of
> block.on.buffer.full?
>
> Thanks
>
> 2016-01-31 6:09 GMT+01:00 James Cheng <jcheng@tivo.com>:
>
> >
> > > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fgiacosa@gmail.com>
> wrote:
> > >
> > > Sorry, this solved my questions: "Setting a value greater than zero
> will
> > > cause the client to resend any record whose send fails with a
> potentially
> > > transient error. Note that this retry is no different than if the
> client
> > > resent the record upon receiving the error. Allowing retries will
> > > potentially change the ordering of records because if two records are
> > sent
> > > to a single partition, and the first fails and is retried but the
> second
> > > succeeds, then the second record may appear first."
> > >
> >
> > Franco,
> >
> > Also, you can avoid the message reordering issue in that description by
> > setting max.in.flight.requests.per.connector to 1.
> >
> > This slide deck has good guidelines on the types of things you are
> talking
> > about:
> >
> >
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
> >
> > -James
> >
> > > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fgiacosa@gmail.com>:
> > >
> > >> Hi,
> > >>
> > >> The at-least-once delivery is generated in part by the network fails
> and
> > >> the retries (that may generate duplicates) right?
> > >>
> > >> In the event of a duplicated (there was an error but the first message
> > >> landed ok on the partition P1) the producer will recalculate the
> > partition
> > >> on the retry? is this done automatically?
> > >>
> > >> If in the retry the partition doesn't change and there is only 1
> > Producer,
> > >> will the duplicated be written next to the original? I mean if I
> poll()
> > >> they will come one after the other?
> > >>
> > >>
> > >>
> >
> >
> > ________________________________
> >
> > This email and any attachments may contain confidential and privileged
> > material for the sole use of the intended recipient. Any review, copying,
> > or distribution of this email (or any attachments) by others is
> prohibited.
> > If you are not the intended recipient, please contact the sender
> > immediately and permanently delete this email and any attachments. No
> > employee or agent of TiVo Inc. is authorized to conclude any binding
> > agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> > Inc. may only be made by a signed written agreement.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message