kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Franco Giacosa <fgiac...@gmail.com>
Subject Re: at-least-once delivery
Date Tue, 02 Feb 2016 16:08:17 GMT
Thanks for the information James, the slides are really good.

One question, in the new producer the property block.on.buffer.full (in the
slides put this value in TRUE is a good practice, I image that this will
avoid a buffer overflow) is deprecated, and instead the use of max.block.ms,
which block for X amount of ms, can someone tell me what a good value will
be for this property in order to mimic the behaviour of
block.on.buffer.full?

Thanks

2016-01-31 6:09 GMT+01:00 James Cheng <jcheng@tivo.com>:

>
> > On Jan 30, 2016, at 4:21 AM, Franco Giacosa <fgiacosa@gmail.com> wrote:
> >
> > Sorry, this solved my questions: "Setting a value greater than zero will
> > cause the client to resend any record whose send fails with a potentially
> > transient error. Note that this retry is no different than if the client
> > resent the record upon receiving the error. Allowing retries will
> > potentially change the ordering of records because if two records are
> sent
> > to a single partition, and the first fails and is retried but the second
> > succeeds, then the second record may appear first."
> >
>
> Franco,
>
> Also, you can avoid the message reordering issue in that description by
> setting max.in.flight.requests.per.connector to 1.
>
> This slide deck has good guidelines on the types of things you are talking
> about:
>
> http://www.slideshare.net/JiangjieQin/no-data-loss-pipeline-with-apache-kafka-49753844
>
> -James
>
> > 2016-01-30 13:18 GMT+01:00 Franco Giacosa <fgiacosa@gmail.com>:
> >
> >> Hi,
> >>
> >> The at-least-once delivery is generated in part by the network fails and
> >> the retries (that may generate duplicates) right?
> >>
> >> In the event of a duplicated (there was an error but the first message
> >> landed ok on the partition P1) the producer will recalculate the
> partition
> >> on the retry? is this done automatically?
> >>
> >> If in the retry the partition doesn't change and there is only 1
> Producer,
> >> will the duplicated be written next to the original? I mean if I poll()
> >> they will come one after the other?
> >>
> >>
> >>
>
>
> ________________________________
>
> This email and any attachments may contain confidential and privileged
> material for the sole use of the intended recipient. Any review, copying,
> or distribution of this email (or any attachments) by others is prohibited.
> If you are not the intended recipient, please contact the sender
> immediately and permanently delete this email and any attachments. No
> employee or agent of TiVo Inc. is authorized to conclude any binding
> agreement on behalf of TiVo Inc. by email. Binding agreements with TiVo
> Inc. may only be made by a signed written agreement.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message