kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomoyuki Saito <aocch...@gmail.com>
Subject Re: KafkaProducer - Oversized batches
Date Wed, 18 Dec 2019 13:46:42 GMT
Hi,

I'll close KAFKA-9301 because it is duplicated by KAFKA-9312.
https://issues.apache.org/jira/browse/KAFKA-9312

Tomoyuki

On Mon, Dec 16, 2019 at 12:33 PM Tomoyuki Saito <aocchoda@gmail.com> wrote:
>
> Hi,
>
> Filed up a JIRA.
> https://issues.apache.org/jira/browse/KAFKA-9301
>
> I'll look into how we can fix the issue, but I might not be able to
> submit a PR very soon.
>
> Thanks,
> Tomoyuki
>
> On Mon, Dec 9, 2019 at 1:52 PM Tomoyuki Saito <aocchoda@gmail.com> wrote:
> >
> > Hi Guozhang,
> >
> > Thank you for your suggestion!
> > I'll create a JIRA within a few days and consider submitting a PR.
> >
> > Thanks,
> > Tomoyuki
> >
> > On 2019/12/09 00:37:49, Guozhang Wang <w...@gmail.com> wrote:
> > > Hello Tomoyuki,>
> > >
> > > It seems that issue in 6494 is indeed valid, and I'd personally suggest we>
> > > do option 3) to fix the flush() behavior. Please feel free to create a JIRA>
> > > (and also submit your PR if you are interested in contributing :).>
> > >
> > >
> > > Guozhang>
> > >
> > > On Sat, Dec 7, 2019 at 7:59 AM Tomoyuki Saito <ao...@gmail.com> wrote:>
> > >
> > > > Hi,>
> > > >>
> > > > ## Questions>
> > > >>
> > > > 1. Any possible way to make sure to avoid batch split, or oversized>
> > > > batches?>
> > > > 2. Any progress/discussion to fix the issue mentioned in the following>
> > > > PR: https://github.com/apache/kafka/pull/6469 (kafka#6469)>
> > > >>
> > > > ## Background>
> > > >>
> > > > `FlinkKafkaProducer` expects that callbacks for sent records will be>
> > > > executed after `KafkaProducer#flush`, while executing snapshot. But,>
> > > > it does not hold when batch split happens, as explained in PR>
> > > > kafka#6469. We are observing IllegalStateException for this issue,>
> > > > with Flink deployment.>
> > > >>
> > > > https://github.com/apache/flink/blob/release-1.9.1/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaProducer.java#L960>
> > > >>
> > > > We are looking for a workaround and a resolution for this issue.>
> > > >>
> > > > Possible workaround/resolution ideas:>
> > > > * Avoid batch split>
> > > > * Make Flink wait for inflight requests to be completed somehow>
> > > > * Make `KafkaProducer#flush` block until inflight requests are>
> > > > completed, even when batch split and reenqueue happens>
> > > >>
> > > > Thanks,>
> > > > Tomoyuki>
> > > >>
> > >
> > >
> > > -- >
> > > -- Guozhang>
> > >

Mime
View raw message