kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Guozhang Wang <wangg...@gmail.com>
Subject Re: Producer errors (failed to send producer request, failed to send requests for topics)
Date Thu, 25 Sep 2014 01:02:43 GMT
Kyle, the new producer will handle this exception with the specific
exception type, and the callback handling function can treat it
accordingly. Could you give the new producer client a try and see if that
is better handled now?

On Tue, Sep 23, 2014 at 8:30 PM, Kyle Banker <kylebanker@gmail.com> wrote:

> Thanks so much, Jun. That seems to have fixed the problem. I increased both
> message.max.bytes and replica.fetch.max.bytes on the broker.
>
> For the benefit of future Kafka users, how hard would it be to build out
> some clearer error messaging for this case?
>
> On Mon, Sep 22, 2014 at 10:38 PM, Jun Rao <junrao@gmail.com> wrote:
>
> > Also, don't forget to increase replica.fetch.max.bytes to be larger than
> > the max message size.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Sep 22, 2014 at 9:35 PM, Jun Rao <junrao@gmail.com> wrote:
> >
> > > What version of Kafka are you using? Have you increased the max message
> > > size on the broker (default to 1MB)?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, Sep 22, 2014 at 3:41 PM, Kyle Banker <kylebanker@gmail.com>
> > wrote:
> > >
> > >> I have a test data set of 1500 messages (~2.5 MB each) that I'm using
> to
> > >> test Kafka throughput. I'm pushing this data using 40 Kafka producers,
> > and
> > >> I'm losing about 10% of the message on each trial.
> > >>
> > >> I'm seeing errors of the following form:
> > >>
> > >> Failed to send producer request with correlation id 80 to broker 6
> with
> > >> data for partitions [kafka_end_test_3r,339]
> > >> Produce request with correlation id 88 failed due to
> > >> [kafka_end_test_3r,317]: kafka.common.RequestTimedOutException
> > >> Failed to send requests for topics kafka_end_test_3r with correlation
> > ids
> > >> in [15,77]
> > >>
> > >> My topic has 400 partitions and a replication factor of 3. I have a
> > 9-node
> > >> broker cluster.
> > >>
> > >> In the producer, I've set request.required.acks to -1,
> retry.backoff.ms
> > >> to
> > >> 500, message.send.max.retries to 20.
> > >>
> > >> What am I doing wrong? How can I make Kafka to accept 100% of
> messages?
> > >> Any
> > >> help would be greatly appreciated.
> > >>
> > >
> > >
> >
>



-- 
-- Guozhang

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message