kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax" <matth...@confluent.io>
Subject Re: What is the performance impact of setting max.poll.records=1
Date Fri, 11 May 2018 16:27:32 GMT
`max.poll.records` only configures how many records are returned from
poll(). Internally, the consumer buffers a batch or records and only if
this batch is empty, if will do a new fetch request within poll().


-Matthias


On 5/10/18 10:46 PM, Mads Tandrup wrote:
> Hi
> 
> I forgot to metion that I have multiple partitions and multiple consumer processes.
> But we can't process the messages in the same partition in parallel since they might
influence the processing of later records.
> 
> Does max.poll.records=1 always go to the remote server each time? What if I increase
fetch.min.bytes to say the expected size of 10 records. What will then happen?
> 
> Best regards,
> Mads
> 
> D. 07/05/2018 06.36 skrev "R Krishna" <krishna81m@gmail.com>:
> 
>     You can always add more partitions/consumer threads each fetching a few
>     more records than 1 but manually commit asynchronously one at a time, not
>     the best but better than doing max.poll.records=1 which fetches one record
>     from remote server at a time.
>     
>     On Fri, May 4, 2018 at 4:19 AM, Mads Tandrup <
>     Mads.Tandrup@schneider-electric.com> wrote:
>     
>     > Hi
>     >
>     > What is the performance impact of setting `max.poll.records=1` as opposed
>     > to the default of 500?
>     >
>     > I have a Java application which process records one at a time. The
>     > processing time varies between messages, so we sometimes exceed the `
>     > max.poll.interval.ms`.
>     > While I could increase `max.poll.interval.ms` it would prevent me from
>     > detecting a livelock in the application quickly.
>     > There's no benefit of batching the records so I'm considering setting
>     > `max.poll.records=1`. We can define a sensible upper limit for the
>     > processing time of a single record.
>     >
>     > I've tried to look at the code and it seems that it fetches up to `
>     > fetch.max.bytes` and then keep it in-memory and returns records from the
>     > fetched data when `poll()` is called.
>     >
>     > So what is the performance impact of a low `max.poll.records`?
>     >
>     > Best regards,
>     > Mads
>     >
>     >
>     
>     
>     -- 
>     Radha Krishna, Proddaturi
>     253-234-5657
>     
>     
>     ______________________________________________________________________
>     This email has been scanned by the Symantec Email Security.cloud service.
>     ______________________________________________________________________
> 


Mime
View raw message