kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mads Tandrup <Mads.Tand...@schneider-electric.com>
Subject Re: What is the performance impact of setting max.poll.records=1
Date Fri, 11 May 2018 05:46:00 GMT
Hi

I forgot to metion that I have multiple partitions and multiple consumer processes.
But we can't process the messages in the same partition in parallel since they might influence
the processing of later records.

Does max.poll.records=1 always go to the remote server each time? What if I increase fetch.min.bytes
to say the expected size of 10 records. What will then happen?

Best regards,
Mads

D. 07/05/2018 06.36 skrev "R Krishna" <krishna81m@gmail.com>:

    You can always add more partitions/consumer threads each fetching a few
    more records than 1 but manually commit asynchronously one at a time, not
    the best but better than doing max.poll.records=1 which fetches one record
    from remote server at a time.
    
    On Fri, May 4, 2018 at 4:19 AM, Mads Tandrup <
    Mads.Tandrup@schneider-electric.com> wrote:
    
    > Hi
    >
    > What is the performance impact of setting `max.poll.records=1` as opposed
    > to the default of 500?
    >
    > I have a Java application which process records one at a time. The
    > processing time varies between messages, so we sometimes exceed the `
    > max.poll.interval.ms`.
    > While I could increase `max.poll.interval.ms` it would prevent me from
    > detecting a livelock in the application quickly.
    > There's no benefit of batching the records so I'm considering setting
    > `max.poll.records=1`. We can define a sensible upper limit for the
    > processing time of a single record.
    >
    > I've tried to look at the code and it seems that it fetches up to `
    > fetch.max.bytes` and then keep it in-memory and returns records from the
    > fetched data when `poll()` is called.
    >
    > So what is the performance impact of a low `max.poll.records`?
    >
    > Best regards,
    > Mads
    >
    >
    
    
    -- 
    Radha Krishna, Proddaturi
    253-234-5657
    
    
    ______________________________________________________________________
    This email has been scanned by the Symantec Email Security.cloud service.
    ______________________________________________________________________

Mime
View raw message