kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brett Rann <br...@zendesk.com.INVALID>
Subject Re: Big Log Retention
Date Fri, 07 Sep 2018 06:08:45 GMT
We have partitions that are in the 100s of GBs.

It shouldn't have to shuffle around GB chunks of data unless you have done
partition moves, or had a broker offline for a while. Is that the case?

If not your ISR problem is probably related to something else other than
retention size.

On Fri, Sep 7, 2018 at 7:25 AM Emmett Butler <emmett@parsely.com> wrote:

> Hi users,
>
> What's the biggest topic you've seen in production use? The factors I'm
> interested in are log.retention.* config parameters and number of
> partitions.
>
> We run several 200-partition topics that have log retention set to roughly
> 1GB per partition. Is this big by common usage standards, or not so big?
>
> I ask because we've had some situations in which ISRs appeared to take an
> inordinately long time to synchronize, and I'm guessing this is partially
> related to the need to shuttle around 1GB hunks of data.
>
> --
> Emmett Butler | Senior Software Engineer
> <
> http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature
> <http://www.parsely.com/?utm_source=Signature&utm_medium=emmett-butler&utm_campaign=Signature>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message