kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shravan R <skr...@gmail.com>
Subject Re: __consumer_offsets too big
Date Tue, 16 Jan 2018 23:07:19 GMT
BTW, I see log segments as old as last year and offsets.retention.minutes
is set to 4 days. Any reason why it may have not been deleted?

-rw-r--r-- 1 kafka kafka 104857532 Apr  5  2017 00000000000000000000.log
-rw-r--r-- 1 kafka kafka 104857564 Apr  6  2017 00000000000001219197.log
-rw-r--r-- 1 kafka kafka 104856962 Apr  6  2017 00000000000002438471.log
-rw-r--r-- 1 kafka kafka 104857392 Apr  6  2017 00000000000003657738.log
-rw-r--r-- 1 kafka kafka 104857564 Apr  6  2017 00000000000004877010.log
-rw-r--r-- 1 kafka kafka 104857392 Apr  7  2017 00000000000006096284.log
-rw-r--r-- 1 kafka kafka 104857478 Apr  7  2017 00000000000007315556.log
-rw-r--r-- 1 kafka kafka 104857306 Apr  7  2017 00000000000008534829.log
-rw-r--r-- 1 kafka kafka 104857134 Apr  7  2017 00000000000009754100.log
-rw-r--r-- 1 kafka kafka 104857564 Apr  7  2017 00000000000010973369.log
-rw-r--r-- 1 kafka kafka 104857564 Apr  7  2017 00000000000012192643.log
-rw-r--r-- 1 kafka kafka 104857578 Apr  7  2017 00000000000013411917.log


On Tue, Jan 16, 2018 at 1:04 PM, Shravan R <skr989@gmail.com> wrote:

> I looked into it. I played with log.cleaner.dedupe.buffer.size between
> 256MB to 2GB while keeping log.cleaner.threads=1 but that did not help
> me. I helped me to recover from __consumer_offsets-33 but got into a
> similar exception on another partition. There no lags on our system and
> that is not a concern at this time. Is there any work around or tuning that
> I can do?
>
> Thanks,
> SK
>
> On Tue, Jan 16, 2018 at 10:56 AM, naresh Goud <nareshgoud.dulam@gmail.com>
> wrote:
>
>> Can you check if jira KAFKA-3894 helps?
>>
>>
>> Thank you,
>> Naresh
>>
>> On Tue, Jan 16, 2018 at 10:28 AM Shravan R <skr989@gmail.com> wrote:
>>
>> > We are running Kafka-0.9 and I am seeing large __consumer_offsets on
>> some
>> > of the partitions of the order of 100GB or more. I see some of the log
>> and
>> > index files are more than a year old.  I see the following properties
>> that
>> > are of interest.
>> >
>> > offsets.retention.minutes=5769 (4 Days)
>> > log.cleaner.dedupe.buffer.size=256000000 (256MB)
>> > num.recovery.threads.per.data.dir=4
>> > log.cleaner.enable=true
>> > log.cleaner.threads=1
>> >
>> >
>> > Upon restarting of the broker, I see the below exception which clearly
>> > indicates a problem with dedupe buffer size. However, I see the dedupe
>> > buffer size is set to 256MB which is far more than what the log
>> complains
>> > about (37MB). What could be the problem here? How can I get the offsets
>> > topic size under manageable size?
>> >
>> >
>> > 2018-01-15 21:26:51,434 ERROR kafka.log.LogCleaner:
>> > [kafka-log-cleaner-thread-0], Error due to
>> > java.lang.IllegalArgumentException: requirement failed: 990238234
>> messages
>> > in segment __consumer_offsets-33/00000000000000000000.log but offset
>> map
>> > can
>> >  fit only 37499999. You can increase log.cleaner.dedupe.buffer.size or
>> > decrease log.cleaner.threads
>> >         at scala.Predef$.require(Predef.scala:219)
>> >         at
>> > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:591)
>> >         at
>> > kafka.log.Cleaner$$anonfun$buildOffsetMap$4.apply(LogCleaner.scala:587)
>> >         at
>> >
>> > scala.collection.immutable.Stream$StreamWithFilter.foreach(
>> Stream.scala:570)
>> >         at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:587)
>> >         at kafka.log.Cleaner.clean(LogCleaner.scala:329)
>> >         at
>> > kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:237)
>> >         at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:
>> 215)
>> >         at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:
>> 63)
>> > 2018-01-15 21:26:51,436 INFO kafka.log.LogCleaner:
>> > [kafka-log-cleaner-thread-0], Stopped
>> >
>> >
>> >
>> > Thanks,
>> > -SK
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message