kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Re: unknown (kafka) offsets after restart
Date Mon, 09 May 2016 07:11:43 GMT
I dug further into this and log.cleaner.enable is false by default in 
0.8.2.x which pretty much explains all this. Fixing it now.

Thanks!

On 09.05.2016 08:17, Jörg Wagner wrote:
> Thank you James.
>
> The log cleaner log is empty and we didn't notice any issue there. The 
> Topic however is ~174GB per Node (3) which would support your 
> information about the log cleaner dying.
>
> Since this is a production system I do not think debug logging is 
> viable even for a short time. Could you maybe point me in the 
> direction of debugging compaction where it went wrong?
>
> Thanks
> Jörg
>
> On 06.05.2016 22:21, James Cheng wrote:
>> Is the log compaction thread correctly working? The offsets are 
>> stored in a log compacted topic, and we have seen issues where the 
>> log cleaner thread dies and therefore the offsets topic just grows 
>> forever, which means it will take a long time to read in the topic.
>>
>> You can look in the log-cleaner.log debuglog file to see if there are 
>> any error messages there.
>>
>> -James
>>
>>
>>> On May 6, 2016, at 6:28 AM, Jörg Wagner <joerg.wagner1@1und1.de> wrote:
>>>
>>> After a bit more looking into this we found out that the 
>>> Offsetmanager is single threaded and due to our setup (few, powerful 
>>> servers: rather bad for kafka I know..) it seems we are limiting 
>>> ourselves severely by using kafka offsets.
>>>
>>> Any more insight is still welcome of course.
>>>
>>>
>>> -------- Forwarded Message --------
>>> Subject:     unknown (kafka) offsets after restart
>>> Date:     Fri, 6 May 2016 14:12:24 +0200
>>> From:     Jörg Wagner <joerg.wagner1@1und1.de>
>>> Reply-To:     users@kafka.apache.org
>>> To:     users@kafka.apache.org
>>>
>>>
>>>
>>> We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
>>> they are stored in kafka topics.
>>>
>>> Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
>>> happen that the offsets are unknown and therefore stop consumers from
>>> consuming without knowing their offset.
>>>
>>> kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
>>> --zookeeper localhost:2181
>>> Could not fetch offset for [topic,19] due to
>>> kafka.common.OffsetsLoadInProgressException.
>>>
>>> Group           Topic                          Pid Offset
>>> logSize         Lag             Owner
>>> group   topic    0   unknown         326606          unknown none
>>>
>>> I currently have no other solution to this than to wait.. but it 
>>> takes a
>>> very long time (hours.. the servers are hopelessly underutilized!), I
>>> would be grateful for any advice.
>>>
>>> Thanks
>>> Jörg
>>>
>>>
>>>
>

-- 
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 München
| Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen


Member of United Internet


Mime
View raw message