kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Re: unknown (kafka) offsets after restart
Date Mon, 09 May 2016 06:17:30 GMT
Thank you James.

The log cleaner log is empty and we didn't notice any issue there. The 
Topic however is ~174GB per Node (3) which would support your 
information about the log cleaner dying.

Since this is a production system I do not think debug logging is viable 
even for a short time. Could you maybe point me in the direction of 
debugging compaction where it went wrong?

Thanks
Jörg

On 06.05.2016 22:21, James Cheng wrote:
> Is the log compaction thread correctly working? The offsets are stored in a log compacted
topic, and we have seen issues where the log cleaner thread dies and therefore the offsets
topic just grows forever, which means it will take a long time to read in the topic.
>
> You can look in the log-cleaner.log debuglog file to see if there are any error messages
there.
>
> -James
>
>
>> On May 6, 2016, at 6:28 AM, Jörg Wagner <joerg.wagner1@1und1.de> wrote:
>>
>> After a bit more looking into this we found out that the Offsetmanager is single
threaded and due to our setup (few, powerful servers: rather bad for kafka I know..) it seems
we are limiting ourselves severely by using kafka offsets.
>>
>> Any more insight is still welcome of course.
>>
>>
>> -------- Forwarded Message --------
>> Subject: 	unknown (kafka) offsets after restart
>> Date: 	Fri, 6 May 2016 14:12:24 +0200
>> From: 	Jörg Wagner <joerg.wagner1@1und1.de>
>> Reply-To: 	users@kafka.apache.org
>> To: 	users@kafka.apache.org
>>
>>
>>
>> We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
>> they are stored in kafka topics.
>>
>> Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
>> happen that the offsets are unknown and therefore stop consumers from
>> consuming without knowing their offset.
>>
>> kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
>> --zookeeper localhost:2181
>> Could not fetch offset for [topic,19] due to
>> kafka.common.OffsetsLoadInProgressException.
>>
>> Group           Topic                          Pid Offset
>> logSize         Lag             Owner
>> group   topic    0   unknown         326606          unknown none
>>
>> I currently have no other solution to this than to wait.. but it takes a
>> very long time (hours.. the servers are hopelessly underutilized!), I
>> would be grateful for any advice.
>>
>> Thanks
>> Jörg
>>
>>
>>

-- 
Mit freundlichem Gruß

Jörg Wagner

Systemadministrator
Search & Account Security

1&1 Mail & Media Development  & Technology GmbH | Sapporobogen 6-8 | 80637 München
| Germany
Phone: +49 89 14339 324
E-Mail: joerg.wagner1@1und1.de | Web: www.1und1.de

Amtsgericht Montabaur, HRB 5452

Geschäftsführer: Frank Einhellinger, Thomas Ludwig, Jan Oetjen


Member of United Internet


Mime
View raw message