kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörg Wagner <joerg.wagn...@1und1.de>
Subject Fwd: unknown (kafka) offsets after restart
Date Fri, 06 May 2016 13:28:03 GMT
After a bit more looking into this we found out that the Offsetmanager 
is single threaded and due to our setup (few, powerful servers: rather 
bad for kafka I know..) it seems we are limiting ourselves severely by 
using kafka offsets.

Any more insight is still welcome of course.


-------- Forwarded Message --------
Subject: 	unknown (kafka) offsets after restart
Date: 	Fri, 6 May 2016 14:12:24 +0200
From: 	Jörg Wagner <joerg.wagner1@1und1.de>
Reply-To: 	users@kafka.apache.org
To: 	users@kafka.apache.org



We're using Kafka 0.8.2 and are puzzled by the offset behaviour when
they are stored in kafka topics.

Upon restart of the Kafka cluster (e.g. due to reconfiguration) it can
happen that the offsets are unknown and therefore stop consumers from
consuming without knowing their offset.

kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group group
--zookeeper localhost:2181
Could not fetch offset for [topic,19] due to
kafka.common.OffsetsLoadInProgressException.

Group           Topic                          Pid Offset
logSize         Lag             Owner
group   topic    0   unknown         326606          unknown none

I currently have no other solution to this than to wait.. but it takes a
very long time (hours.. the servers are hopelessly underutilized!), I
would be grateful for any advice.

Thanks
Jörg




Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message