kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qi Xu <shkir...@gmail.com>
Subject Kafka cluster cannot start anymore after unexpected shutdown
Date Sat, 12 Sep 2015 00:53:27 GMT
Hi,
We're running the Trunk version of Kafka (for its SSL feature) and recently
I'm trying to enable the kafka manager with it.
After enabling that,  I find out some machine's Kafka Server is dead.
Looking at the server.log, it has the following logs.
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space


And then I tried to start Kafka on these dead machines, but find the errors
below:
[2015-09-12 00:43:44,878] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,883] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,888] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,895] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,900] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,906] WARN Found an corrupted index file,
/datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,912] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,917] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index,
deleting and rebuilding index... (kafka.log.Log)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
[2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space

>From zookeeper, I can see that node appear in /brokers/ids, but disappear
when the error above happens.

Do you know what's the problem? Any hint is very appreciated.

Thanks,
Qi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message