kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Qi Xu <shkir...@gmail.com>
Subject Re: Kafka cluster cannot start anymore after unexpected shutdown
Date Sat, 12 Sep 2015 01:02:36 GMT
And I tried to clean up the whole kafka-logs folder, and then starts the
kafka server again.
It will the following errors:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/kafka/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2015-09-12 00:58:32,203] WARN No meta.properties file under dir
/datadrive/kafka-logs/meta.properties
(kafka.server.BrokerMetadataCheckpoint)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-33"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-51"
log4j:ERROR No output stream or file set for the appender named
[kafkaAppender].

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-56"
[2015-09-12 00:59:54,546] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-33': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:34,261] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-51': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:51,108] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-56': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:49,121] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:57,643] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:54,028] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-40': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-58': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-46"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-61"
[2015-09-12 01:00:53,043] WARN Session 0x24f84cb22a90029 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,419] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,051] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:23,489] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-46': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:21,313] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:24,438] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-61': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:18,075] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:02,539] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:34,023] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:35,676] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)


And the broker does not appear in zookeeper.
Any idea?

Thanks,
Qi

On Fri, Sep 11, 2015 at 5:53 PM, Qi Xu <shkirkqi@gmail.com> wrote:

> Hi,
> We're running the Trunk version of Kafka (for its SSL feature) and
> recently I'm trying to enable the kafka manager with it.
> After enabling that,  I find out some machine's Kafka Server is dead.
> Looking at the server.log, it has the following logs.
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
>
> And then I tried to start Kafka on these dead machines, but find the
> errors below:
> [2015-09-12 00:43:44,878] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-139/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,883] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-166/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,888] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-114/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,895] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-15/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,900] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-178/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,906] WARN Found an corrupted index file,
> /datadrive/kafka-logs/__consumer_offsets-4/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,912] WARN Found an corrupted index file,
> /datadrive/kafka-logs/userlogs-147/00000000000000000000.index, deleting and
> rebuilding index... (kafka.log.Log)
> [2015-09-12 00:43:44,917] WARN Found an corrupted index file,
> /datadrive/kafka-logs/exoactivitylogs-95/00000000000000000000.index,
> deleting and rebuilding index... (kafka.log.Log)
> Exception in thread "kafka-log-cleaner-thread-0"
> java.lang.OutOfMemoryError: Java heap space
>
> Exception: java.lang.OutOfMemoryError thrown from the
> UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
> [2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
> 10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
> and attempting reconnect (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
> (org.apache.zookeeper.ClientCnxn)
> [2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
> 'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
>
> From zookeeper, I can see that node appear in /brokers/ids, but disappear
> when the error above happens.
>
> Do you know what's the problem? Any hint is very appreciated.
>
> Thanks,
> Qi
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message