kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chia-Chun Shih <chiachun.s...@gmail.com>
Subject Re: kafka brokers going down within 24 hrs
Date Thu, 15 Jan 2015 06:31:21 GMT
You can use tools (e.g., VisialVM) to diagnose OOM problem.

2015-01-15 14:15 GMT+08:00 Tousif Khazi <tousif@senseforth.com>:

> i see this error
>
>  ERROR [ReplicaFetcherThread-0-1], Error for partition
> [realtimestreaming,1] to broker 1:class
> kafka.common.NotLeaderForPartitionException
> (kafka.server.ReplicaFetcherThread)
> [2015-01-15 10:00:04,348] INFO [ReplicaFetcherManager on broker 0]
> Removed fetcher for partitions [realtimestreaming,1]
> (kafka.server.ReplicaFetcherManager)
> [2015-01-15 10:00:04,355] INFO Closing socket connection to
> /10.0.0.11. (kafka.network.Processor)
> [2015-01-15 10:00:04,444] WARN [KafkaApi-0] Fetch request with
> correlation id 0 from client ReplicaFetcherThread-0-0 on partition
> [realtimestreaming,1] failed due to Leader not local for partition
> [realtimestreaming,1] on broker 0 (kafka.server.KafkaApis)
> [2015-01-15 10:00:04,545] INFO [ReplicaFetcherThread-0-1], Shutting
> down (kafka.server.ReplicaFetcherThread)
> [2015-01-15 10:00:04,848] INFO [ReplicaFetcherThread-0-1], Stopped
> (kafka.server.ReplicaFetcherThread)
> [2015-01-15 10:00:04,849] INFO [ReplicaFetcherThread-0-1], Shutdown
> completed (kafka.server.ReplicaFetcherThread)
> [2015-01-15 10:00:39,256] ERROR Closing socket for /10.0.0.11 because
> of error (kafka.network.Processor)
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
>
> On Wed, Jan 14, 2015 at 10:12 PM, Tousif <tousif.pasha@gmail.com> wrote:
> > Thanks harsha for quick response.
> > I don't see any other error. I used to see replica fetcher error but
> seems
> > to be disappeared after setting replica fetcher threads to 2 as I have 2
> > partitions. Some times I see zookeeper session expiration.
> > On Jan 14, 2015 8:02 PM, "Harsha" <kafka@harsha.io> wrote:
> >
> >> Tousif,
> >>        Do you see any other errors in server.log
> >> -Harsha
> >>
> >> On Wed, Jan 14, 2015, at 01:51 AM, Tousif wrote:
> >> > Hello,
> >> >
> >> > I have configured kafka nodes to run via supervisord  and see
> following
> >> > exceptions
> >> > and eventually brokers going out of memory. i have given enough memory
> >> > and
> >> > process 1 event/second. kafka goes down every day
> >> >
> >> > i'm wondering what configurastion is missing or need to be added
> >> >
> >> > Here are my cluster details:
> >> >  2 brokers
> >> >  1 zookeeper
> >> > and 2 node apache storm
> >> >
> >> >
> >> > INFO zookeeper state changed (SyncConnected)
> >> > (org.I0Itec.zkclient.ZkClient)
> >> > ERROR Closing socket for /10.0.0.11 because of error
> >> > (kafka.network.Processor)
> >> > java.io.IOException: Connection reset by peer
> >> > at sun.nio.ch.FileDispatcher.read0(Native Method)
> >> > at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> >> > at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
> >> > at sun.nio.ch.IOUtil.read(IOUtil.java:171)
> >> > at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
> >> > at kafka.utils.Utils$.read(Utils.scala:375)
> >> > at
> >> >
> >>
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
> >> > at kafka.network.Processor.read(SocketServer.scala:347)
> >> > at kafka.network.Processor.run(SocketServer.scala:245)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > [2015-01-13 23:43:37,962] INFO Closing socket connection to /
> 10.0.0.11.
> >> > (kafka.network.Processor)
> >> > Error occurred during initialization of VM
> >> > Could not reserve enough space for object heap
> >> > Error occurred during initialization of VM
> >> > Could not reserve enough space for object heap
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Tousif
> >> > +918050227279
> >> >
> >> >
> >> > --
> >> >
> >> >
> >> > Regards
> >> > Tousif Khazi
> >>
>
>
>
> --
> Regards,
> Tousif
> +918050227279
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message