hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Zhenyu Zhong <zhongresea...@gmail.com>
Subject Re: regionserver disconnection
Date Wed, 11 Nov 2009 18:34:14 GMT
I searched around and found HBASE-1645 "HBase could be a little more dainty
when zk is pulled out from under it"

I am wondering whether my problem is related to this HBase-1645. Or probably
has nothing to do with it?

thanks


On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <zhongresearch@gmail.com>wrote:

> Hi,
>
> I have been monitoring the system resources such as memory Heap usage for
> Regionserver, load average, network bandwidth, etc.
>
> I experienced a disconnection last night. When I checked the memory Heap
> for Regionserver, it was 2336 which is way below the max Heap 4096. I don't
> think it is Regionserver problem. The requests that the regionserver got was
> about 126 which doesn't seem to be high. The load average is kind of high
> >100. (but I saw disconnections while the load average is around 30). The
> bandwidth is not a problem.
>
> However, I found error messages from the zookeeper. Please see the
> following error messages. There are some
> BufferUnderflowException error, IOException error,etc.
> May I ask what might be the causes that generate these error messages?
>
> Many thanks
> zhenyu
>
>
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
>         at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>         at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>         at org.apache.zookeeper.server.Request.toString(Request.java:199)
>         at java.lang.String.valueOf(String.java:2827)
>         at java.lang.StringBuilder.append(StringBuilder.java:115)
>         at
> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167)
>         at
> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68)
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn:
> Exception causing close of session 0x424dfd908c50009
> due to java.io.IOException: Read error
> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> IOException stack trace
> java.io.IOException: Read error
>         at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372)
>         at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request::
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
>         at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>         at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>         at org.apache.zookeeper.server.Request.toString(Request.java:199)
>         at java.lang.String.valueOf(String.java:2827)
>         at java.lang.StringBuilder.append(StringBuilder.java:115)
>         at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74)
>         at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2009-11-11 04:29:00,810 DEBUG
> org.apache.zookeeper.server.FinalRequestProcessor: Processing request::
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> java.nio.BufferUnderflowException
>         at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127)
>         at java.nio.ByteBuffer.get(ByteBuffer.java:675)
>         at org.apache.zookeeper.server.Request.toString(Request.java:199)
>         at
> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36)
>         at
> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80)
>         at
> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362)
> at
> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403)
>         at
> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
>         at org.apache.log4j.PatternLayout.format(PatternLayout.java:502)
>         at
> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302)
>         at
> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359)
>         at org.apache.log4j.WriterAppender.append(WriterAppender.java:160)
>         at
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
>         at
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
>         at org.apache.log4j.Category.callAppenders(Category.java:206)
>         at org.apache.log4j.Category.forcedLog(Category.java:391)
>         at org.apache.log4j.Category.debug(Category.java:260)
>         at
> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147)
>         at
> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)
> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring
> exception during toString
> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8
> zxid:0xfffffffffffffffe txntype:unknown n/a
> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn:
> closing session:0x424dfd908c50009 NIOServerCnxn:
> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/
> 192.168.100.131:35500]
> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn:
> ignoring exception during input shutdown
> java.net.SocketException: Transport endpoint is not connected
>         at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
>         at
> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
>         at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848)
>         at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498)
>         at
> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221)
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message