hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luke Forehand <luke.foreh...@networkedinsights.com>
Subject Re: Hanging regionservers
Date Mon, 19 Jul 2010 15:55:59 GMT
After looking at the stacktrace on regionserver2 this morning, I seem to be experiencing this
issue:

https://issues.apache.org/jira/browse/HBASE-2322

Two questions:  Would this issue cause the primary issue of all my region servers appearing
to hang, and will migrating to cdh3b2 fix this issue?

Thanks
Luke

On 7/19/10 12:24 AM, "Luke Forehand" <luke.forehand@networkedinsights.com> wrote:

Here are pastebin's of my stacktraces and logs.  Note my comment below these links.

regionserver 1 stack trace: http://pastebin.com/0n9cDeYh
regionserver 2 stack trace: http://pastebin.com/8Sppp68h
regionserver 3 stack trace: http://pastebin.com/qzLEjBN0

regionserver 1 log ~5MB: http://pastebin.com/g3aB5L81
regionserver 2 log ~5MB: http://pastebin.com/NDEaUbJv
regionserver 3 log ~5MB: http://pastebin.com/SAVPnr7S

zookeeper 1,2,3 log: http://pastebin.com/33RPTHKX

So...

Am I seeing a deadlock occurring in the regionserver 2 stacktrace?

"IPC Server handler 18 on 60020" - Thread t@65
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@99de7de
owned by: IPC Server handler 17 on 60020
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
        at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:953)
        at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:846)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352)
        - locked org.apache.hadoop.hbase.regionserver.MemStoreFlusher@4c2fe6bf
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321)
        - locked org.apache.hadoop.hbase.regionserver.MemStoreFlusher@4c2fe6bf
        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1775)
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

   Locked ownable synchronizers:
        - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@5cd62cac

        - locked java.util.concurrent.locks.ReentrantLock$NonfairSync@3cf93af4


"IPC Server handler 17 on 60020" - Thread t@64
   java.lang.Thread.State: BLOCKED on java.util.HashSet@1e1b300f owned by: regionserver/192.168.200.32:60020.cacheFlusher
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.request(MemStoreFlusher.java:172)
        at org.apache.hadoop.hbase.regionserver.HRegion.requestFlush(HRegion.java:1524)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1509)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1292)
        at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:1255)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1781)
        at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

   Locked ownable synchronizers:
        - locked java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@99de7de

"regionserver/192.168.200.32:60020.cacheFlusher" - Thread t@18
   java.lang.Thread.State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@5cd62cac
owned by: IPC Server handler 18 on 60020
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:235)
        - locked java.util.HashSet@1e1b300f
        at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:149)

   Locked ownable synchronizers:
       - None

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message