hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ananth T. Sarathy" <ananth.t.sara...@gmail.com>
Subject Re: Waiting forever on scanner iterator
Date Wed, 21 Oct 2009 18:41:58 GMT
here's what I get

Heap
 par new generation   total 19136K, used 13714K [0x00002aaaafc40000,
0x00002aaab1100000, 0x00002aaab4f70000)
  eden space 17024K,  79% used [0x00002aaaafc40000, 0x00002aaab09816d8,
0x00002aaab0ce0000)
  from space 2112K,   6% used [0x00002aaab0ef0000, 0x00002aaab0f13470,
0x00002aaab1100000)
  to   space 2112K,   0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000,
0x00002aaab0ef0000)
 concurrent mark-sweep generation total 63872K, used 25097K
[0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000)
 concurrent-mark-sweep perm gen total 30220K, used 18263K
[0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000)

2009-10-21 14:40:58
Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):

"pool-1-thread-1" prio=10 tid=0x00002aaafc01a000 nid=0x49f waiting on
condition [0x0000000042e56000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6559490> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
        at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
        at java.lang.Thread.run(Thread.java:619)

"LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaafc01f800
nid=0x49e in Object.wait() [0x0000000042d55000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab6519890> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512)
        - locked <0x00002aaab6519890> (a
org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)

"IPC Server handler 9 on 60020" daemon prio=10 tid=0x00002aaaf7c12000
nid=0x49d waiting on condition [0x0000000042c54000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 8 on 60020" daemon prio=10 tid=0x00002aaaf7c0f800
nid=0x49c waiting on condition [0x0000000042b53000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 7 on 60020" daemon prio=10 tid=0x00002aaaf7c0d800
nid=0x49b waiting on condition [0x0000000042a52000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 6 on 60020" daemon prio=10 tid=0x00002aaafd08b800
nid=0x49a waiting on condition [0x0000000042951000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 5 on 60020" daemon prio=10 tid=0x00002aaafd089800
nid=0x499 waiting on condition [0x0000000042850000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 4 on 60020" daemon prio=10 tid=0x00002aaafd087000
nid=0x498 waiting on condition [0x000000004274f000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 3 on 60020" daemon prio=10 tid=0x00002aaafd085000
nid=0x497 waiting on condition [0x000000004264e000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 2 on 60020" daemon prio=10 tid=0x00002aaaf7adf800
nid=0x496 waiting on condition [0x000000004254d000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 1 on 60020" daemon prio=10 tid=0x00002aaaf7adb800
nid=0x495 waiting on condition [0x000000004244c000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server handler 0 on 60020" daemon prio=10 tid=0x00002aaaf7adb000
nid=0x494 waiting on condition [0x000000004234b000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434798> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)

"IPC Server listener on 60020" daemon prio=10 tid=0x00002aaaf7de9800
nid=0x493 runnable [0x000000004224a000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00002aaab643b548> (a sun.nio.ch.Util$1)
        - locked <0x00002aaab643b530> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00002aaab63ea7e0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)

"IPC Server Responder" daemon prio=10 tid=0x00002aaaf7de9000 nid=0x492
runnable [0x0000000042149000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00002aaab643b458> (a sun.nio.ch.Util$1)
        - locked <0x00002aaab643b440> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00002aaab63ea768> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477)

"regionserver/10.244.9.171:60020.safeMode" daemon prio=10
tid=0x00002aaaf7c84000 nid=0x491 in Object.wait() [0x0000000042048000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab63911f8> (a
java.util.concurrent.atomic.AtomicBoolean)
        at java.lang.Object.wait(Object.java:485)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionLimitThread.run(HRegionServer.java:902)
        - locked <0x00002aaab63911f8> (a
java.util.concurrent.atomic.AtomicBoolean)

"Timer-0" daemon prio=10 tid=0x00002aaaf7de4000 nid=0x490 in Object.wait()
[0x0000000041f47000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab6558e60> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
        - locked <0x00002aaab6558e60> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

"853527367@qtp0-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030" prio=10
tid=0x00002aaafcd3a800 nid=0x48f runnable [0x0000000041e46000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00002aaab651bf38> (a sun.nio.ch.Util$1)
        - locked <0x00002aaab651bf50> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00002aaab6620c60> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
        at
org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
        at
org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
        at
org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

"1096304647@qtp0-0" prio=10 tid=0x00002aaaf7d5b800 nid=0x48e in
Object.wait() [0x0000000041d45000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab653acd0> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)
        at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565)
        - locked <0x00002aaab653acd0> (a
org.mortbay.thread.QueuedThreadPool$PoolThread)

"regionserver/10.244.9.171:60020.leaseChecker" prio=10
tid=0x00002aaafc16a000 nid=0x48d waiting on condition [0x0000000041c44000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6434518> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209)
        at org.apache.hadoop.hbase.Leases.run(Leases.java:81)

"regionserver/10.244.9.171:60020.majorCompactionChecker" daemon prio=10
tid=0x00002aaafc168000 nid=0x48c waiting on condition [0x0000000041b43000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:76)

"regionserver/10.244.9.171:60020.worker" daemon prio=10
tid=0x00002aaafc228000 nid=0x48b waiting on condition [0x0000000041a42000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab63a3b68> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
        at
java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1459)
        at java.lang.Thread.run(Thread.java:619)

"regionserver/10.244.9.171:60020.compactor" daemon prio=10
tid=0x00002aaafc226000 nid=0x48a waiting on condition [0x0000000041941000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:79)

"regionserver/10.244.9.171:60020.cacheFlusher" daemon prio=10
tid=0x00002aaafc225000 nid=0x489 waiting on condition [0x0000000041840000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:137)

"regionserver/10.244.9.171:60020.logFlusher" daemon prio=10
tid=0x00002aaafc212000 nid=0x488 waiting on condition [0x000000004173f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
        at org.apache.hadoop.hbase.Chore.run(Chore.java:76)

"regionserver/10.244.9.171:60020.logRoller" daemon prio=10
tid=0x00002aaafc211800 nid=0x487 in Object.wait() [0x000000004153d000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab63913f0> (a
java.util.concurrent.atomic.AtomicBoolean)
        at
org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65)
        - locked <0x00002aaab63913f0> (a
java.util.concurrent.atomic.AtomicBoolean)

"MultiThreadedHttpConnectionManager cleanup" daemon prio=10
tid=0x00002aaaf7e14000 nid=0x486 in Object.wait() [0x000000004163e000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab65446e0> (a
java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00002aaab65446e0> (a java.lang.ref.ReferenceQueue$Lock)
        at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)

"IPC Client (47) connection to /10.244.9.171:60000 from an unknown user"
daemon prio=10 tid=0x00002aaaf80b6800 nid=0x46d in Object.wait()
[0x000000004143c000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab642ba08> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:405)
        - locked <0x00002aaab642ba08> (a
org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
        at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)

"DestroyJavaVM" prio=10 tid=0x00002aaaf80b4800 nid=0x455 waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"regionserver/10.244.9.171:60020" prio=10 tid=0x00002aaaf80b3800 nid=0x467
waiting on condition [0x000000004133b000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:596)
        at java.lang.Thread.run(Thread.java:619)

"main-EventThread" daemon prio=10 tid=0x00002aaaf80ab800 nid=0x466 waiting
on condition [0x000000004123a000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00002aaab6391598> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
        at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
        at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:376)

"main-SendThread" daemon prio=10 tid=0x00002aaaf80ab000 nid=0x465 runnable
[0x0000000041139000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
        - locked <0x00002aaab6418908> (a sun.nio.ch.Util$1)
        - locked <0x00002aaab64188f0> (a
java.util.Collections$UnmodifiableSet)
        - locked <0x00002aaab63fb0a0> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:873)

"Low Memory Detector" daemon prio=10 tid=0x00002aaaf7858800 nid=0x463
runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00002aaaf7856000 nid=0x462 waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00002aaaf7852800 nid=0x461 waiting on
condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00002aaaf76b9800 nid=0x460 waiting
on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaaf76b7800
nid=0x45f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00002aaaf7695800 nid=0x45e in Object.wait()
[0x0000000040a32000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab6396438> (a
java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        - locked <0x00002aaab6396438> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00002aaaf7693800 nid=0x45c in
Object.wait() [0x0000000040931000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:485)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        - locked <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00002aaaf768d000 nid=0x45b runnable

"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040119000
nid=0x456 runnable

"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x000000004011b000
nid=0x457 runnable

"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x000000004011c800
nid=0x458 runnable

"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x000000004011e800
nid=0x459 runnable

"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaf3d9d800 nid=0x45a
runnable
"VM Periodic Task Thread" prio=10 tid=0x00002aaaf785b000 nid=0x464 waiting
on condition

JNI global references: 806

Heap
 par new generation   total 19136K, used 14022K [0x00002aaaafc40000,
0x00002aaab1100000, 0x00002aaab4f70000)
  eden space 17024K,  81% used [0x00002aaaafc40000, 0x00002aaab09ce548,
0x00002aaab0ce0000)
  from space 2112K,   6% used [0x00002aaab0ef0000, 0x00002aaab0f13470,
0x00002aaab1100000)
  to   space 2112K,   0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000,
0x00002aaab0ef0000)
 concurrent mark-sweep generation total 63872K, used 25097K
[0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000)
 concurrent-mark-sweep perm gen total 30220K, used 18263K
[0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000)



Ananth T Sarathy


On Wed, Oct 21, 2009 at 1:24 PM, stack <stack@duboce.net> wrote:

> If thats all you get, a thread dump on 10.242.71.191:60020 would help ($
> kill -QUIT RS_PID).  Thread dump will be in the RS .out file.  Do a few.
> Paste to pastebin.
> Thanks Ananth,
> St.Ack
>
> On Wed, Oct 21, 2009 at 8:05 AM, Ananth T. Sarathy <
> ananth.t.sarathy@gmail.com> wrote:
>
> > yeah,
> >  I just don't understand why getScanner("Column:X") returns the iterator
> > and
> > process them yet getScanner("Column:Y") just spins and spins, yet
>  Column:Y
> > is a much denser result.
> >
> > When I load from shell
> >
> > *Version: 0.20.0, r810752, Thu Sep  3 00:06:18 PDT 2009
> > hbase(main):001:0> count 'GS_Applications'
> > 09/10/21 11:04:48 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
> > /hbase/root-region-server got 10.244.9.171:60020
> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$ClientZKWatcher: Got
> > ZooKeeper event, state: SyncConnected, type: None, path: null
> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Found
> ROOT
> > at 10.244.9.171:60020
> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME =>
> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> =>
> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
> =>
> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
=>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}]}}
> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 10.242.71.191:60020, regioninfo: REGION => {NAME =>
> > 'GS_Applications,,1255020109210', STARTKEY => '', ENDKEY => '', ENCODED
> =>
> > 1732076772, TABLE => {{NAME => 'GS_Applications', FAMILIES => [{NAME =>
> > 'Application', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
> '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME
> =>
> > 'BinaryData', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
> '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}, {NAME
> =>
> > 'BinaryRetrieval', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'true'}, {NAME => 'Files', COMPRESSION => 'NONE', VERSIONS => '1', TTL
=>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'true'}, {NAME => 'Info', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'true'}, {NAME => 'Network', COMPRESSION => 'NONE', VERSIONS => '1', TTL
> =>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'true'}, {NAME => 'Registry', VERSIONS => '1', COMPRESSION => 'NONE', TTL
> > =>
> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
> > 'true'}]}}
> > 09/10/21 11:04:49 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
> > /hbase/root-region-server got 10.244.9.171:60020
> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Found
> ROOT
> > at 10.244.9.171:60020
> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached
> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME =>
> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
> =>
> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
> =>
> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
=>
> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
> > BLOCKCACHE => 'false'}]}}
> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Creating scanner
> over
> > GS_Applications starting at key ''
> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Advancing internal
> > scanner to startKey at ''
> > 09/10/21 11:04:50 DEBUG client.HConnectionManager$TableServers: Cache hit
> > for row <> in tableName GS_Applications: location server
> > 10.242.71.191:60020,
> > location region name GS_Applications,,1255020109210
> >
> >
> > *Ananth T Sarathy
> >
> >
> > On Wed, Oct 21, 2009 at 11:00 AM, stack <stack@duboce.net> wrote:
> >
> > > In both cases you are doing a full table scan?
> > >
> > > Try from shell with DEBUG enable.  You'll see the regions being loaded.
> > >  May
> > > help you narrow in on problem region or at least on problem
> regionserver.
> > >
> > > St.Ack
> > >
> > > On Wed, Oct 21, 2009 at 7:19 AM, Ananth T. Sarathy <
> > > ananth.t.sarathy@gmail.com> wrote:
> > >
> > > > Anyone have any further thoughts on this?
> > > > Ananth T Sarathy
> > > >
> > > >
> > > > On Tue, Oct 20, 2009 at 6:37 PM, Ananth T. Sarathy <
> > > > ananth.t.sarathy@gmail.com> wrote:
> > > >
> > > > > Well that's not the case. Every Row has that column.  In fact the
> > > second
> > > > > snippet i sent  is with a column with many less rows. (1k vs 25k)
> but
> > > > comes
> > > > > back pretty quickly.
> > > > >
> > > > > By forever, I mean i have watched my logs do nothing for a half
> hour
> > > > before
> > > > > giving up.
> > > > >
> > > > >
> > > > > Ananth T Sarathy
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Oct 20, 2009 at 5:03 PM, Ryan Rawson <ryanobjc@gmail.com>
> > > wrote:
> > > > >
> > > > >> If you are asking for a column that is very sparse and doesnt
> exist,
> > > > >> it will cause HBase to read through the entire region to find
100
> > > > >> matching rows. This could take a while, you said 'forever', but
> > could
> > > > >> you quantify that?
> > > > >>
> > > > >> On Tue, Oct 20, 2009 at 1:58 PM, Jean-Daniel Cryans <
> > > > jdcryans@apache.org>
> > > > >> wrote:
> > > > >> > Scanner pre-fetching is always faster, so something must
be
> wrong
> > > with
> > > > >> > your region server. Check the logs, top, etc
> > > > >> >
> > > > >> > WRT to row size, it's pretty much a matter of how many bytes
you
> > > have
> > > > >> > in each column and sum them up (plus some overhead with
the
> keys).
> > > > >> >
> > > > >> > You want filters, check the filter package in the javadoc.
> > > > >> >
> > > > >> > J-D
> > > > >> >
> > > > >> > On Tue, Oct 20, 2009 at 1:52 PM, Ananth T. Sarathy
> > > > >> > <ananth.t.sarathy@gmail.com> wrote:
> > > > >> >> Ok, but how come
> > > > >> >> when I run a similiar call (with less returned rows
1000 vs 25k
> > in
> > > > the
> > > > >> >> previous one) it runs through the iterator very quickly?
 (See
> > > Below)
> > > > >> >>
> > > > >> >> Also, how do I determine the row size? It's just text
data, and
> > > > really
> > > > >> not
> > > > >> >> much.
> > > > >> >>
> > > > >> >> Finally, is there a way to query for rows that do not
have a
> > > column?
> > > > >> (Ie all
> > > > >> >> rows without Files:path1)
> > > > >> >>
> > > > >> >>        HBaseTableDataManagerImpl htdmni = new
> > > > >> HBaseTableDataManagerImpl(
> > > > >> >>                "GS_Applications");
> > > > >> >>
> > > > >> >>        String[] columns = { "Files:path1" };
> > > > >> >>        log.info("Getting all Rows with Files");
> > > > >> >>        Scanner s = htdmni.getScannerForAllRows(columns);
> > > > >> >>        log.info("Got all Rows with Files");
> > > > >> >>
> > > > >> >>        Iterator<RowResult> iter = s.iterator();
> > > > >> >>        out
> > > > >> >>
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> .write("Application_Full_Name,Version,Application_installer_name,Operating
> > > > >> >> System, Application_platform
> > > > >> >>
> > > > >>
> > > >
> > >
> >
> ,Application_sub_category,md5Hash,Sha1Hash,Sha256Hash,filepath,fileName,modified,size,operation\n");
> > > > >> >>        out.write("<BR>");
> > > > >> >>        while (iter.hasNext())
> > > > >> >>        {
> > > > >> >>
> > > > >> >> Ananth T Sarathy
> > > > >> >>
> > > > >> >>
> > > > >> >> On Tue, Oct 20, 2009 at 4:44 PM, Jean-Daniel Cryans
<
> > > > >> jdcryans@apache.org>wrote:
> > > > >> >>
> > > > >> >>> If you have a very slow data source (S3), then it
fetches 100
> > row
> > > > >> >>> before coming back to your client with all of them
and that
> can
> > > take
> > > > a
> > > > >> >>> lot of time. Also make sure that 100 of your rows
can fit in a
> > > > region
> > > > >> >>> server's memory. How big is each row?
> > > > >> >>>
> > > > >> >>> J-D
> > > > >> >>>
> > > > >> >>> On Tue, Oct 20, 2009 at 1:32 PM, Ananth T. Sarathy
> > > > >> >>> <ananth.t.sarathy@gmail.com> wrote:
> > > > >> >>> > I am running this code where
> > > > >> >>> >
> > > > >> >>> > getScannerForAllRows(columns) just does return
> > > > >> table.getScanner(columns);
> > > > >> >>> >
> > > > >> >>> > and the table   has setScannerCaching(100);
> > > > >> >>> >
> > > > >> >>> > But it spins forever after getting the iterator.
Why would
> > that
> > > > be?
> > > > >> How
> > > > >> >>> can
> > > > >> >>> > I speed it up?
> > > > >> >>> >
> > > > >> >>> >        HBaseTableDataManagerImpl htdmni = new
> > > > >> HBaseTableDataManagerImpl(
> > > > >> >>> >                "GS_Applications");
> > > > >> >>> >
> > > > >> >>> >        String[] columns = { "Files:Name" };
> > > > >> >>> >        log.info("Getting all Rows with Files");
> > > > >> >>> >        Scanner s = htdmni.getScannerForAllRows(columns);
> > > > >> >>> >        log.info("Got all Rows with Files");
> > > > >> >>> >        log.info("Getting Iterator");
> > > > >> >>> >
> > > > >> >>> >        Iterator<RowResult> iter = s.iterator();
> > > > >> >>> >        log.info("Got Iterator");
> > > > >> >>> >
> > > > >> >>> >        while (iter.hasNext())
> > > > >> >>> >        {
> > > > >> >>> >            log.info("Getting next Row");
> > > > >> >>> >            RowResult rr = iter.next();
> > > > >> >>> >
> > > > >> >>> >
> > > > >> >>> > Ananth T Sarathy
> > > > >> >>> >
> > > > >> >>>
> > > > >> >>
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message