hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ananth T. Sarathy" <ananth.t.sara...@gmail.com>
Subject Re: Waiting forever on scanner iterator
Date Wed, 21 Oct 2009 18:49:35 GMT
solrry...

http://pastebin.com/m5aa78a09


Ananth T Sarathy


On Wed, Oct 21, 2009 at 2:41 PM, Ananth T. Sarathy <
ananth.t.sarathy@gmail.com> wrote:

> here's what I get
>
> Heap
>  par new generation   total 19136K, used 13714K [0x00002aaaafc40000,
> 0x00002aaab1100000, 0x00002aaab4f70000)
>   eden space 17024K,  79% used [0x00002aaaafc40000, 0x00002aaab09816d8,
> 0x00002aaab0ce0000)
>   from space 2112K,   6% used [0x00002aaab0ef0000, 0x00002aaab0f13470,
> 0x00002aaab1100000)
>   to   space 2112K,   0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000,
> 0x00002aaab0ef0000)
>  concurrent mark-sweep generation total 63872K, used 25097K
> [0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000)
>  concurrent-mark-sweep perm gen total 30220K, used 18263K
> [0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000)
>
> 2009-10-21 14:40:58
> Full thread dump Java HotSpot(TM) 64-Bit Server VM (14.2-b01 mixed mode):
>
> "pool-1-thread-1" prio=10 tid=0x00002aaafc01a000 nid=0x49f waiting on
> condition [0x0000000042e56000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6559490> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>         at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576)
>         at
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
>         at java.lang.Thread.run(Thread.java:619)
>
> "LruBlockCache.EvictionThread" daemon prio=10 tid=0x00002aaafc01f800
> nid=0x49e in Object.wait() [0x0000000042d55000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab6519890> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread.run(LruBlockCache.java:512)
>         - locked <0x00002aaab6519890> (a
> org.apache.hadoop.hbase.io.hfile.LruBlockCache$EvictionThread)
>
> "IPC Server handler 9 on 60020" daemon prio=10 tid=0x00002aaaf7c12000
> nid=0x49d waiting on condition [0x0000000042c54000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 8 on 60020" daemon prio=10 tid=0x00002aaaf7c0f800
> nid=0x49c waiting on condition [0x0000000042b53000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 7 on 60020" daemon prio=10 tid=0x00002aaaf7c0d800
> nid=0x49b waiting on condition [0x0000000042a52000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 6 on 60020" daemon prio=10 tid=0x00002aaafd08b800
> nid=0x49a waiting on condition [0x0000000042951000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 5 on 60020" daemon prio=10 tid=0x00002aaafd089800
> nid=0x499 waiting on condition [0x0000000042850000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 4 on 60020" daemon prio=10 tid=0x00002aaafd087000
> nid=0x498 waiting on condition [0x000000004274f000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 3 on 60020" daemon prio=10 tid=0x00002aaafd085000
> nid=0x497 waiting on condition [0x000000004264e000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 2 on 60020" daemon prio=10 tid=0x00002aaaf7adf800
> nid=0x496 waiting on condition [0x000000004254d000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 1 on 60020" daemon prio=10 tid=0x00002aaaf7adb800
> nid=0x495 waiting on condition [0x000000004244c000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server handler 0 on 60020" daemon prio=10 tid=0x00002aaaf7adb000
> nid=0x494 waiting on condition [0x000000004234b000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434798> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:901)
>
> "IPC Server listener on 60020" daemon prio=10 tid=0x00002aaaf7de9800
> nid=0x493 runnable [0x000000004224a000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         - locked <0x00002aaab643b548> (a sun.nio.ch.Util$1)
>         - locked <0x00002aaab643b530> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00002aaab63ea7e0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:308)
>
> "IPC Server Responder" daemon prio=10 tid=0x00002aaaf7de9000 nid=0x492
> runnable [0x0000000042149000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         - locked <0x00002aaab643b458> (a sun.nio.ch.Util$1)
>         - locked <0x00002aaab643b440> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00002aaab63ea768> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Responder.run(HBaseServer.java:477)
>
> "regionserver/10.244.9.171:60020.safeMode" daemon prio=10
> tid=0x00002aaaf7c84000 nid=0x491 in Object.wait() [0x0000000042048000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab63911f8> (a
> java.util.concurrent.atomic.AtomicBoolean)
>         at java.lang.Object.wait(Object.java:485)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer$CompactionLimitThread.run(HRegionServer.java:902)
>         - locked <0x00002aaab63911f8> (a
> java.util.concurrent.atomic.AtomicBoolean)
>
> "Timer-0" daemon prio=10 tid=0x00002aaaf7de4000 nid=0x490 in Object.wait()
> [0x0000000041f47000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab6558e60> (a java.util.TaskQueue)
>         at java.util.TimerThread.mainLoop(Timer.java:509)
>         - locked <0x00002aaab6558e60> (a java.util.TaskQueue)
>         at java.util.TimerThread.run(Timer.java:462)
>
> "853527367@qtp0-1 - Acceptor0 SelectChannelConnector@0.0.0.0:60030"
> prio=10 tid=0x00002aaafcd3a800 nid=0x48f runnable [0x0000000041e46000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         - locked <0x00002aaab651bf38> (a sun.nio.ch.Util$1)
>         - locked <0x00002aaab651bf50> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00002aaab6620c60> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:429)
>         at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:185)
>         at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
>         at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:707)
>         at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> "1096304647@qtp0-0" prio=10 tid=0x00002aaaf7d5b800 nid=0x48e in
> Object.wait() [0x0000000041d45000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab653acd0> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>         at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:565)
>         - locked <0x00002aaab653acd0> (a
> org.mortbay.thread.QueuedThreadPool$PoolThread)
>
> "regionserver/10.244.9.171:60020.leaseChecker" prio=10
> tid=0x00002aaafc16a000 nid=0x48d waiting on condition [0x0000000041c44000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6434518> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>         at java.util.concurrent.DelayQueue.poll(DelayQueue.java:209)
>         at org.apache.hadoop.hbase.Leases.run(Leases.java:81)
>
> "regionserver/10.244.9.171:60020.majorCompactionChecker" daemon prio=10
> tid=0x00002aaafc168000 nid=0x48c waiting on condition [0x0000000041b43000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
>         at org.apache.hadoop.hbase.Chore.run(Chore.java:76)
>
> "regionserver/10.244.9.171:60020.worker" daemon prio=10
> tid=0x00002aaafc228000 nid=0x48b waiting on condition [0x0000000041a42000]
>    java.lang.Thread.State: TIMED_WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab63a3b68> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963)
>         at
> java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:395)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1459)
>         at java.lang.Thread.run(Thread.java:619)
>
> "regionserver/10.244.9.171:60020.compactor" daemon prio=10
> tid=0x00002aaafc226000 nid=0x48a waiting on condition [0x0000000041941000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:79)
>
> "regionserver/10.244.9.171:60020.cacheFlusher" daemon prio=10
> tid=0x00002aaafc225000 nid=0x489 waiting on condition [0x0000000041840000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:137)
>
> "regionserver/10.244.9.171:60020.logFlusher" daemon prio=10
> tid=0x00002aaafc212000 nid=0x488 waiting on condition [0x000000004173f000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
>         at org.apache.hadoop.hbase.Chore.run(Chore.java:76)
>
> "regionserver/10.244.9.171:60020.logRoller" daemon prio=10
> tid=0x00002aaafc211800 nid=0x487 in Object.wait() [0x000000004153d000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab63913f0> (a
> java.util.concurrent.atomic.AtomicBoolean)
>         at
> org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:65)
>         - locked <0x00002aaab63913f0> (a
> java.util.concurrent.atomic.AtomicBoolean)
>
> "MultiThreadedHttpConnectionManager cleanup" daemon prio=10
> tid=0x00002aaaf7e14000 nid=0x486 in Object.wait() [0x000000004163e000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab65446e0> (a
> java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00002aaab65446e0> (a java.lang.ref.ReferenceQueue$Lock)
>         at
> org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ReferenceQueueThread.run(MultiThreadedHttpConnectionManager.java:1082)
>
> "IPC Client (47) connection to /10.244.9.171:60000 from an unknown user"
> daemon prio=10 tid=0x00002aaaf80b6800 nid=0x46d in Object.wait()
> [0x000000004143c000]
>    java.lang.Thread.State: TIMED_WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab642ba08> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:405)
>         - locked <0x00002aaab642ba08> (a
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection)
>         at
> org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:448)
>
> "DestroyJavaVM" prio=10 tid=0x00002aaaf80b4800 nid=0x455 waiting on
> condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "regionserver/10.244.9.171:60020" prio=10 tid=0x00002aaaf80b3800 nid=0x467
> waiting on condition [0x000000004133b000]
>    java.lang.Thread.State: TIMED_WAITING (sleeping)
>         at java.lang.Thread.sleep(Native Method)
>         at org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:74)
>         at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:596)
>         at java.lang.Thread.run(Thread.java:619)
>
> "main-EventThread" daemon prio=10 tid=0x00002aaaf80ab800 nid=0x466 waiting
> on condition [0x000000004123a000]
>    java.lang.Thread.State: WAITING (parking)
>         at sun.misc.Unsafe.park(Native Method)
>         - parking to wait for  <0x00002aaab6391598> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>         at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>         at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
>         at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
>         at
> org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:376)
>
> "main-SendThread" daemon prio=10 tid=0x00002aaaf80ab000 nid=0x465 runnable
> [0x0000000041139000]
>    java.lang.Thread.State: RUNNABLE
>         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
>         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:215)
>         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65)
>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
>         - locked <0x00002aaab6418908> (a sun.nio.ch.Util$1)
>         - locked <0x00002aaab64188f0> (a
> java.util.Collections$UnmodifiableSet)
>         - locked <0x00002aaab63fb0a0> (a sun.nio.ch.EPollSelectorImpl)
>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
>         at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:873)
>
> "Low Memory Detector" daemon prio=10 tid=0x00002aaaf7858800 nid=0x463
> runnable [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread1" daemon prio=10 tid=0x00002aaaf7856000 nid=0x462 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "CompilerThread0" daemon prio=10 tid=0x00002aaaf7852800 nid=0x461 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" daemon prio=10 tid=0x00002aaaf76b9800 nid=0x460 waiting
> on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Surrogate Locker Thread (CMS)" daemon prio=10 tid=0x00002aaaf76b7800
> nid=0x45f waiting on condition [0x0000000000000000]
>    java.lang.Thread.State: RUNNABLE
>
> "Finalizer" daemon prio=10 tid=0x00002aaaf7695800 nid=0x45e in
> Object.wait() [0x0000000040a32000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab6396438> (a
> java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
>         - locked <0x00002aaab6396438> (a java.lang.ref.ReferenceQueue$Lock)
>         at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
>         at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> "Reference Handler" daemon prio=10 tid=0x00002aaaf7693800 nid=0x45c in
> Object.wait() [0x0000000040931000]
>    java.lang.Thread.State: WAITING (on object monitor)
>         at java.lang.Object.wait(Native Method)
>         - waiting on <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock)
>         at java.lang.Object.wait(Object.java:485)
>         at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
>         - locked <0x00002aaab63910d8> (a java.lang.ref.Reference$Lock)
>
> "VM Thread" prio=10 tid=0x00002aaaf768d000 nid=0x45b runnable
>
> "Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x0000000040119000
> nid=0x456 runnable
>
> "Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x000000004011b000
> nid=0x457 runnable
>
> "Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x000000004011c800
> nid=0x458 runnable
>
> "Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x000000004011e800
> nid=0x459 runnable
>
> "Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00002aaaf3d9d800 nid=0x45a
> runnable
> "VM Periodic Task Thread" prio=10 tid=0x00002aaaf785b000 nid=0x464 waiting
> on condition
>
> JNI global references: 806
>
> Heap
>  par new generation   total 19136K, used 14022K [0x00002aaaafc40000,
> 0x00002aaab1100000, 0x00002aaab4f70000)
>   eden space 17024K,  81% used [0x00002aaaafc40000, 0x00002aaab09ce548,
> 0x00002aaab0ce0000)
>   from space 2112K,   6% used [0x00002aaab0ef0000, 0x00002aaab0f13470,
> 0x00002aaab1100000)
>   to   space 2112K,   0% used [0x00002aaab0ce0000, 0x00002aaab0ce0000,
> 0x00002aaab0ef0000)
>  concurrent mark-sweep generation total 63872K, used 25097K
> [0x00002aaab4f70000, 0x00002aaab8dd0000, 0x00002aaaee440000)
>  concurrent-mark-sweep perm gen total 30220K, used 18263K
> [0x00002aaaee440000, 0x00002aaaf01c3000, 0x00002aaaf3840000)
>
>
>
> Ananth T Sarathy
>
>
>
> On Wed, Oct 21, 2009 at 1:24 PM, stack <stack@duboce.net> wrote:
>
>> If thats all you get, a thread dump on 10.242.71.191:60020 would help ($
>> kill -QUIT RS_PID).  Thread dump will be in the RS .out file.  Do a few.
>> Paste to pastebin.
>> Thanks Ananth,
>> St.Ack
>>
>> On Wed, Oct 21, 2009 at 8:05 AM, Ananth T. Sarathy <
>> ananth.t.sarathy@gmail.com> wrote:
>>
>> > yeah,
>> >  I just don't understand why getScanner("Column:X") returns the iterator
>> > and
>> > process them yet getScanner("Column:Y") just spins and spins, yet
>>  Column:Y
>> > is a much denser result.
>> >
>> > When I load from shell
>> >
>> > *Version: 0.20.0, r810752, Thu Sep  3 00:06:18 PDT 2009
>> > hbase(main):001:0> count 'GS_Applications'
>> > 09/10/21 11:04:48 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
>> > /hbase/root-region-server got 10.244.9.171:60020
>> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$ClientZKWatcher: Got
>> > ZooKeeper event, state: SyncConnected, type: None, path: null
>> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Found
>> ROOT
>> > at 10.244.9.171:60020
>> > 09/10/21 11:04:48 DEBUG client.HConnectionManager$TableServers: Cached
>> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME =>
>> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
>> =>
>> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
>> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
>> =>
>> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
>> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
>> =>
>> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
>> > BLOCKCACHE => 'false'}]}}
>> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached
>> > location address: 10.242.71.191:60020, regioninfo: REGION => {NAME =>
>> > 'GS_Applications,,1255020109210', STARTKEY => '', ENDKEY => '', ENCODED
>> =>
>> > 1732076772, TABLE => {{NAME => 'GS_Applications', FAMILIES => [{NAME
=>
>> > 'Application', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
>> '2147483647',
>> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME
>> =>
>> > 'BinaryData', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
>> '2147483647',
>> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'},
{NAME
>> =>
>> > 'BinaryRetrieval', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
>> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
>> > 'true'}, {NAME => 'Files', COMPRESSION => 'NONE', VERSIONS => '1',
TTL
>> =>
>> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
>> > 'true'}, {NAME => 'Info', COMPRESSION => 'NONE', VERSIONS => '1', TTL
=>
>> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
>> > 'true'}, {NAME => 'Network', COMPRESSION => 'NONE', VERSIONS => '1',
TTL
>> =>
>> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
>> > 'true'}, {NAME => 'Registry', VERSIONS => '1', COMPRESSION => 'NONE',
>> TTL
>> > =>
>> > '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
>> > 'true'}]}}
>> > 09/10/21 11:04:49 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
>> > /hbase/root-region-server got 10.244.9.171:60020
>> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Found
>> ROOT
>> > at 10.244.9.171:60020
>> > 09/10/21 11:04:49 DEBUG client.HConnectionManager$TableServers: Cached
>> > location address: 10.245.82.160:60020, regioninfo: REGION => {NAME =>
>> > '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE
>> =>
>> > {{NAME => '.META.', IS_META => 'true', MEMSTORE_FLUSHSIZE => '16384',
>> > FAMILIES => [{NAME => 'historian', VERSIONS => '2147483647', COMPRESSION
>> =>
>> > 'NONE', TTL => '604800', BLOCKSIZE => '8192', IN_MEMORY => 'false',
>> > BLOCKCACHE => 'false'}, {NAME => 'info', VERSIONS => '10', COMPRESSION
>> =>
>> > 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'false',
>> > BLOCKCACHE => 'false'}]}}
>> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Creating scanner
>> over
>> > GS_Applications starting at key ''
>> > 09/10/21 11:04:50 DEBUG client.HTable$ClientScanner: Advancing internal
>> > scanner to startKey at ''
>> > 09/10/21 11:04:50 DEBUG client.HConnectionManager$TableServers: Cache
>> hit
>> > for row <> in tableName GS_Applications: location server
>> > 10.242.71.191:60020,
>> > location region name GS_Applications,,1255020109210
>> >
>> >
>> > *Ananth T Sarathy
>> >
>> >
>> > On Wed, Oct 21, 2009 at 11:00 AM, stack <stack@duboce.net> wrote:
>> >
>> > > In both cases you are doing a full table scan?
>> > >
>> > > Try from shell with DEBUG enable.  You'll see the regions being
>> loaded.
>> > >  May
>> > > help you narrow in on problem region or at least on problem
>> regionserver.
>> > >
>> > > St.Ack
>> > >
>> > > On Wed, Oct 21, 2009 at 7:19 AM, Ananth T. Sarathy <
>> > > ananth.t.sarathy@gmail.com> wrote:
>> > >
>> > > > Anyone have any further thoughts on this?
>> > > > Ananth T Sarathy
>> > > >
>> > > >
>> > > > On Tue, Oct 20, 2009 at 6:37 PM, Ananth T. Sarathy <
>> > > > ananth.t.sarathy@gmail.com> wrote:
>> > > >
>> > > > > Well that's not the case. Every Row has that column.  In fact
the
>> > > second
>> > > > > snippet i sent  is with a column with many less rows. (1k vs
25k)
>> but
>> > > > comes
>> > > > > back pretty quickly.
>> > > > >
>> > > > > By forever, I mean i have watched my logs do nothing for a half
>> hour
>> > > > before
>> > > > > giving up.
>> > > > >
>> > > > >
>> > > > > Ananth T Sarathy
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Tue, Oct 20, 2009 at 5:03 PM, Ryan Rawson <ryanobjc@gmail.com>
>> > > wrote:
>> > > > >
>> > > > >> If you are asking for a column that is very sparse and doesnt
>> exist,
>> > > > >> it will cause HBase to read through the entire region to
find 100
>> > > > >> matching rows. This could take a while, you said 'forever',
but
>> > could
>> > > > >> you quantify that?
>> > > > >>
>> > > > >> On Tue, Oct 20, 2009 at 1:58 PM, Jean-Daniel Cryans <
>> > > > jdcryans@apache.org>
>> > > > >> wrote:
>> > > > >> > Scanner pre-fetching is always faster, so something
must be
>> wrong
>> > > with
>> > > > >> > your region server. Check the logs, top, etc
>> > > > >> >
>> > > > >> > WRT to row size, it's pretty much a matter of how many
bytes
>> you
>> > > have
>> > > > >> > in each column and sum them up (plus some overhead with
the
>> keys).
>> > > > >> >
>> > > > >> > You want filters, check the filter package in the javadoc.
>> > > > >> >
>> > > > >> > J-D
>> > > > >> >
>> > > > >> > On Tue, Oct 20, 2009 at 1:52 PM, Ananth T. Sarathy
>> > > > >> > <ananth.t.sarathy@gmail.com> wrote:
>> > > > >> >> Ok, but how come
>> > > > >> >> when I run a similiar call (with less returned rows
1000 vs
>> 25k
>> > in
>> > > > the
>> > > > >> >> previous one) it runs through the iterator very
quickly?  (See
>> > > Below)
>> > > > >> >>
>> > > > >> >> Also, how do I determine the row size? It's just
text data,
>> and
>> > > > really
>> > > > >> not
>> > > > >> >> much.
>> > > > >> >>
>> > > > >> >> Finally, is there a way to query for rows that do
not have a
>> > > column?
>> > > > >> (Ie all
>> > > > >> >> rows without Files:path1)
>> > > > >> >>
>> > > > >> >>        HBaseTableDataManagerImpl htdmni = new
>> > > > >> HBaseTableDataManagerImpl(
>> > > > >> >>                "GS_Applications");
>> > > > >> >>
>> > > > >> >>        String[] columns = { "Files:path1" };
>> > > > >> >>        log.info("Getting all Rows with Files");
>> > > > >> >>        Scanner s = htdmni.getScannerForAllRows(columns);
>> > > > >> >>        log.info("Got all Rows with Files");
>> > > > >> >>
>> > > > >> >>        Iterator<RowResult> iter = s.iterator();
>> > > > >> >>        out
>> > > > >> >>
>> > > > >> >>
>> > > > >>
>> > > >
>> > >
>> >
>> .write("Application_Full_Name,Version,Application_installer_name,Operating
>> > > > >> >> System, Application_platform
>> > > > >> >>
>> > > > >>
>> > > >
>> > >
>> >
>> ,Application_sub_category,md5Hash,Sha1Hash,Sha256Hash,filepath,fileName,modified,size,operation\n");
>> > > > >> >>        out.write("<BR>");
>> > > > >> >>        while (iter.hasNext())
>> > > > >> >>        {
>> > > > >> >>
>> > > > >> >> Ananth T Sarathy
>> > > > >> >>
>> > > > >> >>
>> > > > >> >> On Tue, Oct 20, 2009 at 4:44 PM, Jean-Daniel Cryans
<
>> > > > >> jdcryans@apache.org>wrote:
>> > > > >> >>
>> > > > >> >>> If you have a very slow data source (S3), then
it fetches 100
>> > row
>> > > > >> >>> before coming back to your client with all of
them and that
>> can
>> > > take
>> > > > a
>> > > > >> >>> lot of time. Also make sure that 100 of your
rows can fit in
>> a
>> > > > region
>> > > > >> >>> server's memory. How big is each row?
>> > > > >> >>>
>> > > > >> >>> J-D
>> > > > >> >>>
>> > > > >> >>> On Tue, Oct 20, 2009 at 1:32 PM, Ananth T. Sarathy
>> > > > >> >>> <ananth.t.sarathy@gmail.com> wrote:
>> > > > >> >>> > I am running this code where
>> > > > >> >>> >
>> > > > >> >>> > getScannerForAllRows(columns) just does
return
>> > > > >> table.getScanner(columns);
>> > > > >> >>> >
>> > > > >> >>> > and the table   has setScannerCaching(100);
>> > > > >> >>> >
>> > > > >> >>> > But it spins forever after getting the
iterator. Why would
>> > that
>> > > > be?
>> > > > >> How
>> > > > >> >>> can
>> > > > >> >>> > I speed it up?
>> > > > >> >>> >
>> > > > >> >>> >        HBaseTableDataManagerImpl htdmni
= new
>> > > > >> HBaseTableDataManagerImpl(
>> > > > >> >>> >                "GS_Applications");
>> > > > >> >>> >
>> > > > >> >>> >        String[] columns = { "Files:Name"
};
>> > > > >> >>> >        log.info("Getting all Rows with
Files");
>> > > > >> >>> >        Scanner s = htdmni.getScannerForAllRows(columns);
>> > > > >> >>> >        log.info("Got all Rows with Files");
>> > > > >> >>> >        log.info("Getting Iterator");
>> > > > >> >>> >
>> > > > >> >>> >        Iterator<RowResult> iter =
s.iterator();
>> > > > >> >>> >        log.info("Got Iterator");
>> > > > >> >>> >
>> > > > >> >>> >        while (iter.hasNext())
>> > > > >> >>> >        {
>> > > > >> >>> >            log.info("Getting next Row");
>> > > > >> >>> >            RowResult rr = iter.next();
>> > > > >> >>> >
>> > > > >> >>> >
>> > > > >> >>> > Ananth T Sarathy
>> > > > >> >>> >
>> > > > >> >>>
>> > > > >> >>
>> > > > >> >
>> > > > >>
>> > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message