hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From manua <agarwal.m...@gmail.com>
Subject Re: Java Client hangs after around 2 min
Date Wed, 07 Jul 2010 18:58:19 GMT

Hi,

I have created a swap space of 1Gb, reduced the heap size to 500Mb and
restarted the hbase. Now, when I run the client, I got the following error, 

2010-07-07 14:51:59,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Started memstore flush for region trades,,1278528603802. Current region
memstore size 64.3m
2010-07-07 14:51:59,697 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished snapshotting, commencing flushing stores
2010-07-07 14:51:59,754 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Flush requested on trades,,1278528603802
2010-07-07 14:51:59,755 FATAL
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog
required. Forcing server shutdown
org.apache.hadoop.hbase.DroppedSnapshotException: region:
trades,,1278528603802
        at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1041)
        at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:896)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:258)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:231)
        at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:154)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
com.hadoop.compression.lzo.LzoCodec
        at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:92)
        at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm.getCompressor(Compression.java:197)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.getCompressingStream(HFile.java:387)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.newBlock(HFile.java:373)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.checkBlockBoundary(HFile.java:344)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:515)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Writer.append(HFile.java:481)
        at
org.apache.hadoop.hbase.regionserver.Store.internalFlushCache(Store.java:545)
        at
org.apache.hadoop.hbase.regionserver.Store.flushCache(Store.java:516)
        at
org.apache.hadoop.hbase.regionserver.Store.access$100(Store.java:88)
        at
org.apache.hadoop.hbase.regionserver.Store$StoreFlusherImpl.flushCache(Store.java:1597)
        at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:1000)
        ... 4 more
Caused by: java.lang.ClassNotFoundException:
com.hadoop.compression.lzo.LzoCodec
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at
org.apache.hadoop.hbase.io.hfile.Compression$Algorithm$1.getCodec(Compression.java:88)
        ... 15 more
2010-07-07 14:51:59,757 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=3, stores=8, storefiles=1, storefileIndexSize=0,
memstoreSize=64, compactionQueueSize=0, usedHeap=88, maxHeap=496,
blockCacheSize=489928, blockCacheFree=103699208, blockCacheCount=1,
blockCacheHitRatio=93, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
2010-07-07 14:51:59,757 INFO
org.apache.hadoop.hbase.regionserver.MemStoreFlusher:
RegionServer:0.cacheFlusher exiting
2010-07-07 14:51:59,757 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 52682, call put([B@40d611,
[Lorg.apache.hadoop.hbase.client.Put;@a632be) from 10.208.101.173:47964:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1773)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2010-07-07 14:52:00,709 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 52682
2010-07-07 14:52:00,710 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 52682: exiting
2010-07-07 14:52:00,710 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server listener on 52682
2010-07-07 14:52:00,711 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC
Server Responder
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 52682: exiting
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 52682: exiting
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 52682: exiting
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 52682: exiting
2010-07-07 14:52:00,711 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 52682: exiting
2010-07-07 14:52:00,712 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 52682: exiting
2010-07-07 14:52:00,712 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 52682: exiting
2010-07-07 14:52:00,712 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 52682: exiting
2010-07-07 14:52:00,712 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 10 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 24 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 23 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 22 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 21 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 20 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 19 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 17 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 16 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 15 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 14 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 13 on 52682: exiting
2010-07-07 14:52:00,722 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 12 on 52682: exiting
2010-07-07 14:52:00,721 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 11 on 52682: exiting
2010-07-07 14:52:00,850 DEBUG org.apache.hadoop.hbase.regionserver.HLog:
closing hlog writer in
file:/home/_hbase_/hbase-manu/hbase/.logs/domU-12-31-39-06-62-43.compute-1.internal,52682,1278528522407
2010-07-07 14:52:00,850 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: On abort, closed hlog
2010-07-07 14:52:00,850 INFO
org.apache.hadoop.hbase.regionserver.CompactSplitThread:
RegionServer:0.compactor exiting
2010-07-07 14:52:00,850 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: closing region .META.,,1


After this, again the client hangs and the client logs are repeated getting
the following message,

2010-07-07 14:52:43,640 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2010-07-07 14:53:41,519 INFO org.apache.hadoop.hbase.master.ServerManager: 0
region servers, 1 dead, average load
NaN[domU-12-31-39-06-62-43.compute-1.internal,52682,1278528522407]
2010-07-07 14:53:43,643 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2010-07-07 14:54:41,526 INFO org.apache.hadoop.hbase.master.ServerManager: 0
region servers, 1 dead, average load
NaN[domU-12-31-39-06-62-43.compute-1.internal,52682,1278528522407]
2010-07-07 14:54:43,650 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2010-07-07 14:55:41,534 INFO org.apache.hadoop.hbase.master.ServerManager: 0
region servers, 1 dead, average load
NaN[domU-12-31-39-06-62-43.compute-1.internal,52682,1278528522407]
2010-07-07 14:55:43,658 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2010-07-07 14:56:41,541 INFO org.apache.hadoop.hbase.master.ServerManager: 0
region servers, 1 dead, average load
NaN[domU-12-31-39-06-62-43.compute-1.internal,52682,1278528522407]
2010-07-07 14:56:43,665 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned

Please let me know your views on this.

Thanks,
Manu
-- 
View this message in context: http://old.nabble.com/Java-Client-hangs-after-around-2-min-tp29097955p29099894.html
Sent from the HBase User mailing list archive at Nabble.com.


Mime
View raw message