hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈加俊 <cjjvict...@gmail.com>
Subject java.lang.OutOfMemoryError: Java heap space
Date Mon, 28 Feb 2011 08:06:43 GMT
My cluster hava 12regionserver and HBase version is 0.20.6 .

AverageLoad: 856.3333333333334

Dead: 0


Live Servers: 12

1-> 192.168.5.152:60020 [requests=86, regions=856, usedHeap=2763, maxHeap=2991]
2-> 192.168.5.146:60020 [requests=48, regions=855, usedHeap=2898, maxHeap=2995]
3-> 192.168.5.147:60020 [requests=1362, regions=856, usedHeap=2748,
maxHeap=2995]
4-> 192.168.5.149:60020 [requests=156, regions=855, usedHeap=2519, maxHeap=2995]
5-> 192.168.5.154:60020 [requests=114, regions=855, usedHeap=2924, maxHeap=2991]
6-> 192.168.5.150:60020 [requests=50, regions=859, usedHeap=2522, maxHeap=2995]
7-> 192.168.5.153:60020 [requests=219, regions=858, usedHeap=2687, maxHeap=2991]
8-> 192.168.5.145:60020 [requests=200, regions=856, usedHeap=2791, maxHeap=2995]
9-> 192.168.5.155:60020 [requests=196, regions=857, usedHeap=2909, maxHeap=2991]
10-> 192.168.5.148:60020 [requests=107, regions=855, usedHeap=2935,
maxHeap=2995]
11-> 192.168.5.157:60020 [requests=74, regions=855, usedHeap=2849, maxHeap=2991]
12-> 192.168.5.156:60020 [requests=53, regions=859, usedHeap=2224, maxHeap=2991]

regions: 10276
Requests Count: 2665


details of regionserver's log when RS shutdown:


2011-02-28 00:24:57,172 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x9b2d886685b60082 to sun.nio.ch.SelectionKeyImpl@60451
3be
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2011-02-28 00:24:59,674 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening
cjjIndexHtml,http://www.jfdaily.com/c/1_1_y.htm
EC032FC7C4FB2BC0E332082636B3B707,1298250567983
java.lang.OutOfMemoryError: Java heap space
        at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
        at
org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1380)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1648)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1368)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:848)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:793)
        at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273)
        at
org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129)
        at
org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410)
        at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221)
        at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636)
        at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1566)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1533)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1453)
        at java.lang.Thread.run(Thread.java:619)
2011-02-28 00:24:57,172 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x9a2dd5d1b61e0019 to sun.nio.ch.SelectionKeyImpl@5e9f4
203
java.io.IOException: TIMED OUT
        at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2011-02-28 00:24:59,696 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
aborting.
java.lang.OutOfMemoryError: Java heap space
        at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
        at
org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1380)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1648)
        at
org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
        at java.io.DataInputStream.readFully(DataInputStream.java:178)
        at java.io.DataInputStream.readFully(DataInputStream.java:152)
        at
org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1368)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:848)
        at
org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:793)
        at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273)
        at
org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129)
        at
org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410)
        at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221)
        at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636)
        at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1566)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1533)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1453)
        at java.lang.Thread.run(Thread.java:619)
2011-02-28 00:24:59,700 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60020, call put([B@69f9aa66,
[Lorg.apache.hadoop.hbase.client.Put;@62244e5) from 192.168.5.179:56884:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1773)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2011-02-28 00:24:59,701 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 18 on 60020, call delete([B@721b4a4f,
row=2011022889990015lmth.12611/0220_1102/3_swen/moc.kjj5.swen//:ptth,
ts=9223372036854775807, families={}) from 192.168.5.161:41682: error:
java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:1975)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
2011-02-28 00:24:59,702 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60020, call put([B@5dabc30f,
[Lorg.apache.hadoop.hbase.client.Put;@37933dd4) from 192.168.5.161:41682:
error: java.io.IOException: Server not running, aborting
java.io.IOException: Server not running, aborting
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
        at
org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1773)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at
org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
        at
org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)

-- 
Thanks & Best regards
jiajun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message