hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 陈加俊 <cjjvict...@gmail.com>
Subject Re: java.lang.OutOfMemoryError: Java heap space
Date Tue, 01 Mar 2011 03:07:34 GMT
I reassigned 4G heap to each RS and add one more RS to the cluster. Now the
cluster is as follows.I noticed the this line :requests=7, regions=813,
usedHeap=3823, maxHeap=3991,Is it dangerous signs?

-------------------------------2011-03-01 11:01:33
Hbase Version: 0.20.6
runing : yes
AverageLoad: 797.0

Dead: 0


Live Servers: 13

1-> 192.168.5.157:60020 [requests=44, regions=812, usedHeap=3082,
maxHeap=3991]
2-> 192.168.5.153:60020 [requests=7, regions=813, usedHeap=3823,
maxHeap=3991]
3-> 192.168.5.148:60020 [requests=63, regions=814, usedHeap=3670,
maxHeap=3995]
4-> 192.168.5.150:60020 [requests=157, regions=816, usedHeap=3185,
maxHeap=3995]
5-> 192.168.5.145:60020 [requests=103, regions=582, usedHeap=2683,
maxHeap=3995]
6-> 192.168.5.156:60020 [requests=36, regions=819, usedHeap=2987,
maxHeap=3991]
7-> 192.168.5.155:60020 [requests=14, regions=812, usedHeap=3458,
maxHeap=3991]
8-> 192.168.5.146:60020 [requests=35, regions=822, usedHeap=2483,
maxHeap=3995]
9-> 192.168.5.152:60020 [requests=10, regions=813, usedHeap=3897,
maxHeap=3991]
10-> 192.168.5.149:60020 [requests=59, regions=821, usedHeap=3377,
maxHeap=3995]
11-> 192.168.5.144:60020 [requests=1155, regions=812, usedHeap=3152,
maxHeap=3995]
12-> 192.168.5.147:60020 [requests=942, regions=813, usedHeap=3411,
maxHeap=3995]
13-> 192.168.5.154:60020 [requests=80, regions=812, usedHeap=2639,
maxHeap=3991]

regions: 10361
Requests Count: 2705
Regions In Transition:

On Tue, Mar 1, 2011 at 1:44 AM, Stack <stack@duboce.net> wrote:

> Please update to 0.90.1.
>
> How much heap have you assigned each server?   It looks like its 3G?
> You have close to 1k regions on each of the nodes.  If you look at
> regionserver UI, at the top, you can see some detail on store files
> and how much of the heap their indices occupy.  How much is it?  Is it
> a significant portion of your heap (See HBASE-3551 for more on what
> I'm talking about)?
>
> Can you give hbase more heap?
>
> St.Ack
>
>
> On Mon, Feb 28, 2011 at 12:06 AM, 陈加俊 <cjjvictory@gmail.com> wrote:
> > My cluster hava 12regionserver and HBase version is 0.20.6 .
> >
> > AverageLoad: 856.3333333333334
> >
> > Dead: 0
> >
> >
> > Live Servers: 12
> >
> > 1-> 192.168.5.152:60020 [requests=86, regions=856, usedHeap=2763,
> maxHeap=2991]
> > 2-> 192.168.5.146:60020 [requests=48, regions=855, usedHeap=2898,
> maxHeap=2995]
> > 3-> 192.168.5.147:60020 [requests=1362, regions=856, usedHeap=2748,
> > maxHeap=2995]
> > 4-> 192.168.5.149:60020 [requests=156, regions=855, usedHeap=2519,
> maxHeap=2995]
> > 5-> 192.168.5.154:60020 [requests=114, regions=855, usedHeap=2924,
> maxHeap=2991]
> > 6-> 192.168.5.150:60020 [requests=50, regions=859, usedHeap=2522,
> maxHeap=2995]
> > 7-> 192.168.5.153:60020 [requests=219, regions=858, usedHeap=2687,
> maxHeap=2991]
> > 8-> 192.168.5.145:60020 [requests=200, regions=856, usedHeap=2791,
> maxHeap=2995]
> > 9-> 192.168.5.155:60020 [requests=196, regions=857, usedHeap=2909,
> maxHeap=2991]
> > 10-> 192.168.5.148:60020 [requests=107, regions=855, usedHeap=2935,
> > maxHeap=2995]
> > 11-> 192.168.5.157:60020 [requests=74, regions=855, usedHeap=2849,
> maxHeap=2991]
> > 12-> 192.168.5.156:60020 [requests=53, regions=859, usedHeap=2224,
> maxHeap=2991]
> >
> > regions: 10276
> > Requests Count: 2665
> >
> >
> > details of regionserver's log when RS shutdown:
> >
> >
> > 2011-02-28 00:24:57,172 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x9b2d886685b60082 to sun.nio.ch.SelectionKeyImpl@60451
> > 3be
> > java.io.IOException: TIMED OUT
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > 2011-02-28 00:24:59,674 ERROR
> > org.apache.hadoop.hbase.regionserver.HRegionServer: Error opening
> > cjjIndexHtml,http://www.jfdaily.com/c/1_1_y.htm
> > EC032FC7C4FB2BC0E332082636B3B707,1298250567983
> > java.lang.OutOfMemoryError: Java heap space
> >        at
> java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1380)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1648)
> >        at
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
> >        at java.io.DataInputStream.readFully(DataInputStream.java:178)
> >        at java.io.DataInputStream.readFully(DataInputStream.java:152)
> >        at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1368)
> >        at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:848)
> >        at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:793)
> >        at
> > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273)
> >        at
> > org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129)
> >        at
> > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410)
> >        at
> org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636)
> >        at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1566)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1533)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1453)
> >        at java.lang.Thread.run(Thread.java:619)
> > 2011-02-28 00:24:57,172 WARN org.apache.zookeeper.ClientCnxn: Exception
> > closing session 0x9a2dd5d1b61e0019 to sun.nio.ch.SelectionKeyImpl@5e9f4
> > 203
> > java.io.IOException: TIMED OUT
> >        at
> > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
> > 2011-02-28 00:24:59,696 FATAL
> > org.apache.hadoop.hbase.regionserver.HRegionServer: OutOfMemoryError,
> > aborting.
> > java.lang.OutOfMemoryError: Java heap space
> >        at
> java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1380)
> >        at
> >
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1648)
> >        at
> > org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1767)
> >        at java.io.DataInputStream.readFully(DataInputStream.java:178)
> >        at java.io.DataInputStream.readFully(DataInputStream.java:152)
> >        at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$FixedFileTrailer.deserialize(HFile.java:1368)
> >        at
> > org.apache.hadoop.hbase.io.hfile.HFile$Reader.readTrailer(HFile.java:848)
> >        at
> >
> org.apache.hadoop.hbase.io.hfile.HFile$Reader.loadFileInfo(HFile.java:793)
> >        at
> > org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:273)
> >        at
> > org.apache.hadoop.hbase.regionserver.StoreFile.<init>(StoreFile.java:129)
> >        at
> > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:410)
> >        at
> org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:221)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:1636)
> >        at
> > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:321)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.instantiateRegion(HRegionServer.java:1566)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.openRegion(HRegionServer.java:1533)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer$Worker.run(HRegionServer.java:1453)
> >        at java.lang.Thread.run(Thread.java:619)
> > 2011-02-28 00:24:59,700 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 8 on 60020, call put([B@69f9aa66,
> > [Lorg.apache.hadoop.hbase.client.Put;@62244e5) from 192.168.5.179:56884:
> > error: java.io.IOException: Server not running, aborting
> > java.io.IOException: Server not running, aborting
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1773)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2011-02-28 00:24:59,701 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 18 on 60020, call delete([B@721b4a4f,
> > row=2011022889990015lmth.12611/0220_1102/3_swen/moc.kjj5.swen//:ptth,
> > ts=9223372036854775807, families={}) from 192.168.5.161:41682: error:
> > java.io.IOException: Server not running, aborting
> > java.io.IOException: Server not running, aborting
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.delete(HRegionServer.java:1975)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> > 2011-02-28 00:24:59,702 INFO org.apache.hadoop.ipc.HBaseServer: IPC
> Server
> > handler 2 on 60020, call put([B@5dabc30f,
> > [Lorg.apache.hadoop.hbase.client.Put;@37933dd4) from 192.168.5.161:41682
> :
> > error: java.io.IOException: Server not running, aborting
> > java.io.IOException: Server not running, aborting
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.checkOpen(HRegionServer.java:2307)
> >        at
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1773)
> >        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >        at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >        at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >        at java.lang.reflect.Method.invoke(Method.java:597)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657)
> >        at
> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915)
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>



-- 
Thanks & Best regards
jiajun

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message