hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Xing" <xing...@gmail.com>
Subject Re: out of memory error
Date Fri, 17 Oct 2008 03:40:20 GMT
Sorry for the missing of necessary information.

The hardware configuration is 16G mem/8 CPUs. The cluster is 5 datanodes and
1 name nodes. The version of Hadoop/HBase is hadoop-0.17.1/hbase-0.2.1. We
allocated 65535 file descriptors and the specific changes made to config
files are:

In hadoop-env.sh.
export HADOOP_HEAPSIZE=1000

In hbase-env.sh
export HADOOP_HEAPSIZE=2000

The data volume we're trying to load is about 40G in compression, that's
200G if uncompressed, and one table with 14 column families is created.  Is
the heap size set up correctly?

Thanks,
-Ray

On Fri, Oct 17, 2008 at 12:30 AM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:

> In the future, you will get a more timely response for hbase
> questions if you post them on the hbase-user@hadoop.apache.org
> mailing list.
>
> In order to address your question, it would be helpful to
> know your hardware configuration (memory, # of cores),
> any changes you have made to hbase-site.xml, how many
> file handles are allocated per process, what else is
> running on the same machine as the region server and
> what versions of hadoop and hbase you are running.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
> > -----Original Message-----
> > From: Rui Xing [mailto:xingrui@gmail.com]
> > Sent: Thursday, October 16, 2008 4:52 AM
> > To: core-user@hadoop.apache.org
> > Subject: out of memory error
> >
> > Hello List,
> >
> > We encountered an out-of-memory error in data loading. We have 5 data
> > nodes
> > and 1 name node distributed on 6 machines. Block-level compression was
> > used.
> > Following is the log output. Seems the problem was caused in compression.
> > Is
> > there anybody who ever experienced such error? Any helps or clues are
> > appreciated.
> >
> > 2008-10-15 21:44:33,069 FATAL
> > [regionserver/0:0:0:0:0:0:0:0:60020.compactor]
> > regionserver.HRegionServer$1(579): Set stop flag in
> > regionserver/0:0:0:0:0:0:0:0:60020.compactor
> > java.lang.OutOfMemoryError
> >         at sun.misc.Unsafe.allocateMemory(Native Method)
> >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
> >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:108)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:115)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> > tory.java:104)
> >         at
> >
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> > .java:80)
> >         at
> >
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> > ceFile.java:1458)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1543)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> >         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> > init>(HStoreFile.java:635)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> > <init>(HStoreFile.java:717)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HalfMapFileReader.<init>(H
> > StoreFile.java:915)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> > 408)
> >         at
> > org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:263)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.jav
> > a:1698)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:481)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:421)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:815)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplit
> > Thread.java:133)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitTh
> > read.java:86)
> > 2008-10-15 21:44:33,661 FATAL
> > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> > regionserver.Flusher(183):
> > Replay of hlog required. Forcing server restart
> > org.apache.hadoop.hbase.DroppedSnapshotException: region:
> > p4p_test,,1224072139042
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> > va:1087)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:985)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
> >         at
> > org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
> > Caused by: java.lang.OutOfMemoryError
> >         at sun.misc.Unsafe.allocateMemory(Native Method)
> >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
> >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:107)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:115)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> > tory.java:104)
> >         at
> >
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> > .java:80)
> >         at
> >
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> > ceFile.java:1458)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> >         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> > init>(HStoreFile.java:635)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> > <init>(HStoreFile.java:717)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> > 413)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:665)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java
> > :640)
> >         at
> > org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> > va:1074)
> >         ... 3 more
> > 2008-10-15 21:44:33,661 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> > regionserver.Flusher(109):
> > regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
> > 2008-10-15 21:44:33,665 DEBUG
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.HLog(236): Closing current log writer
> > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078265898
> > 2008-10-15 21:44:33,667 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.HLog(249): New log writer created at
> > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078273665
> > 2008-10-15 21:44:33,667 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.LogRoller(93): LogRoller exiting.
> > 2008-10-15 21:44:34,910 DEBUG [regionserver/0:0:0:0:0:0:0:0:60020]
> > hbase.RegionHistorian(316): Offlined
> > 2008-10-15 21:44:34,911 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> > regionserver.HRegionServer(432): Stopping infoServer
> > 2008-10-15 21:44:34,911 INFO  [Acceptor ServerSocket[addr=
> > 0.0.0.0/0.0.0.0,port=0,localport=60030]]
> > util.ThreadedServer$Acceptor(656):
> > Stopping Acceptor
> > ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
> > 2008-10-15 21:44:34,914 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> > http.SocketListener(212): Stopped SocketListener on 0.0.0.0:60030
> >
> > Thanks,
> > -Ray
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message