hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yossi Ittach" <yoss...@gmail.com>
Subject HBase keeps crushing after load
Date Mon, 27 Oct 2008 08:32:58 GMT
Hi All

I've been benchmarking HBase for quite sometime now , and it keeps crushing
under high load.

The benchmark scheme is running on a dedicated host (320GB disk , 4 GB ram ,
2 dualcore processors) that runs VMware with 5 virtual stations:
1 - Hadoop NameNode
2 - Hbase Master
3 - Hadoop DataNode
4 - Hbase RegionServer
5 - Hbase RegionServer

The benchmark is a program that loads the HBase with ~ 3.5M files , with avg
size of 5k - total of about 30GB.
HBase has 1 table: *row*=file name , column-family (with just one column) *
column* = content (the file content)

When I'm running a smaller benchmark (350K files ~ 3 GB) it works fine. When
I run the full benchmark - hell breaks loose. After ~1M files , the
regionservers start to report connection problems (region server can't
connect to master , master can't connect to region server , ...) and/or shut
themselves down one after another.

I figured it may be some overload on the host server , but the resources are
fine - a lot of cpu , memory and disk are available (even on peak times) .

I attache part of the log files , if you find it helpful.

Thank you very much!

Vale et me ama

  • Unnamed multipart/mixed (inline, None, 0 bytes)
View raw message