hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: Region Servers going down frequently
Date Tue, 07 Apr 2009 17:36:47 GMT

Hi Rakhi,

The "cannot obtain block" error is actually a HDFS problem. Most
likely this block was lost by HDFS during a period of excessive 
load. Usually the first sign you are using insufficient 
resources for your load is filesystem issues such as these. To
address the problems I recommend you do two things at once.

1) The minimum usable instance type for HBase (and Hadoop) is
large in my opinion. The basic rule of thumb for HBase and
Hadoop daemons is you must allocate 1GB of heap/RAM and one
CPU (or vcpu) thread for each daemon. You can search the
hbase-user@ archives for previous discussion on this topic.

2) Allocate more instances to spread the load on DFS. 

On EC2 I recommend running storage such as HDFS/HBase on one set
of instances and mapreduce computations on another set. Hadoop
and HBase daemons are sensitive to thread starvation problems.

Hope this helps,

   - Andy

> From: Rakhi Khatwani
> Subject: Region Servers going down frequently
> Date: Tuesday, April 7, 2009, 2:45 AM
> Hi,
>       I have a 20 node cluster on ec2(small instance).... i
> have a set of tables which store huge amount of data (tried
> wid 10,000 rows... more to be added).... but during my map
> reduce jobs, some of the region servers shut
> down thereby causing data loss, stop in my program
> execution and infact one of my tables got damaged. when ever
> i scan the table, i get the could not obtain block error.


View raw message