hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Geoff Hendrey" <ghend...@decarta.com>
Subject RE: Region server goes away
Date Thu, 15 Apr 2010 17:23:05 GMT
I have a 12-server mapreduce cluster. It is a totally different set of
servers from the two servers that are hosting my Hbase cluster. In the
12-server mapreduce cluster, we easily run 200 concurrent tasks
distributed over the 12 boxes.

-g 

-----Original Message-----
From: Todd Lipcon [mailto:todd@cloudera.com] 
Sent: Wednesday, April 14, 2010 11:19 PM
To: hbase-user@hadoop.apache.org
Cc: Paul Mahon; Bill Brune; Shaheen Bahauddin; Rohit Nigam
Subject: Re: Region server goes away

On Wed, Apr 14, 2010 at 9:20 PM, Geoff Hendrey <ghendrey@decarta.com>
wrote:

> Thanks for your help. See answers below.
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of 
> Stack
> Sent: Wednesday, April 14, 2010 8:45 PM
> To: hbase-user@hadoop.apache.org
> Cc: Paul Mahon; Bill Brune; Shaheen Bahauddin; Rohit Nigam
> Subject: Re: Region server goes away
>
> On Wed, Apr 14, 2010 at 8:27 PM, Geoff Hendrey <ghendrey@decarta.com>
> wrote:
> > Hi,
> >
> > I have posted previously about issues I was having with HDFS when I 
> > was running HBase and HDFS on the same box both pseudoclustered. Now

> > I have two very capable servers. I've setup HDFS with a datanode on 
> > each
> box.
> > I've setup the namenode on one box, and the zookeeper and HDFS 
> > master on the other box. Both boxes are region servers. I am using 
> > hadoop
> > 20.2 and hbase 20.3.
>
> What do you have for replication?  If two datanodes, you've set it to 
> two rather than default 3?
> Geoff: I didn't change the default, so it was 3. I will change it to 2

> moving forward. Actually, for now I am going to make it 1. For initial

> test runs I don't see why I need replication at all.
>
>
> >
> > I have set dfs.datanode.socket.write.timeout to 0 in hbase-site.xml.
> >
> This is probably not necessary.
>
>
> > I am running a mapreduce job with about 200 concurrent reducers, 
> > each of which writes into HBase, with 32,000 row flush buffers.
>

 Do you really mean 200 concurrent reducers?? That is to say 100
reducers per box? I would recommend that only if you have a 100+ core
machine... not likely.

FYI typical values for reduce slots on dual quad core Nehalem with
hyperthreading (ie 16 logical cores) are in the range of 8-10, not 100!

-Todd


--
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message