hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Henry Hung <YTHu...@winbond.com>
Subject RE: HBase cluster design
Date Tue, 27 May 2014 00:54:28 GMT
One thing you should consider: is CPU over saturated?

For instance, if one server has 24 cores, and the task tracker is configured to also execute
24 MR simultaneously, then there is no core left for HBase to do GC, and then crash.

Few weeks ago my region server sometimes crash when MR is running, then I decide to move MR
cluster to another machine leaving only DataNode + RegionServer running.
After change, my regionserver still running without crash.

My suggestion is you could try to decrease the MR task numbers to around 12 or lesser, and
see if the frequency of crash decrease?

Best regards,
Henry Hung

-----Original Message-----
From: Dhaval Shah [mailto:prince_mithibai@yahoo.co.in]
Sent: Tuesday, May 27, 2014 8:03 AM
To: user@hbase.apache.org
Subject: Re: HBase cluster design

A few things pop out to me on cursory glance:
- You are using CMSIncrementalMode which after a long chain of events has a tendency to result
in the famous Juliet pause of death. Can you try Par New GC instead and see if that helps?
- You should try to reduce the CMSInitiatingOccupancyFraction to avoid a full GC
- Your hbase-env.sh is not setting the Xmx at all. Do you know how much RAM you are giving
to your region servers? It may be too small or too large given your use case and machines
- Your client scanner caching is 1 which may be too small depending on your row sizes. You
can also override that setting in your scan for the MR job
- You only have 2 zookeeper instances which is not at all recommended. Zookeeper needs a quorum
to operate and generally works best with an odd number of zookeeper servers. This probably
isn't related to your crashes but it would help stability if you had 1 or 3 zookeepers
- I am not 100% sure if the version of hbase you are using has mslab enabled. If not you should
enable it.
- You can try increasing/decreasing the amount of RAM you provide to block caches and memstores
to suit your use case. I see that you are using the defaults here

On top of these, when you kick off your MR job to scan HBase you should setCacheBlocks to


 From: Flavio Pompermaier <pompermaier@okkam.it>
To: user@hbase.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in>
Sent: Friday, 23 May 2014 3:16 AM
Subject: Re: HBase cluster design

The hardware specs are: 4 nodes with 48g RAM, 24 cores and 1 TB disk each server
Attached my hbase config files.


On Fri, May 23, 2014 at 3:33 AM, Dhaval Shah <prince_mithibai@yahoo.co.in> wrote:

Can you share your hbase-env.sh and hbase-site.xml? And hardware specs of your cluster?
> From: Flavio Pompermaier <pompermaier@okkam.it>
>To: user@hbase.apache.org
>Sent: Saturday, 17 May 2014 2:49 AM
>Subject: Re: HBase cluster design
>Could you tell me please in detail the parameters you'd like to see so i
>can look for them and learn the important ones?i'm using cloudera, cdh4 in
>one cluster and cdh5 in the other.
>On May 17, 2014 2:48 AM, "prince_mithibai@yahoo.co.in" <
>prince_mithibai@yahoo.co.in> wrote:
>> Can you describe your setup in more detail? Specifically the amount of
>> heap hbase region servers have and your GC settings. Is your server
>> swapping when your MR obs are running? Also do your regions go down or your
>> region servers?
>> We run many MR jobs simultaneously on our hbase tables (size is in TBs)
>> along with serving real time requests at the same time. So I can vouch for
>> the fact that a well tuned hbase cluster definitely supports this use case
>> (well-tuned is the key word here)
>> Sent from Yahoo Mail on Android

The privileged confidential information contained in this email is intended for use only by
the addressees as indicated by the original sender of this email. If you are not the addressee
indicated in this email or are not responsible for delivery of the email to such a person,
please kindly reply to the sender indicating this fact and delete all copies of it from your
computer and network server immediately. Your cooperation is highly appreciated. It is advised
that any unauthorized use of confidential information of Winbond is strictly prohibited; and
any information in this email irrelevant to the official business of Winbond shall be deemed
as neither given nor endorsed by Winbond.

View raw message