hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dhaval Shah <prince_mithi...@yahoo.co.in>
Subject Re: HBase cluster design
Date Tue, 27 May 2014 00:02:30 GMT
A few things pop out to me on cursory glance:
- You are using CMSIncrementalMode which after a long chain of events has a tendency to result
in the famous Juliet pause of death. Can you try Par New GC instead and see if that helps?
- You should try to reduce the CMSInitiatingOccupancyFraction to avoid a full GC
- Your hbase-env.sh is not setting the Xmx at all. Do you know how much RAM you are giving
to your region servers? It may be too small or too large given your use case and machines
- Your client scanner caching is 1 which may be too small depending on your row sizes. You
can also override that setting in your scan for the MR job
- You only have 2 zookeeper instances which is not at all recommended. Zookeeper needs a quorum
to operate and generally works best with an odd number of zookeeper servers. This probably
isn't related to your crashes but it would help stability if you had 1 or 3 zookeepers
- I am not 100% sure if the version of hbase you are using has mslab enabled. If not you should
enable it.
- You can try increasing/decreasing the amount of RAM you provide to block caches and memstores
to suit your use case. I see that you are using the defaults here

On top of these, when you kick off your MR job to scan HBase you should setCacheBlocks to


 From: Flavio Pompermaier <pompermaier@okkam.it>
To: user@hbase.apache.org; Dhaval Shah <prince_mithibai@yahoo.co.in> 
Sent: Friday, 23 May 2014 3:16 AM
Subject: Re: HBase cluster design

The hardware specs are: 4 nodes with 48g RAM, 24 cores and 1 TB disk each server
Attached my hbase config files. 


On Fri, May 23, 2014 at 3:33 AM, Dhaval Shah <prince_mithibai@yahoo.co.in> wrote:

Can you share your hbase-env.sh and hbase-site.xml? And hardware specs of your cluster?
> From: Flavio Pompermaier <pompermaier@okkam.it>
>To: user@hbase.apache.org
>Sent: Saturday, 17 May 2014 2:49 AM
>Subject: Re: HBase cluster design
>Could you tell me please in detail the parameters you'd like to see so i
>can look for them and learn the important ones?i'm using cloudera, cdh4 in
>one cluster and cdh5 in the other.
>On May 17, 2014 2:48 AM, "prince_mithibai@yahoo.co.in" <
>prince_mithibai@yahoo.co.in> wrote:
>> Can you describe your setup in more detail? Specifically the amount of
>> heap hbase region servers have and your GC settings. Is your server
>> swapping when your MR obs are running? Also do your regions go down or your
>> region servers?
>> We run many MR jobs simultaneously on our hbase tables (size is in TBs)
>> along with serving real time requests at the same time. So I can vouch for
>> the fact that a well tuned hbase cluster definitely supports this use case
>> (well-tuned is the key word here)
>> Sent from Yahoo Mail on Android
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message