hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jane Tao <jiao....@oracle.com>
Subject BucketCache Configuration
Date Fri, 18 Jul 2014 23:46:09 GMT
Hi there,

Our goal is to fully utilize the free RAM on each node/region server for 
HBase. At the same time, we do not want to incur too much pressure from 
GC (garbage collection). Based on Ted's sugguestion, we are trying to 
using bucket cache.

However, we are not sure:
- The relation between XX:MaxDirectMemorySize and java heap size. Is 
MaxDirectMemorySize part of java heap size ?
- The relation between XX:MaxDirectMemorySize and 
hbase.bucketcache.size. Are they equal?
- How to adjust hbase.bucketcache.percentage.in.combinedcache?

Right now, we have the following configuration. Does it make sense?

- java heap size of each hbase region server to 12 GB
- -XX:MaxDirectMemorySize to be 6GB
- hbase-site.xml :
   <property>
     <name>hbase.offheapcache.percentage</name>
     <value>0</value>
   </property>
   <property>
     <name>hbase.bucketcache.ioengine</name>
     <value>offheap</value>
   </property>
   <property>
<name>hbase.bucketcache.percentage.in.combinedcache</name>
     <value>0.8</value>
   </property>
   <property>
     <name>hbase.bucketcache.size</name>
     <value>6144</value>
   </property>

Thanks,
Jane


On 7/17/2014 3:05 PM, Ted Yu wrote:
> Have you considered using BucketCache ?
>
> Please read 9.6.4.1 under
> http://hbase.apache.org/book.html#regionserver.arch
>
> Note: remember to verify the config values against the hbase release you're
> using.
>
> Cheers
>
>
> On Thu, Jul 17, 2014 at 2:53 PM, Jane Tao <jiao.tao@oracle.com> wrote:
>
>> Hi Ted,
>>
>> In my case, there is a 6 Node HBase cluster setup (running on Oracle BDA).
>> Each node has plenty of RAM (64GB) and CPU cores. Several articles seem to
>> suggest
>> that it is not a good idea to allocate too much RAM to region server's
>> heap setting.
>>
>> If each region server has 10GB heap and there is only one region server
>> per node, then
>> I have 10x6=60GB for the whole HBase. This setting is good for ~100M rows
>> but starts
>> to incur lots of GC activities on region servers when loading billions of
>> rows.
>>
>> Basically, I need a configuration that can fully utilize the free RAM on
>> each node for HBase.
>>
>> Thanks,
>> Jane
>> On 7/16/2014 4:17 PM, Ted Yu wrote:
>>
>>> Jane:
>>> Can you briefly describe the use case where multiple region servers are
>>> needed on the same host ?
>>>
>>> Cheers
>>>
>>>
>>>
>>> On Wed, Jul 16, 2014 at 3:14 PM, Dhaval Shah <prince_mithibai@yahoo.co.in
>>> wrote:
>>>
>>>   Its certainly possible (atleast with command line) but probably very
>>>> messy. You will need to have different ports, different log files,
>>>> different pid files, possibly even different configs on the same machine.
>>>>
>>>>
>>>> Regards,
>>>> Dhaval
>>>>
>>>>
>>>> ________________________________
>>>>    From: Jane Tao <jiao.tao@oracle.com>
>>>> To: user@hbase.apache.org
>>>> Sent: Wednesday, 16 July 2014 6:06 PM
>>>> Subject: multiple region servers at one machine
>>>>
>>>>
>>>> Hi there,
>>>>
>>>> Is it possible to run multiple region servers at one machine/node? If
>>>> this is possible, how to start multiple region servers with command
>>>> lines or cloudera manager?
>>>>
>>>> Thanks,
>>>> Jane
>>>>
>>>>
>>>> --
>>>>
>> --
>>
>>

-- 


Mime
View raw message