hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Slava Gorelik" <slava.gore...@gmail.com>
Subject Re: Regionserver fails to serve region
Date Tue, 04 Nov 2008 19:45:12 GMT
Hi.Regarding the failure of new block creation - i failed to run hbase till
i reformatted HDFS again.
I just wandering if hadoop re balancing is necessary? Will it balance itself
? As i understand hadoop balancer is moving data between data nodes, but in
my case this is during massive (8 clients just adding a records - about 400
requests for all region servers - 6). So, is it good idea to run
balancer during heavy load ?

Best Regards.


On Tue, Nov 4, 2008 at 9:33 PM, stack <stack@duboce.net> wrote:

> Slava Gorelik wrote:
>
>> Hi.I happened yesterday after 28 hours of running i started the hadoop
>> balancer after 2 hours working (with some exception- ca't move block) my
>> hbase started to throws exception that can't create new block. It happened
>> on on couple of region servers and then master is failed to connect to
>> then
>> and eventually it crashed. Tomorrow I can cut last 2-3 hours from logs
>> (they
>> are huge) and send you.
>>
>>
>
> Sorry about that.  I presumed it safe given that balancer has been around a
> few releases and we're running it here continuously w/o issue.  Did estart
> fix things or are there now missing blocks?
>
>> BTW, some i sent email to list couple of days ago about blockCache
>> parameter
>> on column family descriptor, what is it and how it affect on performance ?
>>
>>
> Sorry, missed it.
>
> The blockcache is client-side caching of pieces of store files.  You can
> set the size of the blocks to cache client-side.  It uses java Soft
> References.  Blocks are evicted on roughly an LRU basis when memory is low.
>  It was added a good while ago by Tom White.
>
> By default it has been off but as of HBASE-953 commit of about a week or so
> ago, after some playing and tuning, the default has been flipped and now
> blockcache is on by default.  Block caching along with other performance
> improvements including rpc fixes and J-D's scanner pre-fetching and batch
> writing, will make the 0.19.0 release run faster than its predecessors in
> many regards.
>
> Some rough benchmarking running our performance test -- keep in mind, this
> is not-very-real-world just a single client going against a single
> regionserver (see wiki for more) -- shows writes running at ~3X speed, scans
> at ~7X, sequential reads at ~2X and random reads anywhere from slower to 2
> to 3 times faster dependent on how well the block cache is helping (or
> hindering).  If the regionserver has more memory, random reads run faster.
>  If not enough, regionserver is just spinning filling cache and random read
> times plummet.  I'll put up some numbers when we come closer to the 0.19.0
> release.
>
> If you do enable block cache, be sure to update your hbase-default.xml.
>  The old block size tends to provoke OOMEs.
> St.Ack
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message