hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: HBase stability
Date Mon, 13 Dec 2010 16:00:51 GMT
HI Anze,

In word, yes - 0.20.4 is not that stable in my experience, and
upgrading to the latest CDH3 beta (which includes HBase 0.89.20100924)
should give you a huge improvement in stability.

You'll still need to do a bit of tuning of settings, but once it's
well tuned it should be able to hold up under load without crashing.

-Todd

On Mon, Dec 13, 2010 at 2:41 AM, Anze <anzenews@volja.net> wrote:
> Hi all!
>
> We have been using HBase 0.20.4 (cdh3b1) in production on 2 nodes for a few
> months now and we are having constant issues with it. We fell over all
> standard traps (like "Too many open files", network configuration
> problems,...). All in all, we had about one crash every week or so.
> Fortunately we are still using it just for background processing so our
> service didn't suffer directly, but we have lost huge amounts of time just
> fixing the data errors that resulted from data not being written to permanent
> storage. Not to mention fixing the issues.
> As you can probably understand, we are very frustrated with this and are
> seriously considering moving to another bigtable.
>
> Right now, HBase crashes whenever we run very intensive rebuild of secondary
> index (normal table, but we use it as secondary index) to a huge table. I have
> found this:
> http://wiki.apache.org/hadoop/Hbase/Troubleshooting
> (see problem 9)
> One of the lines read:
> "Make sure you give plenty of RAM (in hbase-env.sh), the default of 1GB won't
> be able to sustain long running imports."
>
> So, if I understand correctly, no matter how HBase is set up, if I run an
> intensive enough application, it will choke? I would expect it to be slower
> when under (too much) pressure, but not to crash.
>
> Of course, we will somehow solve this issue (working on it), but... :(
>
> What are your experiences with HBase? Is it stable? Is it just us and the way
> we set it up?
>
> Also, would upgrading to 0.89 (cdh3b3) help?
>
> Thanks,
>
> Anze
>
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message