hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Very long time between node failure and reasing of regions.
Date Tue, 27 Apr 2010 18:48:55 GMT
Hi Michal,

Thanks for the details on your versions, etc.

The 0.20.5 branch (and the Cloudera HDFS CDH3 dev build branch) are both
very much works in progress. My guess is that you're running into HBASE-2482
which is causing your regions to go missing, etc. There are also several
known bugs in the HDFS patch series that you're using - I'm still working
through one or two of them and will update the tarball again when they're
fixed. For now, I don't think the logs will be particularly helpful, as they
most likely will just point to issues I've either already fixed locally or
am aware of and working out.

Regarding your uptime requirements, my personal recommendation is that, if
you want something like 5 nines of uptime, you will need to run two entirely
distinct HBase clusters. Hadoop does not support rolling upgrades, and HBase
only supports such between certain releases. So, your goals of running
bleeding edge software with patches and updates are at odds with your goals
of uptime.

Obviously, HA is an important thing for a lot of use cases like yours, and
it is something the Hadoop community at large is working towards over the
next year (eg the BackupNode in Hadoop trunk, Avro work for RPC
compatibility, etc).


2010/4/27 Michał Podsiadłowski <podsiadlowski@gmail.com>

> > Hi Michal,
> >
> > I agree that this use case should not cause the cluster to fail. By "just
> > shutting down" do you mean you are running hbase-daemon.sh stop
> regionserver
> > on 3 of the nodes? Are you doing all three at once or in quick
> succession?
> > I'd like to try to reproduce your problem so we can get it fixed for
> 0.20.5.
> >
> > Thanks
> > -Todd
> >
> Todd, James,
> I'm in the office now so I can tell you more about our cluster: exact
> version of hbase is 0.20.4-dev, r926930
> hadoop underneath is 0.20.2+228 + patches from cloudera
> hbase-patches-0b8ca9b5.tar.gz 22-Mar-2010 17:45   33K
> as far as i can see there is already something new available.
> We shutdown nodes one by one by moving from one shell to the other. We
> are doing this just like you wrote but using cloudera wrapping
> scripts. So in theory everything should be stable and there shouldn't
> be any data loss. Practice shows ...
> In your environment we are using this for L2 web cache layer with mem
> based layer in each applications server. We have plans for more use
> cases - storing some of users data there instead of MySql. But of
> course we need to by sure about it's stability and availability which
> are by now dubious for us. This implies that hbase must be available
> for most of the times otherwise we can't store invalidations to data
> which were changed by users requests. If hbase is not available we
> still can generate page which sometimes is very expensive or if
> available in mem cache layer serve it from there. Posting request can
> be rejected for short time but short is not few minutes. The worst
> scenario is when hbase connections hangs and threadpool gets filled up
> as i wrote earlier.
> In respect to what James wrote about not touching anything when
> working. This is not the idea that we want to use. First of all we can
> stop all request to hbase and do maintenance then because we need to
> clear all the date then because invalidations were not saved to it.
> Alternative is to stop whole site which is impermissible from
> marketing point of view and also whole thing contradict with HA idea.
> One of the base assumptions of hadoop is to utilize commodity hardware
> that is likely to fail when there is more then few nodes. There is
> also need for doing updates of hadoop and hbase software because it's
> a bleeding edge of this technology and updates are released quite
> frequently. There are also JVM updates, kernel updates etc. so system
> must be resistant to loss of one of the nodes. (We already manage to
> make disaster with only one node see my earlier post on group).
> Todd if you need more logs drop me an email. I can provide you will
> all logs from hbase and hadoop.
> Thanks,
> Michal.

Todd Lipcon
Software Engineer, Cloudera

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message