hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: stop-hbase.sh bug or feature?
Date Thu, 03 Mar 2011 01:27:31 GMT
Mis feature, basically a master will tell the regionservers to
'shutdown and flush gracefully' via RPC.

Since we don't ship with any cluster management tools - to make your
life easier we have a 'master tells RS to shutdown' path.  I wouldn't
be against removing it and relying on regular process management (Eg:
ssh $host hbase-daemon.sh stop regionserver which uses kill -QUIT) to
do cluster shutdown.

At SU we have abstracted HBase away from the devs, they use thrift and
never do anymore more than hbase shell.

-ryan

On Wed, Mar 2, 2011 at 5:23 PM, Bill Graham <billgraham@gmail.com> wrote:
> Hi,
>
> We had a troubling experience today that I wanted to share. Our dev
> cluster got completely shut down by a developer by mistake, without
> said developer even realizing it. Here's how...
>
> We have multiple sets of HBase configs checked into SVN that
> developers can checkout and point their HBASE_CONF_DIR to to easily
> change from developing in local mode or testing against our
> distributed dev cluster.
>
> In local mode someone might do something like this:
>
> bin/start-hbase.sh
> bin/hbase shell
>
> ... do some work ...
>
> bin/stop-hbase.sh
>
> The problem arose when a developer accidentally tried to do this with
> their HBASE_CONF_DIR pointing to our dev cluster configs. When this
> happens, the first command will add another master to the cluster and
> the last command will shut down the entire cluster. I assume this
> happens via Zookeeper somehow, since we don't have ssh keys to
> remotely start/stop as the user running the processes.
>
> So the question is, is this a bug or a feature? If it's a feature it
> seems like an incredibly dangerous one. Once our live cluster is
> running, those configs will also be needed on the client so really bad
> things could happen by mistake.
>
> thanks,
> Bill
>

Mime
View raw message