hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ananth T. Sarathy" <ananth.t.sara...@gmail.com>
Subject Re: hbase on s3 and safemode
Date Wed, 07 Oct 2009 17:35:20 GMT
Here is the log  since I started it...

Wed Oct  7 13:27:26 EDT 2009 Starting master on ip-10-244-9-171
ulimit -n 1024
2009-10-07 13:27:26,404 INFO org.apache.hadoop.hbase.master.HMaster:
vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Sun Microsystems Inc.,
vmVersion=14.2-b01
2009-10-07 13:27:26,405 INFO org.apache.hadoop.hbase.master.HMaster:
vmInputArguments=[-Xmx2000m, -XX:+HeapDumpOnOutOfMemoryError,
-Djava.io.tmpdir=/mnt/tmp,
-Dhbase.log.dir=/mnt/apps/hadoop/hbase/bin/../logs,
-Dhbase.log.file=hbase-root-master-ip-10-244-9-171.log,
-Dhbase.home.dir=/mnt/apps/hadoop/hbase/bin/.., -Dhbase.id.str=root,
-Dhbase.root.logger=INFO,DRFA,
-Djava.library.path=/mnt/apps/hadoop/hbase/bin/../lib/native/Linux-amd64-64]
2009-10-07 13:27:27,525 INFO org.apache.hadoop.hbase.master.HMaster: Root
region dir: s3://hbase2.s3.amazonaws.com:80/hbasedata/-ROOT-/70236052
2009-10-07 13:27:27,751 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics:
Initializing RPC Metrics with hostName=HMaster, port=60000
2009-10-07 13:27:27,827 INFO org.apache.hadoop.hbase.master.HMaster: HMaster
initialized on 10.244.9.171:60000
2009-10-07 13:27:27,829 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=Master, sessionId=HMaster
2009-10-07 13:27:27,830 INFO
org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized
2009-10-07 13:27:27,932 INFO org.mortbay.util.Credential: Checking Resource
aliases
2009-10-07 13:27:27,936 INFO org.mortbay.http.HttpServer: Version
Jetty/5.1.4
2009-10-07 13:27:27,936 INFO org.mortbay.util.Container: Started
HttpContext[/logs,/logs]
2009-10-07 13:27:28,202 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@3209fa8f
2009-10-07 13:27:28,244 INFO org.mortbay.util.Container: Started
WebApplicationContext[/static,/static]
2009-10-07 13:27:28,361 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@b0c0f66
2009-10-07 13:27:28,364 INFO org.mortbay.util.Container: Started
WebApplicationContext[/,/]
2009-10-07 13:27:28,636 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.servlet.WebApplicationHandler@3c2d7440
2009-10-07 13:27:28,638 INFO org.mortbay.util.Container: Started
WebApplicationContext[/api,rest]
2009-10-07 13:27:28,639 INFO org.mortbay.http.SocketListener: Started
SocketListener on 0.0.0.0:60010
2009-10-07 13:27:28,639 INFO org.mortbay.util.Container: Started
org.mortbay.jetty.Server@28b301f2
2009-10-07 13:27:28,640 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
Responder: starting
2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
listener on 60000: starting
2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 0 on 60000: starting
2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 1 on 60000: starting
2009-10-07 13:27:28,641 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 2 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 3 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 4 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 5 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 6 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 60000: starting
2009-10-07 13:27:28,642 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 8 on 60000: starting
2009-10-07 13:27:28,642 DEBUG org.apache.hadoop.hbase.master.HMaster:
Started service threads
2009-10-07 13:27:28,643 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 9 on 60000: starting
2009-10-07 13:28:09,519 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:11,542 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:13,543 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:15,545 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:17,548 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:19,555 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:28:27,834 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2009-10-07 13:29:27,832 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2009-10-07 13:29:37,593 INFO org.apache.hadoop.hbase.master.RegionManager:
in safe mode
2009-10-07 13:30:27,834 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2009-10-07 13:31:27,836 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2009-10-07 13:32:27,838 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned
2009-10-07 13:33:27,840 INFO org.apache.hadoop.hbase.master.BaseScanner: All
0 .META. region(s) scanned


Ananth T Sarathy


On Wed, Oct 7, 2009 at 1:20 PM, stack <stack@duboce.net> wrote:

> Thats interesting to hear.  Keep us posted.
>
> HBase asks the filesystem if its in safe mode and if it is, it parks
> itself.  Here is code from master:
>
>    if (this.fs instanceof DistributedFileSystem) {
>      // Make sure dfs is not in safe mode
>      String message = "Waiting for dfs to exit safe mode...";
>      while (((DistributedFileSystem) fs).setSafeMode(
>          FSConstants.SafeModeAction.SAFEMODE_GET)) {
>        LOG.info(message);
>        try {
>          Thread.sleep(this.threadWakeFrequency);
>        } catch (InterruptedException e) {
>          //continue
>        }
>      }
>    }
>
>
> Then there is hbase's notion of safemode.  It will be in safe mode until it
> does initial scan of catalog tables.  The master keeps a flag in zookeeper
> while its in safemode so regionservers are aware of the state:
>
>  public boolean inSafeMode() {
>    if (safeMode) {
>      if(isInitialMetaScanComplete() && regionsInTransition.size() == 0 &&
>         tellZooKeeperOutOfSafeMode()) {
>        master.connection.unsetRootRegionLocation();
>        safeMode = false;
>        LOG.info("exiting safe mode");
>      } else {
>        LOG.info("in safe mode");
>      }
>    }
>    return safeMode;
>  }
>
> Have you seen the .META. and -ROOT- deploy to regionservers?  Have you seen
> that these regions being scanned in the master log?  (Enable DEBUG if not
> already enabled).
>
> Yours,
> ST.Ack
>
>
> On Wed, Oct 7, 2009 at 10:06 AM, Ananth T. Sarathy <
> ananth.t.sarathy@gmail.com> wrote:
>
> > We have been running Hbase on a s3 filesystem. It's the hbase
> regionserver,
> > not HDFS since we are using s3.  We haven't felt like it's been too slow,
> > though the amount of data we are pushing isn't sufficiently large enough
> to
> > notice yet.
> > Ananth T Sarathy
> >
> >
> > On Wed, Oct 7, 2009 at 12:47 PM, stack <stack@duboce.net> wrote:
> >
> > > HBase or HDFS is in safe mode.  My guess is that its the latter.   Can
> > you
> > > figure from HDFS logs why it won't leave safe mode?  Usually
> > > under-replication or a loss of a large swath of the cluster will flip
> on
> > > the
> > > safe-mode switch.
> > >
> > > Are you trying to run HBASE on an S3 filesystem?  An HBasista tried it
> in
> > > the past and, FYI, found it insufferably slow.  Let us know how it goes
> > for
> > > you.
> > >
> > > Thanks,
> > > St.Ack
> > >
> > > On Wed, Oct 7, 2009 at 9:33 AM, Ananth T. Sarathy <
> > > ananth.t.sarathy@gmail.com> wrote:
> > >
> > > > my  regionserver has been stuck in safemode. What can i do to get it
> > out
> > > > safemode?
> > > >
> > > > Ananth T Sarathy
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message