hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane O'Donnell" <sha...@knownormal.com>
Subject Re: Hbase master not starting
Date Mon, 04 Jan 2016 16:42:40 GMT
I attempted to restore a backed-up "meta" directory from a copy made by
'hbase hbck', so my directory structure may be messed up:

[user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
/hbase/data/hbase

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc

-rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
/hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
/hbase/data/hbase/meta/.tmp

drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740

-rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/.regioninfo

drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
/hbase/data/hbase/meta/1588230740/info

drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits

-rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
/hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid

And I'm not seeing any "namespace" directory.

Will check out the server now...

Shane O.

========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!

On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:

> Can you log onto the server hosting region 82432aca9ede964943b40753cb64e808
> and see what happened ?
>
> See if the namespace table can be found under rootdir.
> e.g. assuming /apps/hbase/data is the rootdir, you should see something
> similar to the following:
>
> hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
> Found 3 items
> drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
> /apps/hbase/data/data/hbase/namespace/.tabledesc
> drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
> /apps/hbase/data/data/hbase/namespace/.tmp
> drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:51
> /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>
> On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <shaneo@knownormal.com>
> wrote:
>
> > Some progress...
> >
> > /hbase did NOT have either the hbase.id or hbase.version files so I
> > temporarily changed hbase.rootdir and started the master so the files
> would
> > be recreated elsewhere and copied them in.
> >
> > Now it starts fine, but my hbase tables are gone.  Specifically, I'm
> > getting this error:
> >
> > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
> > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
> > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
> >
> > Is my existing data toast, or is there a crafty way out of this?
> >
> > Thanks -
> >
> > Shane O.
> >
> > ========================
> > Shane O'Donnell
> > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
> > >
> > 4819 Emperor Blvd., Ste 400
> > Durham, North Carolina 27703
> > tel: +1.424.262.KNOW x703
> > skype: shaneodonnell
> > email: shaneo@knownormal.com
> > ========================
> > :wq!
> >
> > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <shaneo@knownormal.com>
> > wrote:
> >
> > > Hi -
> > >
> > > My cluster has been running perfectly until the other day when I found
> it
> > > down.
> > >
> > > The error seems to be related not being able to get the ClusterID from
> > > zookeeper, but I'm stumped as to what to do about it.  This seems to be
> > the
> > > relevant part of the master's log:
> > >
> > >      http://pastebin.com/C3iaxM3p
> > >
> > > starting at line 235 (also highlighted).
> > >
> > > Any help is appreciated!
> > >
> > > Thanks -
> > >
> > > Shane O.
> > > ========================
> > > Shane O'Donnell
> > >
> > > <
> >
> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
> > >
> > > 4819 Emperor Blvd., Ste 400
> > > Durham, North Carolina 27703
> > > tel: +1.424.262.KNOW x703
> > > skype: shaneodonnell
> > > email: shaneo@knownormal.com
> > > ========================
> > > :wq!
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message