hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane O'Donnell" <sha...@knownormal.com>
Subject Re: Hbase master not starting
Date Mon, 04 Jan 2016 17:26:09 GMT
I looked for region 82432aca9ede964943b40753cb64e808 on each of my region
servers and none of them had it.  They all had identical failures in the
log of attempting to open it, but none had it (or at least were successful
in opening it).

My solution to "finding it" was grepping for
"82432aca9ede964943b40753cb64e808" in the logs.  If there's a better or
more reliable way of searching for this, let me know.

Thanks -

Shane O.

========================
Shane O'Donnell
<http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=27fa6676-ade7-41f1-f304-76654be07bdd>
4819 Emperor Blvd., Ste 400
Durham, North Carolina 27703
tel: +1.424.262.KNOW x703
skype: shaneodonnell
email: shaneo@knownormal.com
========================
:wq!

On Mon, Jan 4, 2016 at 12:03 PM, Shane O'Donnell <shaneo@knownormal.com>
wrote:

> It's not there.  The directory listing was for the right directory, but
> the namespace directory is not there.
>
> Here it is one from directory level up:
>
> [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls -R
> /hbase/data
>
> drwxr-xr-x   - hdfs hadoop          0 2016-01-04 14:47 /hbase/data/hbase
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> /hbase/data/hbase/meta
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc
>
> -rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
> /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
> /hbase/data/hbase/meta/.tmp
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740
>
> -rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/.regioninfo
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
> /hbase/data/hbase/meta/1588230740/info
>
> drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits
>
> -rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
> /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>
> ========================
> Shane O'Donnell
>
> <http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6d1d2ed3-3d92-4821-eb9f-f589f936d34c>
> 4819 Emperor Blvd., Ste 400
> Durham, North Carolina 27703
> tel: +1.424.262.KNOW x703
> skype: shaneodonnell
> email: shaneo@knownormal.com
> ========================
> :wq!
>
> On Mon, Jan 4, 2016 at 11:59 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> In your listing, meta table directory structure was shown.
>>
>> Please look under /hbase/data/hbase for namespace table.
>>
>> Cheers
>>
>> On Mon, Jan 4, 2016 at 8:42 AM, Shane O'Donnell <shaneo@knownormal.com>
>> wrote:
>>
>> > I attempted to restore a backed-up "meta" directory from a copy made by
>> > 'hbase hbck', so my directory structure may be messed up:
>> >
>> > [user@hbase-prod2-master (ip-10-0-1-165) ~]$ sudo -u hdfs hdfs dfs -ls
>> -R
>> > /hbase/data/hbase
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> > /hbase/data/hbase/meta
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tabledesc
>> >
>> > -rw-r--r--   2 hbase hadoop        372 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tabledesc/.tableinfo.0000000001
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:48
>> > /hbase/data/hbase/meta/.tmp
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740
>> >
>> > -rw-r--r--   3 hdfs  hadoop         32 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/.regioninfo
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 14:47
>> > /hbase/data/hbase/meta/1588230740/info
>> >
>> > drwxr-xr-x   - hbase hadoop          0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/recovered.edits
>> >
>> > -rw-r--r--   3 hdfs  hadoop          0 2016-01-04 16:28
>> > /hbase/data/hbase/meta/1588230740/recovered.edits/2.seqid
>> >
>> > And I'm not seeing any "namespace" directory.
>> >
>> > Will check out the server now...
>> >
>> > Shane O.
>> >
>> > ========================
>> > Shane O'Donnell
>> > <
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=6a0326e0-41e4-46fd-bcce-114f44302f69
>> > >
>> > 4819 Emperor Blvd., Ste 400
>> > Durham, North Carolina 27703
>> > tel: +1.424.262.KNOW x703
>> > skype: shaneodonnell
>> > email: shaneo@knownormal.com
>> > ========================
>> > :wq!
>> >
>> > On Mon, Jan 4, 2016 at 9:44 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>> >
>> > > Can you log onto the server hosting region
>> > 82432aca9ede964943b40753cb64e808
>> > > and see what happened ?
>> > >
>> > > See if the namespace table can be found under rootdir.
>> > > e.g. assuming /apps/hbase/data is the rootdir, you should see
>> something
>> > > similar to the following:
>> > >
>> > > hdfs dfs -ls /apps/hbase/data/data/hbase/namespace
>> > > Found 3 items
>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
>> > > /apps/hbase/data/data/hbase/namespace/.tabledesc
>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:15
>> > > /apps/hbase/data/data/hbase/namespace/.tmp
>> > > drwxr-xr-x   - hbase hdfs          0 2015-12-15 19:51
>> > > /apps/hbase/data/data/hbase/namespace/844e1bab028e0ecc07d3bd8e34cc76a8
>> > >
>> > > On Mon, Jan 4, 2016 at 6:37 AM, Shane O'Donnell <
>> shaneo@knownormal.com>
>> > > wrote:
>> > >
>> > > > Some progress...
>> > > >
>> > > > /hbase did NOT have either the hbase.id or hbase.version files so
I
>> > > > temporarily changed hbase.rootdir and started the master so the
>> files
>> > > would
>> > > > be recreated elsewhere and copied them in.
>> > > >
>> > > > Now it starts fine, but my hbase tables are gone.  Specifically, I'm
>> > > > getting this error:
>> > > >
>> > > > hbase:namespace,,1451917103275.82432aca9ede964943b40753cb64e808.
>> > > > state=FAILED_OPEN, ts=Mon Jan 04 14:28:32 UTC 2016 (18s ago),
>> > > > server=ip-10-0-1-29.ec2.internal,60020,1451524442749
>> > > >
>> > > > Is my existing data toast, or is there a crafty way out of this?
>> > > >
>> > > > Thanks -
>> > > >
>> > > > Shane O.
>> > > >
>> > > > ========================
>> > > > Shane O'Donnell
>> > > > <
>> > > >
>> > >
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=9387b86f-bfad-4ebe-c506-e91a94d0c960
>> > > > >
>> > > > 4819 Emperor Blvd., Ste 400
>> > > > Durham, North Carolina 27703
>> > > > tel: +1.424.262.KNOW x703
>> > > > skype: shaneodonnell
>> > > > email: shaneo@knownormal.com
>> > > > ========================
>> > > > :wq!
>> > > >
>> > > > On Sun, Jan 3, 2016 at 10:21 PM, Shane O'Donnell <
>> > shaneo@knownormal.com>
>> > > > wrote:
>> > > >
>> > > > > Hi -
>> > > > >
>> > > > > My cluster has been running perfectly until the other day when
I
>> > found
>> > > it
>> > > > > down.
>> > > > >
>> > > > > The error seems to be related not being able to get the ClusterID
>> > from
>> > > > > zookeeper, but I'm stumped as to what to do about it.  This seems
>> to
>> > be
>> > > > the
>> > > > > relevant part of the master's log:
>> > > > >
>> > > > >      http://pastebin.com/C3iaxM3p
>> > > > >
>> > > > > starting at line 235 (also highlighted).
>> > > > >
>> > > > > Any help is appreciated!
>> > > > >
>> > > > > Thanks -
>> > > > >
>> > > > > Shane O.
>> > > > > ========================
>> > > > > Shane O'Donnell
>> > > > >
>> > > > > <
>> > > >
>> > >
>> >
>> http://t.sidekickopen32.com/e1t/c/5/f18dQhb0S7lC8dDMPbW2n0x6l2B9nMJW7t5XZs7grXnlW7d-5Xx3MqgJjW65jBJH56dSQdf3HZlqR02?t=https%3A%2F%2Fwww.knownormal.com%2F&si=5370834256920576&pi=ddbedf4b-a44c-4171-af8d-cc6a5e903dac
>> > > > >
>> > > > > 4819 Emperor Blvd., Ste 400
>> > > > > Durham, North Carolina 27703
>> > > > > tel: +1.424.262.KNOW x703
>> > > > > skype: shaneodonnell
>> > > > > email: shaneo@knownormal.com
>> > > > > ========================
>> > > > > :wq!
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message