hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cecile, Adam" <Adam.Cec...@hitec.lu>
Subject RE: Table is disabled an no way to get it back online
Date Tue, 22 Nov 2016 15:16:39 GMT
Hey Ted,

Thank you. Heading home right now but I'll start the laptop again. Not sure exactly how I
should turn debug log on so if you have the information it'd be appreciated, otherwise I'll
look at the xml files.

Regards, Adam.
________________________________________
De : Ted Yu <yuzhihong@gmail.com>
Envoyé : mardi 22 novembre 2016 15:46
À : user@hbase.apache.org
Objet : Re: Table is disabled an no way to get it back online

Master log contained entries in the following form:

2016-11-22 13:13:41,836 INFO  [ProcedureExecutor-3]
procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
(table=sentinel-meta) id=43538 owner=hbase         state=ROLLEDBACK
exec-time=242hrs, 10mins, 28.896sec
exception=org.apache.hadoop.hbase.TableNotEnabledException: sentinel-meta

Note the procedure Id was around 43000, far lower than 147464.

Can you turn debug log on and repost master log ?

Thanks

On Tue, Nov 22, 2016 at 4:16 AM, Cecile, Adam <Adam.Cecile@hitec.lu> wrote:

> Hello,
>
>
> We're having a table stuck in disabled state. First I'd like to start with
> what I tried already:
>
>
> * Restart all machines involved in HBase cluster
>
> * hbase hbck with varios arguments
>
> * hdfs fsck
>
> * Purge ZK /hbase and restart masters
>
>
> Now more details anout what happens:
>
> * When enabling from hbase shell:
>
>
> hbase(main):002:0> enable "sentinel-meta"
> ERROR: The procedure 147464 is still running
>
>
> The task ID changes every time I run the command so I think it's talking
> about itself (and it gets stuck for a while before saying anything)
>
>
> In the log, all I can see is:
>
> 2016-11-22 13:10:50,776 INFO  [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43220 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 52mins, 7.454sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:50,781 INFO  [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
> 2016-11-22 13:10:51,084 INFO  [ProcedureExecutor-0]
> procedure2.ProcedureExecutor: Rolledback procedure DisableTableProcedure
> (table=sentinel-meta) id=43221 owner=hbase state=ROLLEDBACK
> exec-time=242hrs, 51mins, 42.288sec exception=org.apache.hadoop.hbase.TableNotEnabledException:
> sentinel-meta
> 2016-11-22 13:10:51,088 INFO  [ProcedureExecutor-0] procedure.DisableTableProcedure:
> Table sentinel-meta isn't enabled; skipping disable
>
>
> Please also find attached a complete log from startup to shutdown on a
> single active master. You'll see the table is found as well as the regions
> but it gets deactivated with no reason.
>
>
> Thanks a lot for your help, we're kinda running out if ideas here.
>
>
> Best regards,
>
>
> Adam.
>
>
>

Mime
View raw message