hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: hbck -fix
Date Sun, 03 Jul 2011 04:40:26 GMT
You have a snapshot of the state of .META. at time you noticed it
messed up?  And the master log from around the time of the startup
post-fs-fillup?
St.Ack

On Sat, Jul 2, 2011 at 7:27 PM, Wayne <wav100@gmail.com> wrote:
> Like most problems we brought it on ourselves. To me the bigger issue is how
> to get out. Since region definitions are the core of what hbase does, it
> would be great to have a bullet proof recovery process that we can invoke to
> get us out. Bugs and human error will bring on problems and nothing will
> ever change that, but not having tools to help recover out of the hole is
> where I think it is lacking. HDFS is very stable. The hbase .META. table
> (and -ROOT-?) are the core how HBase manages things. If this gets out of
> whack all is lost. I think it would be great to have automatic backup of the
> meta table and the ability to recover everything based on the HDFS data out
> there and the backup. Something like a recovery mode that goes through and
> sees what is out there and rebuilds the meta based on it. With corrupted
> data and lost regions etc. etc. like any relational database there should be
> one or more recovery modes that goes through everything and rebuilds it
> consistently. Data may be lost but at least the cluster will be left in a
> 100% consistent/clean state. Manual editing of .META. is not something
> anyone should do (especially me). It is prone to human error...it should be
> easy to have well tested recover tools that can do the hard work for us.
>
> Below is an attempt at the play by play in case it helps. It all started
> with the root partition of the namenode/hmaster filling up due to a table
> export.
>
> When I restarted hadoop this error was in the namenode log;
> "java.io.IOException: Incorrect data format. logVersion is -18 but
> writables.length is 0"
>
> So i found this<https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/e35ee876da1a3bbc>,
> which mentioned editing the namenode log files after verifying our namenode
> log files seem to have the same symptom. So I copied each namenode "name"
> file to root's home directory and followed their advice.
> That allowed the namenode to start, but then HDFS wouldn't come up. It kept
> hanging in safe-mode with the repeated error;
> "The ratio of reported blocks 0.9925 has not reached the threshold 0.9990.
> Safe mode will be turned off automatically."
> So i turned safe-mode off with; "hadoop dfsadmin -safemode leave" and I
> tried to run "hadoop fsck" a few times and it still showed HDFS as
> "corrupt", so i did "hadoop fsck -move" and this is the last part of the
> output;
> ....................................................................................Status:
> CORRUPT
>  Total size: 1423140871890 B (Total open files size: 668770828 B)
>  Total dirs: 3172
>  Total files: 2584 (Files currently being written: 11)
>  Total blocks (validated): 23095 (avg. block size 61621167 B) (Total open
> file blocks (not validated): 10)
>  ********************************
>  CORRUPT FILES: 65
>  MISSING BLOCKS: 173
>  MISSING SIZE: 8560948988 B
>  CORRUPT BLOCKS: 173
>  ********************************
>  Minimally replicated blocks: 22922 (99.25092 %)
>  Over-replicated blocks: 0 (0.0 %)
>  Under-replicated blocks: 0 (0.0 %)
>  Mis-replicated blocks: 0 (0.0 %)
>  Default replication factor: 3
>  Average block replication: 2.9775276
>  Corrupt blocks: 173
>  Missing replicas: 0 (0.0 %)
>  Number of data-nodes: 10
>  Number of racks: 1
>
> I ran it again and got this;
> .Status: HEALTHY
>  Total size: 1414579922902 B (Total open files size: 668770828 B)
>  Total dirs: 3272
>  Total files: 2519 (Files currently being written: 11)
>  Total blocks (validated): 22922 (avg. block size 61712761 B) (Total open
> file blocks (not validated): 10)
>  Minimally replicated blocks: 22922 (100.0 %)
>  Over-replicated blocks: 0 (0.0 %)
>  Under-replicated blocks: 0 (0.0 %)
>  Mis-replicated blocks: 0 (0.0 %)
>  Default replication factor: 3
>  Average block replication: 3.0
>  Corrupt blocks: 0
>  Missing replicas: 0 (0.0 %)
>  Number of data-nodes: 10
>  Number of racks: 1
>
>
> The filesystem under path '/' is HEALTHY
>
> So i started everything and it seemed to be superficially functional.
>
> I then shutdown hadoop and restarted. hadoop came up in a matter of a few
> minutes, then hbase took about ten minutes of seeming to copy files around,
> based on the hbase master logs.
>
> After this we saw region not found client errors on some tables. I ran hbase
> hbck to look for problems and saw the errors I reported in the original
> post. Add in the ganglia problems and a botched attempt to edit the .META.
> table which brought us even further into the rabbit hole. I then decided to
> drop the affected tables but lo and behold one can not disable a table that
> has messed up regions...so I manually deleted the data but some of the
> .META. table entries were still there. Finally this afternoon we reformatted
> the entire cluster.
>
> Thanks.
>
>
>
> On Sat, Jul 2, 2011 at 5:25 PM, Stack <stack@duboce.net> wrote:
>
>> On Sat, Jul 2, 2011 at 9:55 AM, Wayne <wav100@gmail.com> wrote:
>> > It just returns a ton of errors (import: command not found). Our cluster
>> is
>> > hosed anyway. I am waiting to get it completely re-installed from
>> scratch.
>> > Hope has long since flown out the window. I just changed my opinion of
>> what
>> > it takes to manage hbase. A Java engineer is required on staff. I also
>> > realized now a backup strategy is more important than for a RDBMS. Having
>> > RF=3 in HDFS offers no insurance against hbase lossing its shirt and
>> having
>> > .META. getting corrupted. I think I just found the achilles heel.
>> >
>> >
>>
>> Yeah, stability is primary but I do not know how you got into the
>> circumstance you find yourself in.  All I can offer is to try and do
>> diagnostics since avoiding hitting this situation again is of utmost
>> importance.
>>
>> St.Ack
>>
>

Mime
View raw message