hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anil gupta <anilgupt...@gmail.com>
Subject Re: Unable to see old tables after migrating from 0.94 to 0.98
Date Wed, 30 Jul 2014 00:45:34 GMT
Just reporting back on mailing list that i was able to fix the issue by a
lot of manual work. This problem happened because we forgot to run "upgrade
-execute" after installing hbase0.98 and before starting 0.98.
If you dont run "upgrade -execute" then user gets and error regarding
"hbase.version". I deleted that file and then HBase came up. But since, i
deleted hbase.versions file so HBase didnt do any migration of my 0.94 data.

Here is the high level stuff that i did to fix this issue(NOTE: these steps
may vary as per cluster setup):
In ZK i did following stuff:
Deleted all znodes under /hbase/table
Recursively Deleted /hbase/table-lock znode
delete /hbase/meta-region-server
delete /hbase/region-in-transition
delete /hbase/balancer
rmr /hbase/namespace
rmr /hbase/replication
rmr /hbase/recovering-regions
rmr /hbase/splitWAL

In HDFS:
I removed /hbase/data directory and moved around some other related
directory.
Updated the hbase.version file to 0.94  version file.

Then i ran "upgrade -execute" and it ran successfully. Started the cluster
and i was able to see all the tables along with their data.
Thanks Stack for your help.

~Anil


On Mon, Jul 28, 2014 at 11:01 PM, anil gupta <anilgupta84@gmail.com> wrote:

> Please find my reply inline.
>
>
> On Mon, Jul 28, 2014 at 10:10 PM, Stack <stack@duboce.net> wrote:
>
>> On Mon, Jul 28, 2014 at 8:39 PM, anil gupta <anilgupta84@gmail.com>
>> wrote:
>>
>> > Please find my reply inline.
>> >
>> >
>> > On Mon, Jul 28, 2014 at 6:09 PM, Stack <stack@duboce.net> wrote:
>> >
>> > > On Mon, Jul 28, 2014 at 5:45 PM, anil gupta <anilgupta84@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Stack,
>> > > >
>> > > > I basically did full table scan for hbase:meta by running this(since
>> > > > cluster has 2-3 regions only): scan 'hbase:meta', {RAW => true,
>> > VERSIONS
>> > > =>
>> > > > 10}
>> > > > Output of above query shows rows for "hbase:namespace" and "dummy"
>> > table.
>> > > > So, it seems like hbase:meta doesn't have any old data.
>> > > >
>> > > >
>> > > Yeah. Check size of the hbase:meta region.  If small, yeah, may not
>> have
>> > > the above.  If you look around in hdfs, you see an old .META. dir?
>> > >
>> > > Anil: Yes, it looks like this dir "/hbase/.META." has data of meta
>> table
>> > before upgrade to 0.98. Are there any non-backward compatible changes in
>> > META?
>> > Is it possible to just import the HFile of 0.94 in hbase:meta table? It
>> > seems like directory structure under hdfs has changes in 0.98. I am
>> unable
>> > to find directories for a table "dummy" i created in 0.98.
>> >
>> >
>>
>> I think what happened is that because you failed run the upgrade script,
>> 0.98 tried to start over a 0.94 and complained about the hbase.version
>> file
>> it found because it had not be protobuf'd.  You did what I would have
>> done,
>> 'fixed it', by removing the obstacle only when hbase does not find an
>> hbase.version, it thinks it all a fresh install.
>>
> Anil: Well, deleting hbase.version turned out be a hack that proved
> costly. I take back my word 'fixed it' :)
>
>>
>>
>> Without knowing much more, if you put back a 0.94 era hbase.version in
>> place of the one written by 0.98, then you ran the upgrade script, I
>> wonder
>> if that'd fix it?
>>
> Anil: Gonna try this soon and report back.
>
>>
>>
>>
>> > > > Our sysadmin team is still trying to locate all the logs. However,
i
>> > > > remember one error that i resolved. I was related to
>> "hbase.versions"
>> > > file.
>> > > > It was some complaint about Protbouf version. I deleted
>> hbase.versions
>> > > file
>> > > > and then HBase came up.
>> > >
>> > >
>> > > Hmm. So, it probably thought this a new install and just created the
>> new
>> > > layout.  Is the old 0.94 data is adjacent to the new layout?
>> > >
>> > Anil: Yes, that's what i feel. The upgrade happened as it was a fresh
>> > install. Does it seems like a bug in upgrade?
>> >
>> >
>> Well, if you ended up in this state, yeah, there is a problem with it.
>>
>> St.Ack
>>
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
>



-- 
Thanks & Regards,
Anil Gupta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message