hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ananth T. Sarathy" <ananth.t.sara...@gmail.com>
Subject Re: None of my tables are showing up
Date Tue, 27 Oct 2009 19:07:29 GMT
ok. We'll look into this. One last question. Now whenever HBase restarts I
have to reload the tables with the add table. Any ideas?
Ananth T Sarathy


On Mon, Oct 26, 2009 at 7:31 PM, Andrew Purtell <apurtell@apache.org> wrote:

> HBase is not developed or tested against Hadoop's S3FS. Have you considered
> using HDFS backed by EBS volumes instead of S3? HDFS on top of EBS insures
> the filesystem
> semantics that HBase expects.
>
> Getting back to the first pastebin, I see this
> > 2009-10-26 11:38:12,282 INFO
> org.apache.hadoop.hbase.master.HMaster: Found log folder :
> ip-10-244-9-171.ec2.internal,60020,1255635341245
> > 2009-10-26 11:38:12,282 INFO
> org.apache.hadoop.hbase.master.HMaster: Log folder doesn't belong to a
> known region server, splitting
>
> On a clean shutdown, logs will be deleted.
>
> And in your filesystem listing, there are some pretty big reconstruction
> logs:
>
> > -rwxrwxrwx   1  148161186 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/oldlogfile.log
>
> Was this a clean shutdown of HBase and Hadoop? (Stop services with
> stop-hbase.sh etc. then shut down, not just ec2-terminate-instance.)Did some
> region server crash? Any errors in the region server logs during shutdown?
>
>   - Andy
>
>
>
>
>
> ________________________________
> From: Ananth T. Sarathy <ananth.t.sarathy@gmail.com>
> To: hbase-user@hadoop.apache.org
> Sent: Mon, October 26, 2009 10:58:56 AM
> Subject: Re: None of my tables are showing up
>
> Here is the output...
>
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/-ROOT-
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052
> -rwxrwxrwx   1        622 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/.regioninfo
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/-ROOT-/70236052/info
> -rwxrwxrwx   1        841 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/1996912273062753577
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/2193779228149112035
> -rwxrwxrwx   1        462 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/2351329094215113700
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/246199753790950739
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/2525006410357531294
> -rwxrwxrwx   1        462 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/2791558595254923243
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/4021322725584207875
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/4849111854392651007
> -rwxrwxrwx   1        462 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/5363712926256617749
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/5611302436720701967
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/6831054531189082272
> -rwxrwxrwx   1        462 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/68808045018098702
> -rwxrwxrwx   1        462 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/7253796415415350972
> -rwxrwxrwx   1        461 1969-12-31 19:00
> /hbasedata/-ROOT-/70236052/info/7757549506442648999
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/.META.
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/.META./1028785192
> -rwxrwxrwx   1        932 1969-12-31 19:00
> /hbasedata/.META./1028785192/.regioninfo
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian
> -rwxrwxrwx   1        911 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/1968569179310304585
> -rwxrwxrwx   1        907 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/2158222582121734940
> -rwxrwxrwx   1        907 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/2646558927756874624
> -rwxrwxrwx   1        907 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/3553325538317585634
> -rwxrwxrwx   1       2037 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/4092877814999091772
> -rwxrwxrwx   1       1185 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/4380516119952135238
> -rwxrwxrwx   1       2437 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/6276910425096618885
> -rwxrwxrwx   1        911 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/7295248711127991783
> -rwxrwxrwx   1        911 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/7481877460145101553
> -rwxrwxrwx   1        911 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/752086821053742201
> -rwxrwxrwx   1        907 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/7817013763537032897
> -rwxrwxrwx   1       1173 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/8114421049620083173
> -rwxrwxrwx   1       1563 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/8322527460340654381
> -rwxrwxrwx   1        911 1969-12-31 19:00
> /hbasedata/.META./1028785192/historian/8999997006964561665
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.META./1028785192/info
> -rwxrwxrwx   1        694 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/1512108120879032405
> -rwxrwxrwx   1       3971 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/2775950188169392903
> -rwxrwxrwx   1        646 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/3098482905740093325
> -rwxrwxrwx   1        696 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/3098738884562255978
> -rwxrwxrwx   1       1561 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/383466394636378400
> -rwxrwxrwx   1        694 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/4974231550598739590
> -rwxrwxrwx   1        694 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/5034205868382579974
> -rwxrwxrwx   1        694 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/5137021646723327707
> -rwxrwxrwx   1        696 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/5626349865156916725
> -rwxrwxrwx   1        696 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/6707659272909568633
> -rwxrwxrwx   1        696 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/7346625667936340286
> -rwxrwxrwx   1        696 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/7438651887567350087
> -rwxrwxrwx   1       5699 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/7671471165225263042
> -rwxrwxrwx   1       1202 1969-12-31 19:00
> /hbasedata/.META./1028785192/info/8586093328550564694
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/.META./compaction.dir
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/.logs
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-242-71-191.ec2.internal,60020,1256571494907
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1255635341245
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-244-9-171.ec2.internal,60020,1256571493935
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-245-82-160.ec2.internal,58439,1255623467258
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1255977369779
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/.logs/ip-10-245-82-160.ec2.internal,60020,1256571494768
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/GS_Applications
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711
> -rwxrwxrwx   1       2632 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/.regioninfo
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Application
> -rwxrwxrwx   1   13303446 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Application/2124252833220218245
> -rwxrwxrwx   1      26149 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Application/5509287308900349804
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryData
> -rwxrwxrwx   1     267219 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryData/2009456488953384412
> -rwxrwxrwx   1      66582 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryData/4704251150705174118
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryRetrieval
> -rwxrwxrwx   1    1325402 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryRetrieval/1389090912532859839
> -rwxrwxrwx   1       3093 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/BinaryRetrieval/9178892710156256966
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Files
> -rwxrwxrwx   1   69548974 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Files/6175860477346559387
> -rwxrwxrwx   1  157418897 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Files/7670651200769000729
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Info
> -rwxrwxrwx   1      71844 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Info/1715357090325080119
> -rwxrwxrwx   1   36052800 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Info/7204941744454360171
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Network
> -rwxrwxrwx   1   42060118 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/Network/3039242992290063476
> -rwxrwxrwx   1  148161186 1969-12-31 19:00
> /hbasedata/GS_Applications/1264304711/oldlogfile.log
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1732076772
> -rwxrwxrwx   1       2484 1969-12-31 19:00
> /hbasedata/GS_Applications/1732076772/.regioninfo
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139
> -rwxrwxrwx   1       2780 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/.regioninfo
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Application
> -rwxrwxrwx   1   27428283 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Application/7122413424433440685
> -rwxrwxrwx   1      47203 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Application/8284859005131013460
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/BinaryData
> -rwxrwxrwx   1     311872 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/BinaryData/8301036036223997974
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/BinaryRetrieval
> -rwxrwxrwx   1       6368 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/BinaryRetrieval/1610935246237259262
> -rwxrwxrwx   1    2738315 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/BinaryRetrieval/8651784529191787005
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Files
> -rwxrwxrwx   1  127201108 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Files/7247655344177400794
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Info
> -rwxrwxrwx   1   74860411 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Info/514906809998228574
> -rwxrwxrwx   1     162245 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Info/8300037259597514515
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Network
> -rwxrwxrwx   1        228 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/Network/2748465749080635338
> -rwxrwxrwx   1   32857683 1969-12-31 19:00
> /hbasedata/GS_Applications/1888816139/oldlogfile.log
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_Applications/compaction.dir
> drwxrwxrwx   -          0 1969-12-31 19:00 /hbasedata/GS_EventLists
> drwxrwxrwx   -          0 1969-12-31 19:00
> /hbasedata/GS_EventLists/453741364
> -rwxrwxrwx   1       1191 1969-12-31 19:00
> /hbasedata/GS_EventLists/453741364/.regioninfo
> -rwxrwxrwx   1          3 1969-12-31 19:00 /hbasedata/hbase.version
>
> Ananth T Sarathy
>
>
> On Mon, Oct 26, 2009 at 1:56 PM, stack <stack@duboce.net> wrote:
>
> > If you open the .regioninfo file inside in each region, you can see its
> > content?  You might be able to make loadtable.rb work by making it go via
> > jets3s but would take some tinkering on your part.
> >
> > As to why the regions were lost from .META. in first place, I'm not sure
> > why; was there no flush of .META.?    Usually this is frequent so you
> > should
> > have some of your table info.
> >
> > We're about hdfs here in this channel, usually, and speaking for myself,
> > I've played with hadoop on s3 but not hbase so I'm not much good to you
> I'm
> > afraid.
> >
> > St.Ack
> >
> >
> > On Mon, Oct 26, 2009 at 10:42 AM, Ananth T. Sarathy <
> > ananth.t.sarathy@gmail.com> wrote:
> >
> > > yeah I  see a folder with my table name. and inside that I see 3 other
> > > folders named with a bunch of numbers (which i assume are the regions).
> > >
> > > Ananth T Sarathy
> > >
> > >
> > > On Mon, Oct 26, 2009 at 1:38 PM, stack <stack@duboce.net> wrote:
> > >
> > > > If you list the hbase homedir in s3, do you see tables and then under
> > > that
> > > > regions?
> > > >
> > > > The loadtable.rb script won't work against a fs that is other than
> hdfs
> > > > unfortunately.
> > > > St.Ack
> > > >
> > > > On Mon, Oct 26, 2009 at 10:36 AM, Ananth T. Sarathy <
> > > > ananth.t.sarathy@gmail.com> wrote:
> > > >
> > > > > well where i have hadoop installed (not running) since not using
it
> > for
> > > > > hdfs
> > > > >
> > > > >  bin/hadoop fs -lsr /hbase.rootdir
> > > > > lsr: Cannot access /hbase.rootdir: No such file or directory.
> > > > >
> > > > >
> > > > >
> > > > > Ananth T Sarathy
> > > > >
> > > > >
> > > > > On Mon, Oct 26, 2009 at 1:24 PM, stack <stack@duboce.net> wrote:
> > > > >
> > > > > > When you do bin/hadoop fs -lsr /hbase.rootdir what happens?
> > > > > > St.Ack
> > > > > >
> > > > > > On Mon, Oct 26, 2009 at 10:05 AM, Ananth T. Sarathy <
> > > > > > ananth.t.sarathy@gmail.com> wrote:
> > > > > >
> > > > > > > @ the command line? I get a command not found.
> > > > > > > Ananth T Sarathy
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Oct 26, 2009 at 1:02 PM, stack <stack@duboce.net>
> wrote:
> > > > > > >
> > > > > > > > Its in head of the 0.20 branch (We should roll a 0.20.2
> soon).
> > > > > > > >
> > > > > > > > What happens if you do a lsr /hbase?
> > > > > > > >
> > > > > > > > St.Ack
> > > > > > > >
> > > > > > > > On Mon, Oct 26, 2009 at 9:52 AM, Ananth T. Sarathy
<
> > > > > > > > ananth.t.sarathy@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > I don't have a loadtable.rb
> > > > > > > > >
> > > > > > > > > is that in 0.20.0?
> > > > > > > > > This is what i have in bin
> > > > > > > > >
> > > > > > > > > Formatter.rb   hbase-config.sh   regionservers.sh
> > >  zookeepers.sh
> > > > > > > > > HBase.rb       hbase-daemon.sh   rename_table.rb
> > > > > > > > > copy_table.rb  hbase-daemons.sh  start-hbase.sh
> > > > > > > > > hbase          hirb.rb           stop-hbase.sh
> > > > > > > > >
> > > > > > > > > Ananth T Sarathy
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Mon, Oct 26, 2009 at 12:45 PM, stack <stack@duboce.net>
> > > > wrote:
> > > > > > > > >
> > > > > > > > > > Your log would seem to say that there are
no tables in
> > hbase:
> > > > > > > > > >
> > > > > > > > > > 2009-10-26 11:40:13,984 INFO
> > > > > > > > org.apache.hadoop.hbase.master.BaseScanner:
> > > > > > > > > > RegionManager.metaScanner scan of 0 row(s)
of meta region
> > > > > {server:
> > > > > > > > > > 10.245.82.160:60020, regionname: .META.,,1,
startKey:
> <>}
> > > > > complete
> > > > > > > > > >
> > > > > > > > > > Do as Jon suggests.  Do you see listing
of regions?  If
> so,
> > > it
> > > > > > would
> > > > > > > > seem
> > > > > > > > > > that edits to .META. table are not persisting
on your s3
> > > hdfs.
> > > > > > You
> > > > > > > > > might
> > > > > > > > > > be able to add in all tables using the bin/loadtable.rb
> > > script;
> > > > > it
> > > > > > > > reads
> > > > > > > > > > the
> > > > > > > > > > .regioninfo files in all regions and per
region adds to
> > > .META.
> > > > an
> > > > > > > > entry.
> > > > > > > > > >
> > > > > > > > > > St.Ack
> > > > > > > > > >
> > > > > > > > > > On Mon, Oct 26, 2009 at 9:31 AM, Ananth
T. Sarathy <
> > > > > > > > > > ananth.t.sarathy@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > I am confused , why would I need a
hadoop home if I am
> > > using
> > > > s3
> > > > > > and
> > > > > > > > the
> > > > > > > > > > > jets3t package to write to s3?
> > > > > > > > > > > Ananth T Sarathy
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Oct 26, 2009 at 12:25 PM, Jonathan
Gray <
> > > > > > jlist@streamy.com
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Not S3, HDFS.  Can you checkout
the web ui or using
> the
> > > > > > > > command-line
> > > > > > > > > > > > interface?
> > > > > > > > > > > >
> > > > > > > > > > > > $HADOOP_HOME/bin/hadoop dfs -lsr
/hbase
> > > > > > > > > > > >
> > > > > > > > > > > > ...would be a good start
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Ananth T. Sarathy wrote:
> > > > > > > > > > > >
> > > > > > > > > > > >> i see all my blocks in my
s3 bucket.
> > > > > > > > > > > >> Ananth T Sarathy
> > > > > > > > > > > >>
> > > > > > > > > > > >>
> > > > > > > > > > > >> On Mon, Oct 26, 2009 at 12:17
PM, Jonathan Gray <
> > > > > > > > jlist@streamy.com>
> > > > > > > > > > > >> wrote:
> > > > > > > > > > > >>
> > > > > > > > > > > >>  Do you see the files/blocks
in HDFS?
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Ananth T. Sarathy wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>  I just restarted Hbase
and when I go into the
> shell
> > > and
> > > > > type
> > > > > > > > list,
> > > > > > > > > > > none
> > > > > > > > > > > >>>> of
> > > > > > > > > > > >>>> my tables are listed,
but I see all the
> data/blocks
> > in
> > > > s3.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> here is the master
log when it's restarted
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> http://pastebin.com/m1ebb7217
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> this happened once
before, but we just started
> over
> > > > since
> > > > > it
> > > > > > > was
> > > > > > > > > > early
> > > > > > > > > > > >>>> in
> > > > > > > > > > > >>>> our process. This
time we have a lot of data, and
> > need
> > > > to
> > > > > > keep
> > > > > > > > it.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Ananth T Sarathy
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message