hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amandeep Khurana <ama...@gmail.com>
Subject Re: Connection problem during data import into hbase
Date Sat, 21 Feb 2009 11:17:39 GMT
I changed the config and restarted the cluster. This time the job went upto
16% and the same problem started.

I'll do the stuff with the logs now and see what comes out.


Amandeep Khurana
Computer Science Graduate Student
University of California, Santa Cruz


On Sat, Feb 21, 2009 at 3:08 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:

> you have to change hadoop-site.xml and restart HDFS.
>
> you should also change the logging to be more verbose in hbase - check out
> the hbase FAQ (link missing -ed).
>
> if you get the problem again, peruse the hbase logs and post what is going
> on there.  the client errors dont really include the root cause on the
> regionserver side.
>
> good luck,
> -ryan
>
>
> On Sat, Feb 21, 2009 at 2:21 AM, Amandeep Khurana <amansk@gmail.com>
> wrote:
>
> > I have 1 master + 2 slaves. I did set the timout to zero. I'll set the
> > xceivers to 2047 and try again. Can this be done in the job config or
> does
> > the site.xml need to be changed and the cluster restarted?
> >
> > Amandeep
> >
> >
> > Amandeep Khurana
> > Computer Science Graduate Student
> > University of California, Santa Cruz
> >
> >
> > On Sat, Feb 21, 2009 at 2:16 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
> >
> > > So the usual suspects are:
> > >
> > > - xcievers (i hvae mine set to 2047)
> > > - timeout (i have mine set to 0)
> > >
> > > I can import a few hundred million records with these settings.
> > >
> > > how many nodes do you have again?
> > >
> > > On Sat, Feb 21, 2009 at 2:14 AM, Amandeep Khurana <amansk@gmail.com>
> > > wrote:
> > >
> > > > Yes, I noticed it this time. The regionserver gets slow or stops
> > > responding
> > > > and then this error comes. How do I get this to work? Is there a way
> of
> > > > limiting the resources that the map red job should take?
> > > >
> > > > I did make the changes in the config site similar to Larry Comton's
> > > config.
> > > > It only made the job go from dying at 7% to 12% this time.
> > > >
> > > > Amandeep
> > > >
> > > >
> > > > Amandeep Khurana
> > > > Computer Science Graduate Student
> > > > University of California, Santa Cruz
> > > >
> > > >
> > > > On Sat, Feb 21, 2009 at 1:14 AM, stack <stack@duboce.net> wrote:
> > > >
> > > > > It looks like regionserver hosting root crashed:
> > > > >
> > > > > org.apache.hadoop.hbase.client.NoServerForRegionException: Timed
> out
> > > > trying
> > > > > to locate root region
> > > > >
> > > > > How many servers you running?
> > > > >
> > > > > You made similar config. to that reported by Larry Compton in a
> mail
> > > from
> > > > > earlier today?  (See FAQ and Troubleshooting page for more on his
> > > listed
> > > > > configs.)
> > > > >
> > > > > St.Ack
> > > > >
> > > > >
> > > > > On Sat, Feb 21, 2009 at 1:01 AM, Amandeep Khurana <
> amansk@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Yes, the table exists before I start the job.
> > > > > >
> > > > > > I am not using TableOutputFormat. I picked up the sample code
> from
> > > the
> > > > > docs
> > > > > > and am using it.
> > > > > >
> > > > > > Here's the job conf:
> > > > > >
> > > > > > JobConf conf = new JobConf(getConf(), IN_TABLE_IMPORT.class);
> > > > > >        FileInputFormat.setInputPaths(conf, new
> > Path("import_data"));
> > > > > >        conf.setMapperClass(MapClass.class);
> > > > > >        conf.setNumReduceTasks(0);
> > > > > >        conf.setOutputFormat(NullOutputFormat.class);
> > > > > >        JobClient.runJob(conf);
> > > > > >
> > > > > > Interestingly, the hbase shell isnt working now either. Its
> giving
> > > > errors
> > > > > > even when I give the command "list"...
> > > > > >
> > > > > >
> > > > > >
> > > > > > Amandeep Khurana
> > > > > > Computer Science Graduate Student
> > > > > > University of California, Santa Cruz
> > > > > >
> > > > > >
> > > > > > On Sat, Feb 21, 2009 at 12:10 AM, stack <stack@duboce.net>
> wrote:
> > > > > >
> > > > > > > The table exists before you start the MR job?
> > > > > > >
> > > > > > > When you say 'midway through the job', are you using
> > > > tableoutputformat
> > > > > to
> > > > > > > insert into your table?
> > > > > > >
> > > > > > > Which version of hbase?
> > > > > > >
> > > > > > > St.Ack
> > > > > > >
> > > > > > > On Fri, Feb 20, 2009 at 9:55 PM, Amandeep Khurana <
> > > amansk@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I dont know if this is related or not, but it seems
to be.
> > After
> > > > this
> > > > > > map
> > > > > > > > reduce job, I tried to count the number of entries
in the
> table
> > > in
> > > > > > hbase
> > > > > > > > through the shell. It failed with the following error:
> > > > > > > >
> > > > > > > > hbase(main):002:0> count 'in_table'
> > > > > > > > NativeException: java.lang.NullPointerException: null
> > > > > > > >    from java.lang.String:-1:in `<init>'
> > > > > > > >    from org/apache/hadoop/hbase/util/Bytes.java:92:in
> > `toString'
> > > > > > > >    from
> > > > > > >
> > org/apache/hadoop/hbase/client/RetriesExhaustedException.java:50:in
> > > > > > > > `getMessage'
> > > > > > > >    from
> > > > > > >
> > org/apache/hadoop/hbase/client/RetriesExhaustedException.java:40:in
> > > > > > > > `<init>'
> > > > > > > >    from
> > > > org/apache/hadoop/hbase/client/HConnectionManager.java:841:in
> > > > > > > > `getRegionServerWithRetries'
> > > > > > > >    from org/apache/hadoop/hbase/client/MetaScanner.java:56:in
> > > > > > `metaScan'
> > > > > > > >    from org/apache/hadoop/hbase/client/MetaScanner.java:30:in
> > > > > > `metaScan'
> > > > > > > >    from
> > > > org/apache/hadoop/hbase/client/HConnectionManager.java:411:in
> > > > > > > > `getHTableDescriptor'
> > > > > > > >    from org/apache/hadoop/hbase/client/HTable.java:219:in
> > > > > > > > `getTableDescriptor'
> > > > > > > >    from sun.reflect.NativeMethodAccessorImpl:-2:in
`invoke0'
> > > > > > > >    from sun.reflect.NativeMethodAccessorImpl:-1:in
`invoke'
> > > > > > > >    from sun.reflect.DelegatingMethodAccessorImpl:-1:in
> `invoke'
> > > > > > > >    from java.lang.reflect.Method:-1:in `invoke'
> > > > > > > >    from org/jruby/javasupport/JavaMethod.java:250:in
> > > > > > > > `invokeWithExceptionHandling'
> > > > > > > >    from org/jruby/javasupport/JavaMethod.java:219:in
`invoke'
> > > > > > > >    from org/jruby/javasupport/JavaClass.java:416:in
`execute'
> > > > > > > > ... 145 levels...
> > > > > > > >    from
> > > org/jruby/internal/runtime/methods/DynamicMethod.java:74:in
> > > > > > > `call'
> > > > > > > >    from
> > > > org/jruby/internal/runtime/methods/CompiledMethod.java:48:in
> > > > > > > `call'
> > > > > > > >    from org/jruby/runtime/CallSite.java:123:in `cacheAndCall'
> > > > > > > >    from org/jruby/runtime/CallSite.java:298:in `call'
> > > > > > > >    from
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:429:in
> > > > > > > > `__file__'
> > > > > > > >    from
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > > > `__file__'
> > > > > > > >    from
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> ruby/hadoop/install/hbase_minus_0_dot_19_dot_0/bin//hadoop/install/hbase/bin/../bin/hirb.rb:-1:in
> > > > > > > > `load'
> > > > > > > >    from org/jruby/Ruby.java:512:in `runScript'
> > > > > > > >    from org/jruby/Ruby.java:432:in `runNormally'
> > > > > > > >    from org/jruby/Ruby.java:312:in `runFromMain'
> > > > > > > >    from org/jruby/Main.java:144:in `run'
> > > > > > > >    from org/jruby/Main.java:89:in `run'
> > > > > > > >    from org/jruby/Main.java:80:in `main'
> > > > > > > >    from /hadoop/install/hbase/bin/../bin/HBase.rb:444:in
> > `count'
> > > > > > > >    from /hadoop/install/hbase/bin/../bin/hirb.rb:348:in
> `count'
> > > > > > > >    from (hbase):3:in `binding'
> > > > > > > >
> > > > > > > >
> > > > > > > > Amandeep Khurana
> > > > > > > > Computer Science Graduate Student
> > > > > > > > University of California, Santa Cruz
> > > > > > > >
> > > > > > > >
> > > > > > > > On Fri, Feb 20, 2009 at 9:46 PM, Amandeep Khurana
<
> > > > amansk@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Here's what it throws on the console:
> > > > > > > > >
> > > > > > > > > 09/02/20 21:45:29 INFO mapred.JobClient: Task
Id :
> > > > > > > > > attempt_200902201300_0019_m_000006_0, Status
: FAILED
> > > > > > > > > java.io.IOException: table is null
> > > > > > > > >         at
> > > IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:33)
> > > > > > > > >         at
> > IN_TABLE_IMPORT$MapClass.map(IN_TABLE_IMPORT.java:1)
> > > > > > > > >         at
> > > > > org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
> > > > > > > > >         at
> > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
> > > > > > > > >         at
> > org.apache.hadoop.mapred.Child.main(Child.java:155)
> > > > > > > > >
> > > > > > > > > attempt_200902201300_0019_m_000006_0:
> > > > > > > > > org.apache.hadoop.hbase.client.NoServerForRegionException:
> > > Timed
> > > > > out
> > > > > > > > trying
> > > > > > > > > to locate root region
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:768)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:448)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:457)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.relocateRegion(HConnectionManager.java:430)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:557)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:461)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:423)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:114)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:97)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> IN_TABLE_IMPORT$MapClass.configure(IN_TABLE_IMPORT.java:120)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > >
> > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > >
> > > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:83)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > > org.apache.hadoop.mapred.MapTask.run(MapTask.java:328)
> > > > > > > > > attempt_200902201300_0019_m_000006_0:   at
> > > > > > > > > org.apache.hadoop.mapred.Child.main(Child.java:155)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Amandeep Khurana
> > > > > > > > > Computer Science Graduate Student
> > > > > > > > > University of California, Santa Cruz
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Fri, Feb 20, 2009 at 9:43 PM, Amandeep Khurana
<
> > > > > amansk@gmail.com
> > > > > > > > >wrote:
> > > > > > > > >
> > > > > > > > >> I am trying to import data from a flat file
into Hbase
> using
> > a
> > > > Map
> > > > > > > > Reduce
> > > > > > > > >> job. There are close to 2 million rows. Mid
way into the
> > job,
> > > it
> > > > > > > starts
> > > > > > > > >> giving me connection problems and eventually
kills the
> job.
> > > When
> > > > > the
> > > > > > > > error
> > > > > > > > >> comes, the hbase shell also stops working.
> > > > > > > > >>
> > > > > > > > >> This is what I get:
> > > > > > > > >>
> > > > > > > > >> 2009-02-20 21:37:14,407 INFO
> > org.apache.hadoop.ipc.HBaseClass:
> > > > > > > Retrying
> > > > > > > > connect to server: /171.69.102.52:60020. Already tried
0
> > > time(s).
> > > > > > > > >>
> > > > > > > > >> What could be going wrong?
> > > > > > > > >>
> > > > > > > > >> Amandeep
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Amandeep Khurana
> > > > > > > > >> Computer Science Graduate Student
> > > > > > > > >> University of California, Santa Cruz
> > > > > > > > >>
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message