hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From George Stathis <gstat...@gmail.com>
Subject Re: org.apache.hadoop.hbase.mapreduce.Export fails with an NPE
Date Sat, 10 Apr 2010 16:11:37 GMT
OK, the issue remains in our Ubuntu EC2 dev environment, so it's not just my
local setup. Here are some more observations based on some tests I just ran:

   - If the zookeeper JAR is omitted from HADOOP_CLASSPATH, then there are
   ClassNotFoundExceptions thrown as would be expected
   - If the zookeeper JAR is included in HADOOP_CLASSPATH,
   the ClassNotFoundExceptions go away, but then the original NPE
   re-appears: org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:110)
   - If the zookeeper JAR in physically included in $HADOOP_HOME/lib, then
   the NPE goes away as well

So, while it seems that the HADOOP_CLASSPATH is indeed being read, something
is missing during the MapRed process that keeps the htable from being
instantiated properly in TableInputFormatBase unless some JARs are
physically present in $HADOOP_HOME/lib. Note that this issue is not specific
to the zookeeper JAR either. We have enabled the transactional contrib
indexed tables and we have the same problem if we don't physically
include hbase-transactional-0.20.3.jar in the hadoop lib even though it's
included in HADOOP_CLASSPATH.

It feels like there is a discrepancy in the way classloading is done between
the various components. But I'm not sure whether this is even an HBase issue
and not a Hadoop one. Seems like this might be a JIRA ticket candidate. Any
thoughts on which project should look at this first?

-GS

On Fri, Apr 9, 2010 at 8:29 PM, George Stathis <gstathis@gmail.com> wrote:

> Here is mine:
>
> export
> HADOOP_CLASSPATH="$HBASE_HOME/hbase-0.20.3.jar:$HBASE_HOME/hbase-0.20.3-test.jar:$HBASE_HOME/lib/zookeeper-3.2.2.jar:$HBASE_HOME/conf"
>
> $HBASE_HOME is defined in my .bash_profile, so it's already there and I see
> it expanded in the debug statements with the correct path. I even tried
> hard-coding the $HBASE_HOME path above just in case and I had the same
> issue.
>
> I any case, I'm passed it now. I'll have to check whether the same issue
> happens on our dev environment running on Ubuntu on EC2. If not, then at
> least it's localized to my OSX environment.
>
> -GS
>
>
> On Fri, Apr 9, 2010 at 7:32 PM, Stack <stack@duboce.net> wrote:
>
>> Very odd.  I don't have to do that running MR jobs.  I wonder whats
>> different? (I'm using 0.20.4 near-candidate rather than 0.20.3,
>> 1.6.0u14).  I have a HADOOP_ENV like this.
>>
>> export HBASE_HOME=/home/hadoop/0.20
>> export HBASE_VERSION=20.4-dev
>> #export
>> HADOOP_CLASSPATH="$HBASE_HOME/conf:$HBASE_HOME/build/hbase-0.20.4-dev.jar:$HBASE_HOME/build/hbase-0.20.4-dev-test.jar:$HBASE_HOME/lib/zookeeper-3.2.2.jar"
>> export
>> HADOOP_CLASSPATH="$HBASE_HOME/conf:$HBASE_HOME/build/hbase-0.${HBASE_VERSION}.jar:$HBASE_HOME/build/hbase-0.${HBASE_VERSION}-test.jar:$HBASE_HOME/lib/zookeeper-3.2.2.jar"
>>
>> St.Ack
>>
>> On Fri, Apr 9, 2010 at 4:19 PM, George Stathis <gstathis@gmail.com>
>> wrote:
>> > Solved: for those interested, I had to explicitly copy
>> zookeeper-3.2.2.jar
>> > to $HADOOP_HOME/lib even though I had added its' path to
>> $HADOOP_CLASSPATH
>> > under $HADOOP_HOME/conf/hadoop-env.sh.
>> >
>> > It makes no sense to me why that particular JAR would not get picked up.
>> It
>> > was even listed in the classpath debug output when I ran the job using
>> the
>> > hadoop shell script. If anyone can enlighten, please do.
>> >
>> > -GS
>> >
>> > On Fri, Apr 9, 2010 at 5:56 PM, George Stathis <gstathis@gmail.com>
>> wrote:
>> >
>> >> No dice. Classpath is now set. Same error. Meanwhile, I'm running "$
>> hadoop
>> >> org.apache.hadoop.hbase.PerformanceEvaluation sequentialWrite 1" just
>> fine,
>> >> so MapRed is working at least.
>> >>
>> >> Still looking for suggestions then I guess.
>> >>
>> >> -GS
>> >>
>> >>
>> >> On Fri, Apr 9, 2010 at 5:31 PM, George Stathis <gstathis@gmail.com>
>> wrote:
>> >>
>> >>> RTFMing
>> >>>
>> http://hadoop.apache.org/hbase/docs/current/api/org/apache/hadoop/hbase/mapreduce/package-summary.htmlright
>> >>> now...Hadoop classpath not being set properly could be the issue...
>> >>>
>> >>>
>> >>> On Fri, Apr 9, 2010 at 5:26 PM, George Stathis <gstathis@gmail.com
>> >wrote:
>> >>>
>> >>>> Hi folks,
>> >>>>
>> >>>> I hope this is just a newbie problem.
>> >>>>
>> >>>> Context:
>> >>>> - Running 0.20.3 tag locally in pseudo cluster mode
>> >>>> - $HBASE_HOME is in env and $PATH
>> >>>> - Running org.apache.hadoop.hbase.mapreduce.Export in the shell
such
>> >>>> as: $ hbase org.apache.hadoop.hbase.mapreduce.Export channels
>> >>>> /bkps/channels/01
>> >>>>
>> >>>> Symptom:
>> >>>> - Getting an NPE at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:110):
>> >>>>
>> >>>> [...]
>> >>>> 110      this.scanner = this.htable.getScanner(newScan);
>> >>>> [...]
>> >>>>
>> >>>> Full output is bellow. Not sure why htable is still null at that
>> point.
>> >>>> User error?
>> >>>>
>> >>>> Any help is appreciated.
>> >>>>
>> >>>> -GS
>> >>>>
>> >>>> Full output:
>> >>>>
>> >>>> $ hbase org.apache.hadoop.hbase.mapreduce.Export channels
>> >>>> /bkps/channels/01
>> >>>> 2010-04-09 17:13:57.407::INFO:  Logging to STDERR via
>> >>>> org.mortbay.log.StdErrLog
>> >>>> 2010-04-09 17:13:57.408::INFO:  verisons=1, starttime=0,
>> >>>> endtime=9223372036854775807
>> >>>> 10/04/09 17:13:58 DEBUG zookeeper.ZooKeeperWrapper: Read ZNode
>> >>>> /hbase/root-region-server got 192.168.1.16:52159
>> >>>> 10/04/09 17:13:58 DEBUG client.HConnectionManager$TableServers:
Found
>> >>>> ROOT at 192.168.1.16:52159
>> >>>> 10/04/09 17:13:58 DEBUG client.HConnectionManager$TableServers:
>> Cached
>> >>>> location for .META.,,1 is 192.168.1.16:52159
>> >>>> 10/04/09 17:13:58 DEBUG client.HConnectionManager$TableServers:
>> Cached
>> >>>> location for channels,,1270753106916 is 192.168.1.16:52159
>> >>>> 10/04/09 17:13:58 DEBUG client.HConnectionManager$TableServers:
Cache
>> hit
>> >>>> for row <> in tableName channels: location server 192.168.1.16:52159
>> ,
>> >>>> location region name channels,,1270753106916
>> >>>> 10/04/09 17:13:58 DEBUG mapreduce.TableInputFormatBase: getSplits:
>> split
>> >>>> -> 0 -> 192.168.1.16:,
>> >>>> 10/04/09 17:13:58 INFO mapred.JobClient: Running job:
>> >>>> job_201004091642_0009
>> >>>> 10/04/09 17:13:59 INFO mapred.JobClient:  map 0% reduce 0%
>> >>>> 10/04/09 17:14:09 INFO mapred.JobClient: Task Id :
>> >>>> attempt_201004091642_0009_m_000000_0, Status : FAILED
>> >>>> java.lang.NullPointerException
>> >>>>  at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:110)
>> >>>> at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.init(TableInputFormatBase.java:119)
>> >>>>  at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:262)
>> >>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>> >>>>  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >>>> at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >>>>
>> >>>> 10/04/09 17:14:15 INFO mapred.JobClient: Task Id :
>> >>>> attempt_201004091642_0009_m_000000_1, Status : FAILED
>> >>>> java.lang.NullPointerException
>> >>>> at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:110)
>> >>>>  at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.init(TableInputFormatBase.java:119)
>> >>>> at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:262)
>> >>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>> >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >>>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >>>>
>> >>>> 10/04/09 17:14:21 INFO mapred.JobClient: Task Id :
>> >>>> attempt_201004091642_0009_m_000000_2, Status : FAILED
>> >>>> java.lang.NullPointerException
>> >>>> at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.restart(TableInputFormatBase.java:110)
>> >>>>  at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase$TableRecordReader.init(TableInputFormatBase.java:119)
>> >>>> at
>> >>>>
>> org.apache.hadoop.hbase.mapreduce.TableInputFormatBase.createRecordReader(TableInputFormatBase.java:262)
>> >>>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:588)
>> >>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>> >>>>  at org.apache.hadoop.mapred.Child.main(Child.java:170)
>> >>>>
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient: Job complete:
>> >>>> job_201004091642_0009
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient: Counters: 3
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient:   Job Counters
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient:     Launched map tasks=4
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient:     Data-local map tasks=4
>> >>>> 10/04/09 17:14:30 INFO mapred.JobClient:     Failed map tasks=1
>> >>>> 10/04/09 17:14:30 DEBUG zookeeper.ZooKeeperWrapper: Closed connection
>> >>>> with ZooKeeper
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message