hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: Command line integration question
Date Sun, 05 Dec 2010 23:37:11 GMT
Ok, i think i got it. Mahout uses standard ToolRunner to preconfigure the
client. Got it. Thanks.

On Sun, Dec 5, 2010 at 3:28 PM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:

> PS. also if I needed to play with various MR settings, such as child
> processes arguments, could i pass that on to Configuration object thru a
> command line? Or i would have to add a definition for a custom job setting
> for every instance where i'd want to supply a custom MR setting?
>
>
> Thank you in advance.
> -Dmitriy
>
>
>
> On Sun, Dec 5, 2010 at 3:17 PM, Dmitriy Lyubimov <dlieu.7@gmail.com>wrote:
>
>> Dear all,
>>
>> I am testing the command line integration for the SSVD patch in hadoop
>> mode and running into some difficulties.
>> Even that I defined $HADDOP_HOME and $HADOOP_CONF_DIR, apparently dfs
>> configuration is not being picked up.
>>
>> I do run on CDH3b3, however, all hadoop configuration is 100% compatible
>> with 0.20. I am using AbstractJob.getConf() to acquire initial properties
>> but it looks like fs.default.name is not being set still. And i tried to
>> locate theh code that loads that hadoop conf but wasn't immediately able to
>> find it. Could you please help me what i need to do to retrieve initial
>> hadoop configuration correctly? I am missing something very simple here.
>>
>> Thank you in advance.
>> -Dmitriy
>>
>> bin/mahout ssvd -i /mahout/ssvdtest/A -o /mahout/ssvd-out/1 -k 100 -p 100
>> -r 200
>> Running on hadoop, using HADOOP_HOME=/home/dmitriy/tools/hadoop
>> HADOOP_CONF_DIR=/home/dmitriy/tools/hadoop/conf
>> 10/12/05 15:09:55 INFO common.AbstractJob: Command line arguments:
>> {--blockHeight=200, --computeU=true, --computeV=true, --endPhase=2147483647,
>> --input=/mahout/ssvdtest/A, --minSplitSize=-1, --output=/mahout/ssvd-out/1,
>> --oversampling=100, --rank=100, --reduceTasks=1, --startPhase=0,
>> --tempDir=temp}
>> Exception in thread "main" java.lang.NullPointerException
>>         at
>> org.apache.hadoop.fs.FileSystem.getDefaultUri(FileSystem.java:118)
>>         at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:110)
>>         at
>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDSolver.run(SSVDSolver.java:177)
>>         at
>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli.run(SSVDCli.java:75)
>>         at
>> org.apache.mahout.math.hadoop.stochasticsvd.SSVDCli.main(SSVDCli.java:108)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>         at
>> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>>         at
>> org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:182)
>>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>         at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>         at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>         at java.lang.reflect.Method.invoke(Method.java:597)
>>         at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message