mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Lyubimov <dlie...@gmail.com>
Subject Re: SSVD error
Date Thu, 06 Sep 2012 18:39:06 GMT
Actually, it turns out unit tests explicitly do setBroadcast(false)...
here's probably why. But i never remember having this problem or being
motivated to set it false by anything... Hm..

On Thu, Sep 6, 2012 at 11:37 AM, Dmitriy Lyubimov <dlieu.7@gmail.com> wrote:
> I don't believe it doesn't work in local mode because its unit tests
> are run in local mode. With exception of # of reducers, everything
> else works there just the same.
>
> That said, you can disable DistributedCache in some cases using
> SSVDSolver#setBroadcast(false). (in spite of what javadoc says, it is
> enabled by default... my bad).
>
> On Thu, Sep 6, 2012 at 11:18 AM, Pat Ferrel <pat@occamsmachete.com> wrote:
>> To reiterate the situation. In local mode using the local file system SSVD dies with
a file not found. In pseudo-cluster mode using hdfs SSVD on the same data it runs correctly.
All the rest of the analysis pipeline works fine in either mode. I am using local mode to
debug my surrounding code.
>>
>> From the error output it looks like the code is using the DistributedCache of hadoop.
This is said to not work with local hadoop, though the comment was for a pre 0.20.205 version
(my version).
>>
>> The implication being that when MAHOUT_LOCAL is set you shouldn't use the DistributedCache.
>>
>> Could this be the problem? stochasticSVD uses DistributedCache in several spots
>>
>> ==========================
>>
>> The /tmp file does not exist, the local
>>
>> java.io.FileNotFoundException: File /tmp/hadoop-pat/mapred/local/archive/5543644668644532045_1587570556_2120541978/file/Users/pat/Projects/big-data/b/ssvd/Q-job/R-m-00000
does not exist.
>>
>> Maclaurin:big-data pat$ ls -al b/ssvd/Q-job/
>> total 72
>> drwxr-xr-x  10 pat  staff   340 Aug 31 13:35 .
>> drwxr-xr-x   4 pat  staff   136 Aug 31 13:35 ..
>> -rw-r--r--   1 pat  staff    80 Aug 31 13:35 .QHat-m-00000.crc
>> -rw-r--r--   1 pat  staff    28 Aug 31 13:35 .R-m-00000.crc
>> -rw-r--r--   1 pat  staff     8 Aug 31 13:35 ._SUCCESS.crc
>> -rw-r--r--   1 pat  staff    12 Aug 31 13:35 .part-m-00000.deflate.crc
>> -rwxrwxrwx   1 pat  staff  9154 Aug 31 13:35 QHat-m-00000
>> -rwxrwxrwx   1 pat  staff  2061 Aug 31 13:35 R-m-00000
>> -rwxrwxrwx   1 pat  staff     0 Aug 31 13:35 _SUCCESS
>> -rwxrwxrwx   1 pat  staff     8 Aug 31 13:35 part-m-00000.deflate

Mime
View raw message