mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vikas Kumar <kumar...@umn.edu>
Subject Re: How to change /tmp directory for mahout usage of map-reduce?
Date Wed, 01 Apr 2015 06:34:44 GMT
The following line specifically:

SpectralKMeansDriver.run(conf, affinities, output, vectors.size(),
noOfClusters, measure, onvergenceDelta, maxIterations, tmp, false);

where other variables are set accordingly. I can send the whole file if
required.

It shows the following in the log which helped me identify that it is using
the user tmp directory:

15/04/01 01:18:13 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
15/04/01 01:18:13 INFO input.FileInputFormat: Total input paths to process
: 1
15/04/01 01:18:13 INFO filecache.TrackerDistributedCacheManager: Creating
vector in */tmp/hadoop-vikas/mapred/local*/archive/-623590149816891030_-1428839080_1939951392/file/export/scratch/vikas/<<<<PRIVATE
DIRECTORIES>>>>>/tmp/calculations-work--3390146237769593830 with rwxr-xr-x
15/04/01 01:18:13 INFO filecache.TrackerDistributedCacheManager: Cached
file:/export/scratch/vikas/<<<<PRIVATE DIRECTORIES>>>>>/tmp/calculations/vector
as */tmp/hadoop-vikas/mapred/local*
/archive/-623590149816891030_-1428839080_1939951392/file/export/scratch/vikas/
<<<<PRIVATE DIRECTORIES>>>>>/tmp/calculations/vector
15/04/01 01:18:13 INFO mapred.JobClient: Running job:
job_local1446072412_0003
15/04/01 01:18:13 INFO mapred.LocalJobRunner: Waiting for map tasks
15/04/01 01:18:13 INFO mapred.LocalJobRunner: Starting task:
attempt_local1446072412_0003_m_000000_0
15/04/01 01:18:13 INFO mapred.Task:  Using ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3effe655
15/04/01 01:18:13 INFO mapred.MapTask: Processing split:
file:/export/scratch/vikas/<<<<PRIVATE DIRECTORIES>>>>>
/tmp/calculations/seqfile/part-r-00000:0+86000
15/04/01 01:18:13 INFO spectral.VectorCache: Files are:
[file:/tmp/hadoop-vikas/mapred/local/archive/-623590149816891030_-1428839080_1939951392/file/export/scratch/vikas/
<<<<PRIVATE DIRECTORIES>>>>>/tmp/calculations/vector]
15/04/01 01:18:13 INFO spectral.VectorCache: Loading vector from: file:
*/tmp/hadoop-vikas/mapred/local/archive*
/-623590149816891030_-1428839080_1939951392/file/export/scratch/vikas/<<<<
PRIVATE DIRECTORIES>>>>>/tmp/calculations/vector


Thanks
Vikas



On Wed, Apr 1, 2015 at 12:55 AM, Andrew Musselman <
andrew.musselman@gmail.com> wrote:

> Can you let us know which code/scripts you're using?
>
> On Tuesday, March 31, 2015, Vikas Kumar <kumar093@umn.edu> wrote:
>
> > Hello,
> >
> > I am using Mahout Spectral clustering example which internally calls a
> map
> > reduce job. Right now, it is using */tmp/hadoop-<username>/mapred/..*
> > directory by default for its operations.
> >
> > Can someone please let me know how to make mahout to use a different
> path?
> >
> > Thanks
> > Vikas
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message