mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lee Ho Yeung <jobmatt...@gmail.com>
Subject recommenditembased has error
Date Tue, 21 Jun 2016 05:16:44 GMT
just try to guess value in a table or matrix

firstly is that do not know where the result file is.
secondly is that it seems have error

vi rdata.txt
1,1,5
1,2,4
1,3,5
2,1,4
2,2,5
2,3,4
3,1,5
3,2,4
4,1,1
4,2,2
5,1,2
5,2,1
5,3,1

hadoop-2.7.2/bin/hadoop fs -rm -r temp

mahout/bin/mahout recommenditembased -s similarity_euclidean_distance -i
/home/martin/Downloads/rdata.txt -o /home/martin/Downloads/output.txt
--numRecommendations 5

...
16/06/20 22:13:47 INFO AbstractJob: Command line arguments:
{--endPhase=[2147483647], --excludeSelfSimilarity=[true],
--input=[temp/preparePreferenceMatrix/ratingMatrix],
--maxObservationsPerColumn=[500], --maxObservationsPerRow=[500],
--maxSimilaritiesPerRow=[100], --numberOfColumns=[5],
--output=[temp/similarityMatrix], --randomSeed=[-9223372036854775808],
--similarityClassname=[similarity_euclidean_distance], --startPhase=[0],
--tempDir=[temp], --threshold=[4.9E-324]}
16/06/20 22:13:47 INFO deprecation: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
16/06/20 22:13:47 INFO deprecation: mapred.compress.map.output is
deprecated. Instead, use mapreduce.map.output.compress
16/06/20 22:13:47 INFO deprecation: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
16/06/20 22:13:47 INFO JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
16/06/20 22:13:47 INFO FileInputFormat: Total input paths to process : 1
16/06/20 22:13:47 INFO JobSubmitter: number of splits:1
16/06/20 22:13:47 INFO JobSubmitter: Submitting tokens for job:
job_local1682295755_0004
16/06/20 22:13:47 WARN Configuration:
file:/tmp/hadoop-martin/mapred/staging/martin1682295755/.staging/job_local1682295755_0004/job.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
16/06/20 22:13:47 WARN Configuration:
file:/tmp/hadoop-martin/mapred/staging/martin1682295755/.staging/job_local1682295755_0004/job.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
16/06/20 22:13:48 WARN Configuration:
file:/tmp/hadoop-martin/mapred/local/localRunner/martin/job_local1682295755_0004/job_local1682295755_0004.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
16/06/20 22:13:48 WARN Configuration:
file:/tmp/hadoop-martin/mapred/local/localRunner/martin/job_local1682295755_0004/job_local1682295755_0004.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
16/06/20 22:13:48 INFO Job: The url to track the job: http://localhost:8080/
16/06/20 22:13:48 INFO Job: Running job: job_local1682295755_0004
16/06/20 22:13:48 INFO LocalJobRunner: OutputCommitter set in config null
16/06/20 22:13:48 INFO LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/06/20 22:13:48 INFO LocalJobRunner: Waiting for map tasks
16/06/20 22:13:48 INFO LocalJobRunner: Starting task:
attempt_local1682295755_0004_m_000000_0
16/06/20 22:13:48 INFO Task:  Using ResourceCalculatorProcessTree : [ ]
16/06/20 22:13:48 INFO MapTask: Processing split:
file:/home/martin/Downloads/temp/preparePreferenceMatrix/ratingMatrix/part-r-00000:0+219
16/06/20 22:13:48 INFO MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/06/20 22:13:48 INFO MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/06/20 22:13:48 INFO MapTask: mapreduce.task.io.sort.mb: 100
16/06/20 22:13:48 INFO MapTask: soft limit at 83886080
16/06/20 22:13:48 INFO MapTask: bufstart = 0; bufvoid = 104857600
16/06/20 22:13:48 INFO MapTask: kvstart = 26214396; length = 6553600
16/06/20 22:13:48 INFO LocalJobRunner:
16/06/20 22:13:48 INFO MapTask: Starting flush of map output
16/06/20 22:13:48 INFO MapTask: Spilling map output
16/06/20 22:13:48 INFO MapTask: bufstart = 0; bufend = 52; bufvoid =
104857600
16/06/20 22:13:48 INFO MapTask: kvstart = 26214396(104857584); kvend =
26214396(104857584); length = 1/6553600
16/06/20 22:13:48 INFO MapTask: Finished spill 0
16/06/20 22:13:48 INFO Task: Task:attempt_local1682295755_0004_m_000000_0
is done. And is in the process of committing
16/06/20 22:13:48 INFO LocalJobRunner: map
16/06/20 22:13:48 INFO Task: Task 'attempt_local1682295755_0004_m_000000_0'
done.
16/06/20 22:13:48 INFO LocalJobRunner: Finishing task:
attempt_local1682295755_0004_m_000000_0
16/06/20 22:13:48 INFO LocalJobRunner: map task executor complete.
16/06/20 22:13:48 INFO LocalJobRunner: Waiting for reduce tasks
16/06/20 22:13:48 INFO LocalJobRunner: Starting task:
attempt_local1682295755_0004_r_000000_0
16/06/20 22:13:48 INFO Task:  Using ResourceCalculatorProcessTree : [ ]
16/06/20 22:13:48 INFO ReduceTask: Using ShuffleConsumerPlugin:
org.apache.hadoop.mapreduce.task.reduce.Shuffle@14f5a62
16/06/20 22:13:48 INFO MergeManagerImpl: MergerManager:
memoryLimit=363285696, maxSingleShuffleLimit=90821424,
mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
16/06/20 22:13:48 INFO EventFetcher:
attempt_local1682295755_0004_r_000000_0 Thread started: EventFetcher for
fetching Map Completion Events
16/06/20 22:13:48 INFO LocalFetcher: localfetcher#4 about to shuffle output
of map attempt_local1682295755_0004_m_000000_0 decomp: 56 len: 40 to MEMORY
16/06/20 22:13:48 INFO InMemoryMapOutput: Read 56 bytes from map-output for
attempt_local1682295755_0004_m_000000_0
16/06/20 22:13:48 INFO MergeManagerImpl: closeInMemoryFile -> map-output of
size: 56, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->56
16/06/20 22:13:48 INFO EventFetcher: EventFetcher is interrupted.. Returning
16/06/20 22:13:48 INFO LocalJobRunner: 1 / 1 copied.
16/06/20 22:13:48 INFO MergeManagerImpl: finalMerge called with 1 in-memory
map-outputs and 0 on-disk map-outputs
16/06/20 22:13:48 INFO Merger: Merging 1 sorted segments
16/06/20 22:13:48 INFO Merger: Down to the last merge-pass, with 1 segments
left of total size: 54 bytes
16/06/20 22:13:48 INFO MergeManagerImpl: Merged 1 segments, 56 bytes to
disk to satisfy reduce memory limit
16/06/20 22:13:48 INFO MergeManagerImpl: Merging 1 files, 48 bytes from disk
16/06/20 22:13:48 INFO MergeManagerImpl: Merging 0 segments, 0 bytes from
memory into reduce
16/06/20 22:13:48 INFO Merger: Merging 1 sorted segments
16/06/20 22:13:48 INFO Merger: Down to the last merge-pass, with 1 segments
left of total size: 12 bytes
16/06/20 22:13:48 INFO LocalJobRunner: 1 / 1 copied.
16/06/20 22:13:48 INFO Task: Task:attempt_local1682295755_0004_r_000000_0
is done. And is in the process of committing
16/06/20 22:13:48 INFO LocalJobRunner: 1 / 1 copied.
16/06/20 22:13:48 INFO Task: Task attempt_local1682295755_0004_r_000000_0
is allowed to commit now
16/06/20 22:13:48 INFO FileOutputCommitter: Saved output of task
'attempt_local1682295755_0004_r_000000_0' to
file:/home/martin/Downloads/temp/notUsed/_temporary/0/task_local1682295755_0004_r_000000
16/06/20 22:13:48 INFO LocalJobRunner: reduce > reduce
16/06/20 22:13:48 INFO Task: Task 'attempt_local1682295755_0004_r_000000_0'
done.
16/06/20 22:13:48 INFO LocalJobRunner: Finishing task:
attempt_local1682295755_0004_r_000000_0
16/06/20 22:13:48 INFO LocalJobRunner: reduce task executor complete.
16/06/20 22:13:49 INFO Job: Job job_local1682295755_0004 running in uber
mode : false
16/06/20 22:13:49 INFO Job:  map 100% reduce 100%
16/06/20 22:13:49 INFO Job: Job job_local1682295755_0004 completed
successfully
16/06/20 22:13:49 INFO Job: Counters: 33
File System Counters
FILE: Number of bytes read=608810760
FILE: Number of bytes written=615360192
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
Map-Reduce Framework
Map input records=3
Map output records=1
Map output bytes=52
Map output materialized bytes=40
Input split bytes=147
Combine input records=1
Combine output records=1
Reduce input groups=1
Reduce shuffle bytes=40
Reduce input records=1
Reduce output records=0
Spilled Records=2
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=34
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0
Total committed heap usage (bytes)=389464064
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=231
File Output Format Counters
Bytes Written=110
16/06/20 22:13:49 INFO deprecation: mapred.input.dir is deprecated.
Instead, use mapreduce.input.fileinputformat.inputdir
16/06/20 22:13:49 INFO deprecation: mapred.compress.map.output is
deprecated. Instead, use mapreduce.map.output.compress
16/06/20 22:13:49 INFO deprecation: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
16/06/20 22:13:49 INFO JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
16/06/20 22:13:49 INFO FileInputFormat: Total input paths to process : 1
16/06/20 22:13:49 INFO JobSubmitter: number of splits:1
16/06/20 22:13:49 INFO JobSubmitter: Submitting tokens for job:
job_local1242220696_0005
16/06/20 22:13:49 WARN Configuration:
file:/tmp/hadoop-martin/mapred/staging/martin1242220696/.staging/job_local1242220696_0005/job.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
16/06/20 22:13:49 WARN Configuration:
file:/tmp/hadoop-martin/mapred/staging/martin1242220696/.staging/job_local1242220696_0005/job.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
16/06/20 22:13:49 WARN Configuration:
file:/tmp/hadoop-martin/mapred/local/localRunner/martin/job_local1242220696_0005/job_local1242220696_0005.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval;  Ignoring.
16/06/20 22:13:49 WARN Configuration:
file:/tmp/hadoop-martin/mapred/local/localRunner/martin/job_local1242220696_0005/job_local1242220696_0005.xml:an
attempt to override final parameter:
mapreduce.job.end-notification.max.attempts;  Ignoring.
16/06/20 22:13:49 INFO Job: The url to track the job: http://localhost:8080/
16/06/20 22:13:49 INFO Job: Running job: job_local1242220696_0005
16/06/20 22:13:49 INFO LocalJobRunner: OutputCommitter set in config null
16/06/20 22:13:49 INFO LocalJobRunner: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
16/06/20 22:13:49 INFO LocalJobRunner: Waiting for map tasks
16/06/20 22:13:49 INFO LocalJobRunner: Starting task:
attempt_local1242220696_0005_m_000000_0
16/06/20 22:13:49 INFO Task:  Using ResourceCalculatorProcessTree : [ ]
16/06/20 22:13:49 INFO MapTask: Processing split:
file:/home/martin/Downloads/temp/preparePreferenceMatrix/ratingMatrix/part-r-00000:0+219
16/06/20 22:13:49 INFO MapTask: Map output collector class =
org.apache.hadoop.mapred.MapTask$MapOutputBuffer
16/06/20 22:13:50 INFO MapTask: (EQUATOR) 0 kvi 26214396(104857584)
16/06/20 22:13:50 INFO MapTask: mapreduce.task.io.sort.mb: 100
16/06/20 22:13:50 INFO MapTask: soft limit at 83886080
16/06/20 22:13:50 INFO MapTask: bufstart = 0; bufvoid = 104857600
16/06/20 22:13:50 INFO MapTask: kvstart = 26214396; length = 6553600
16/06/20 22:13:50 INFO MapTask: Starting flush of map output
16/06/20 22:13:50 INFO LocalJobRunner: map task executor complete.
16/06/20 22:13:50 WARN LocalJobRunner: job_local1242220696_0005
java.lang.Exception: java.lang.IllegalStateException:
java.lang.ClassNotFoundException: similarity_euclidean_distance
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.IllegalStateException:
java.lang.ClassNotFoundException: similarity_euclidean_distance
at org.apache.mahout.common.ClassUtils.instantiateAs(ClassUtils.java:30)
at
org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob$VectorNormMapper.setup(RowSimilarityJob.java:270)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: similarity_euclidean_distance
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at org.apache.mahout.common.ClassUtils.instantiateAs(ClassUtils.java:28)
... 10 more
16/06/20 22:13:50 INFO Job: Job job_local1242220696_0005 running in uber
mode : false
16/06/20 22:13:50 INFO Job:  map 0% reduce 0%
16/06/20 22:13:50 INFO Job: Job job_local1242220696_0005 failed with state
FAILED due to: NA
16/06/20 22:13:50 INFO Job: Counters: 0
16/06/20 22:13:50 INFO deprecation: mapred.compress.map.output is
deprecated. Instead, use mapreduce.map.output.compress
16/06/20 22:13:50 INFO deprecation: mapred.output.dir is deprecated.
Instead, use mapreduce.output.fileoutputformat.outputdir
16/06/20 22:13:50 INFO JvmMetrics: Cannot initialize JVM Metrics with
processName=JobTracker, sessionId= - already initialized
16/06/20 22:13:51 INFO FileInputFormat: Total input paths to process : 1
16/06/20 22:13:51 INFO JobSubmitter: Cleaning up the staging area
file:/tmp/hadoop-martin/mapred/staging/martin1737468057/.staging/job_local1737468057_0006
Exception in thread "main"
org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path
does not exist: file:/home/martin/Downloads/temp/similarityMatrix
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:320)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:263)
at
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
at
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:375)
at
org.apache.hadoop.mapreduce.lib.input.DelegatingInputFormat.getSplits(DelegatingInputFormat.java:115)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:493)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:510)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1303)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.run(RecommenderJob.java:249)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at
org.apache.mahout.cf.taste.hadoop.item.RecommenderJob.main(RecommenderJob.java:335)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:145)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:153)
at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message