mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Musselman <andrew.mussel...@gmail.com>
Subject Re: Mahout error : seq2sparse
Date Thu, 04 Feb 2016 04:38:00 GMT
Ah; looks like that config can be set in Hadoop's core-site.xml but if
you're running Mahout in local mode that shouldn't help.

Can you try this with local mode off, in other words on a running
Hadoop/Spark cluster?

Looking for empty lines could be run via a command like `grep -r "^$"
input-file-directory`; blank lines will show up before your next prompt if
so.

On Wed, Feb 3, 2016 at 8:30 PM, Alok Tanna <tannaalok@gmail.com> wrote:

> Thank you Andrew for the quick response . I have around 300 input files.
> It would take a while for me to go though each file. I will try to look
> into that, but then I had successfully generated the sequence file use mahout
> seqdirectory for the same dataset. How can I find which mahout release I am
> on? also can you let me know how can I increase io.sort.mb = 100 when I
> have Mahout running in local mode.
>
> In the earlier attach file you can see it says 16/02/03 22:59:04 INFO
> mapred.MapTask: Record too large for in-memory buffer: 99614722 bytes
>
> How can I increase in-memory buffer for Mahout local mode.
>
> I hope this has nothing to do with this error.
>
> Thanks,
> Alok Tanna
>
> On Wed, Feb 3, 2016 at 10:50 PM, Andrew Musselman <
> andrew.musselman@gmail.com> wrote:
>
>> Is it possible you have any empty lines or extra whitespace at the end or
>> in the middle of any of your input files?  I don't know for sure but
>> that's
>> where I'd start looking.
>>
>> Are you on the most recent release?
>>
>> On Wed, Feb 3, 2016 at 7:33 PM, Alok Tanna <tannaalok@gmail.com> wrote:
>>
>> > Mahout in local mode
>> >
>> > I am able to successfully run the below command on smaller data set, but
>> > then when I am running this command on large data set I am getting below
>> > error.  Its looks like I need to increase size of some parameter but
>> then I
>> > am not sure which one.  It is failing with this error
>> java.io.EOFException
>> >   which creating the dictionary-0 file
>> >
>> > Please fine the attached file for more details.
>> >
>> > command: mahout seq2sparse -i /home/ubuntu/AT/AT-Seq/ -o
>> > /home/ubuntu/AT/AT-vectors/ -lnorm -nv -wt tfidf
>> >
>> > Main error :
>> >
>> >
>> > 16/02/03 23:02:06 INFO mapred.LocalJobRunner: reduce > reduce
>> > 16/02/03 23:02:17 INFO mapred.LocalJobRunner: reduce > reduce
>> > 16/02/03 23:02:18 WARN mapred.LocalJobRunner: job_local1308764206_0003
>> > java.io.EOFException
>> >         at java.io.DataInputStream.readByte(DataInputStream.java:267)
>> >         at
>> > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:299)
>> >         at
>> > org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:320)
>> >         at org.apache.hadoop.io.Text.readFields(Text.java:263)
>> >         at
>> > org.apache.mahout.common.StringTuple.readFields(StringTuple.java:142)
>> >         at
>> >
>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
>> >         at
>> >
>> org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
>> >         at
>> >
>> org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:117)
>> >         at
>> > org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
>> >         at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:176)
>> >         at
>> > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649)
>> >         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
>> >         at
>> > org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
>> > 16/02/03 23:02:18 INFO mapred.JobClient: Job complete:
>> > job_local1308764206_0003
>> > 16/02/03 23:02:18 INFO mapred.JobClient: Counters: 20
>> > 16/02/03 23:02:18 INFO mapred.JobClient:   File Output Format Counters
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Bytes Written=14923244
>> > 16/02/03 23:02:18 INFO mapred.JobClient:   FileSystemCounters
>> > 16/02/03 23:02:18 INFO mapred.JobClient:
>>  FILE_BYTES_READ=1412144036729
>> > 16/02/03 23:02:18 INFO mapred.JobClient:
>> > FILE_BYTES_WRITTEN=323876626568
>> > 16/02/03 23:02:18 INFO mapred.JobClient:   File Input Format Counters
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Bytes Read=11885543289
>> > 16/02/03 23:02:18 INFO mapred.JobClient:   Map-Reduce Framework
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Reduce input groups=223
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Map output materialized
>> > bytes=2214020551
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Combine output records=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Map input records=223
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Reduce shuffle bytes=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Physical memory (bytes)
>> > snapshot=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Reduce output records=222
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Spilled Records=638
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Map output bytes=2214019100
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     CPU time spent (ms)=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Total committed heap usaAT
>> > (bytes)=735978192896
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Virtual memory (bytes)
>> > snapshot=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Combine input records=0
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Map output records=223
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     SPLIT_RAW_BYTES=9100
>> > 16/02/03 23:02:18 INFO mapred.JobClient:     Reduce input records=222
>> > Exception in thread "main" java.lang.IllegalStateException: Job failed!
>> >         at
>> >
>> org.apache.mahout.vectorizer.DictionaryVectorizer.makePartialVectors(DictionaryVectorizer.java:329)
>> >         at
>> >
>> org.apache.mahout.vectorizer.DictionaryVectorizer.createTermFrequencyVectors(DictionaryVectorizer.java:199)
>> >         at
>> >
>> org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.run(SparseVectorsFromSequenceFiles.java:274)
>> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>> >         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>> >         at
>> >
>> org.apache.mahout.vectorizer.SparseVectorsFromSequenceFiles.main(SparseVectorsFromSequenceFiles.java:56)
>> >         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >         at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> >         at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> >         at java.lang.reflect.Method.invoke(Method.java:606)
>> >         at
>> >
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>> >         at
>> > org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>> >         at
>> > org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195)
>> > .
>> > .
>> >
>> >
>> >
>> > --
>> > Thanks & Regards,
>> >
>> > Alok R. Tanna
>> >
>> >
>>
>
>
>
> --
> Thanks & Regards,
>
> Alok R. Tanna
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message