mahout-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <sro...@gmail.com>
Subject Re: mapreduce memory issues
Date Thu, 06 May 2010 10:36:13 GMT
Oh duh, I was calling Matcher.matches() instead of Matcher.find(),
which caused it to only match the whole string. I can fix that and
improve this regex, and make the default buffer smaller.

Where do you see it setting mappers and reducers? it shouldn't, and
doesn't as far as I can see. Hadoop may be overriding your settings.

On Wed, May 5, 2010 at 11:11 PM, Tamas Jambor <jamborta@googlemail.com> wrote:
> oh ok. this is the problem. I set mapred.child.java.opts to -Xmx1024m, but
> that doesn't match the regex expression, so it sets io.sort.mb to 1000.
> Changing it to -Xmx([0-9]+)m solved the problem.
>
> On 05/05/2010 22:26, Sean Owen wrote:
>>
>> You are welcome to test this by modifying setIOSort() in
>> RecommenderJob. Just crank down that "1000", or divide heapMB by a
>> larger value. This should let you run with somewhat smaller heap
>> sizes. If that's the culprit I'm happy to turn down these values a
>> bit.
>>
>

Mime
View raw message