hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Radim Kolar <...@filez.com>
Subject oom during shuffle
Date Mon, 06 May 2013 02:35:58 GMT
i retested it and oom error is also in 2.0.3, but not in 1.x.

how much memory is needed for reducer during shuffle and sort? Peak 
usage is memoryLimit or there have to be additional memory for 
mapred.reduce.parallel.copies fetch buffers?

MergerManager: memoryLimit=326264416, maxSingleShuffleLimit=81566104, 
mergeThreshold=215334528, ioSortFactor=10, memToMemMergeOutputsThreshold=10

View raw message