hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liyin Liang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5691) Throttle shuffle's bandwidth utilization
Date Thu, 19 Dec 2013 07:17:06 GMT
Liyin Liang created MAPREDUCE-5691:
--------------------------------------

             Summary: Throttle shuffle's bandwidth utilization
                 Key: MAPREDUCE-5691
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5691
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
    Affects Versions: 2.2.0
            Reporter: Liyin Liang


In our hadoop cluster, a reducer of a big job can utilize all the bandwidth during shuffle
phase. Then any task reading data from  the machine which running that reducer becomes very
very slow.
It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common. And create a
throttler for Shuffle to throttle each Fetcher.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message