Liyin Liang created MAPREDUCE-5691:
--------------------------------------
Summary: Throttle shuffle's bandwidth utilization
Key: MAPREDUCE-5691
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5691
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.2.0
Reporter: Liyin Liang
In our hadoop cluster, a reducer of a big job can utilize all the bandwidth during shuffle
phase. Then any task reading data from the machine which running that reducer becomes very
very slow.
It's better to move DataTransferThrottler from hadoop-hdfs to hadoop-common. And create a
throttler for Shuffle to throttle each Fetcher.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
|