hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kay Ousterhout <...@eecs.berkeley.edu>
Subject 5 second minimum shuffle time
Date Wed, 29 May 2013 03:00:29 GMT
Hi,

I'm running v0.23 in a large cluster, and have found that the shuffle time
for reduce tasks is always at least 5 seconds, even when the amount of data
read by the reduce task is tiny (e.g., just 18 bytes).  This shuffle time
floor suggests that there's a heartbeat interval or something that has to
elapse before the shuffle begins, but I can't find any sign of such a delay
in the code base.  Can anyone shed some light on why this is occurring?

Thanks,
Kay

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message