spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xuelin Cao <xuelincao2...@gmail.com>
Subject Can spark provide an option to start reduce stage early?
Date Tue, 03 Feb 2015 05:48:53 GMT
In hadoop MR, there is an option *mapred.reduce.slowstart.completed.maps*

which can be used to start reducer stage when X% mappers are completed. By
doing this, the data shuffling process is able to parallel with the map
process.

In a large multi-tenancy cluster, this option is usually tuned off. But, in
some cases, turn on the option could accelerate some high priority jobs.

Will spark provide similar option?

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message