spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcelo Vanzin <van...@cloudera.com>
Subject Re: YARN Shuffle service and its compatibility
Date Mon, 18 Apr 2016 20:59:37 GMT
On Mon, Apr 18, 2016 at 1:53 PM, Reynold Xin <rxin@databricks.com> wrote:
> That's not the only one. For example, the hash shuffle manager has been off
> by default since Spark 1.2, and we'd like to remove it in 2.0:
> https://github.com/apache/spark/pull/12423

If I understand things correctly, Mark's option B (running a single
shuffle service, the one from the older Spark release) would still
work, wouldn't it?

You'd run into problems when Spark adds a new shuffle manager that is
not known to the old shuffle service, though. Perhaps at that time we
should investigate making the shuffle service more agnostic to the
app's shuffle manager.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org


Mime
View raw message