spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Fritzler <andreas.fritz...@gmail.com>
Subject [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Date Mon, 05 Oct 2015 13:22:42 GMT
Hi,

I was just wondering, if it is possible to register multiple versions of
the aux-services with YARN as described in the documentation:



   1. In the yarn-site.xml on each node, add spark_shuffle to
   yarn.nodemanager.aux-services, then set
   yarn.nodemanager.aux-services.spark_shuffle.class to
   org.apache.spark.network.yarn.YarnShuffleService. Additionally, set all
   relevantspark.shuffle.service.* configurations
   <http://spark.apache.org/docs/latest/configuration.html>.

The reason for the question is: I am trying to run multiple versions of
Spark in parallel. Does anybody have any experience on how such a dual
version operation holds up in terms of downward-compatibility?

Maybe sticking to the latest version of the aux-service will do the trick?

Regards,
Andreas

[1]
http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

Mime
View raw message