spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andreas Fritzler <>
Subject [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Date Mon, 05 Oct 2015 13:22:42 GMT

I was just wondering, if it is possible to register multiple versions of
the aux-services with YARN as described in the documentation:

   1. In the yarn-site.xml on each node, add spark_shuffle to
   yarn.nodemanager.aux-services, then set
   yarn.nodemanager.aux-services.spark_shuffle.class to Additionally, set all
   relevantspark.shuffle.service.* configurations

The reason for the question is: I am trying to run multiple versions of
Spark in parallel. Does anybody have any experience on how such a dual
version operation holds up in terms of downward-compatibility?

Maybe sticking to the latest version of the aux-service will do the trick?



View raw message