I have the same question about the history server. We are trying to run multiple versions of Spark and are wondering if the history server is backwards compatible.

Alex Rovner
Director, Data Engineering 


On Mon, Oct 5, 2015 at 9:22 AM, Andreas Fritzler <andreas.fritzler@gmail.com> wrote:
Hi,

I was just wondering, if it is possible to register multiple versions of the aux-services with YARN as described in the documentation:


  1. In the yarn-site.xml on each node, add spark_shuffle to yarn.nodemanager.aux-services, then set yarn.nodemanager.aux-services.spark_shuffle.class to org.apache.spark.network.yarn.YarnShuffleService. Additionally, set all relevantspark.shuffle.service.* configurations.
The reason for the question is: I am trying to run multiple versions of Spark in parallel. Does anybody have any experience on how such a dual version operation holds up in terms of downward-compatibility?

Maybe sticking to the latest version of the aux-service will do the trick?

Regards,
Andreas