spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Rovner <alex.rov...@magnetic.com>
Subject Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Date Mon, 05 Oct 2015 13:59:44 GMT
I have the same question about the history server. We are trying to run
multiple versions of Spark and are wondering if the history server is
backwards compatible.

*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

On Mon, Oct 5, 2015 at 9:22 AM, Andreas Fritzler <andreas.fritzler@gmail.com
> wrote:

> Hi,
>
> I was just wondering, if it is possible to register multiple versions of
> the aux-services with YARN as described in the documentation:
>
>
>
>    1. In the yarn-site.xml on each node, add spark_shuffle to
>    yarn.nodemanager.aux-services, then set
>    yarn.nodemanager.aux-services.spark_shuffle.class to
>    org.apache.spark.network.yarn.YarnShuffleService. Additionally, set
>    all relevantspark.shuffle.service.* configurations
>    <http://spark.apache.org/docs/latest/configuration.html>.
>
> The reason for the question is: I am trying to run multiple versions of
> Spark in parallel. Does anybody have any experience on how such a dual
> version operation holds up in terms of downward-compatibility?
>
> Maybe sticking to the latest version of the aux-service will do the trick?
>
> Regards,
> Andreas
>
> [1]
> http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
>
>

Mime
View raw message