spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Rovner <alex.rov...@magnetic.com>
Subject Re: [Spark on YARN] Multiple Auxiliary Shuffle Service Versions
Date Mon, 05 Oct 2015 23:37:08 GMT
We are running CDH 5.4 with Spark 1.3 as our main version and that version
is configured to use the external shuffling service. We have also installed
Spark 1.5 and have configured it not to use the external shuffling service
and that works well for us so far. I would be interested myself how to
configure multiple versions to use the same shuffling service.

*Alex Rovner*
*Director, Data Engineering *
*o:* 646.759.0052

* <http://www.magnetic.com/>*

On Mon, Oct 5, 2015 at 11:06 AM, Andreas Fritzler <
andreas.fritzler@gmail.com> wrote:

> Hi Steve, Alex,
>
> how do you handle the distribution and configuration of
> the spark-*-yarn-shuffle.jar on your NodeManagers if you want to use 2
> different Spark versions?
>
> Regards,
> Andreas
>
> On Mon, Oct 5, 2015 at 4:54 PM, Steve Loughran <stevel@hortonworks.com>
> wrote:
>
>>
>> > On 5 Oct 2015, at 16:48, Alex Rovner <alex.rovner@magnetic.com> wrote:
>> >
>> > Hey Steve,
>> >
>> > Are you referring to the 1.5 version of the history server?
>> >
>>
>>
>> Yes. I should warn, however, that there's no guarantee that a history
>> server running the 1.4 code will handle the histories of a 1.5+ job. In
>> fact, I'm fairly confident it won't, as the events to get replayed are
>> different.
>>
>
>

Mime
View raw message