spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anders Arpteg <arp...@spotify.com>
Subject Re: Dynamic Allocation in Spark 1.2.0
Date Mon, 29 Dec 2014 09:58:01 GMT
Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation
about the feature!

Was afraid that the node manager needed reconfiguration (and restart). Any
idea of how much resources will the shuffle service take on the node
manager? In a multi-tenant Hadoop cluster environment, it would be
undesirable to have a Spark-specific long running service taking up
resources from other types of jobs on the cluster.

Thanks again,
Anders

On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu <zsxwing@gmail.com> wrote:

> I encountered the following issue when enabling dynamicAllocation. You may
> want to take a look at it.
>
> https://issues.apache.org/jira/browse/SPARK-4951
>
> Best Regards,
> Shixiong Zhu
>
> 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>:
>
>> Hi Anders,
>>
>> I faced the same issue as you mentioned. Yes, you need to install
>> spark shuffle plugin for YARN. Please check following PRs which add
>> doc to enable dynamicAllocation:
>>
>> https://github.com/apache/spark/pull/3731
>> https://github.com/apache/spark/pull/3757
>>
>> I could run Spark on YARN with dynamicAllocation by following the
>> instructions described in the docs.
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg <arpteg@spotify.com>
>> wrote:
>> > Hey,
>> >
>> > Tried to get the new spark.dynamicAllocation.enabled feature working on
>> Yarn
>> > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following
>> > settings:
>> >
>> >       conf
>> >         .set("spark.dynamicAllocation.enabled", "true")
>> >         .set("spark.shuffle.service.enabled", "true")
>> >         .set("spark.dynamicAllocation.minExecutors", "10")
>> >         .set("spark.dynamicAllocation.maxExecutors", "700")
>> >
>> > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but
>> > with the settings above, it will start the app and the first job is
>> listed
>> > in the web ui. However, no tasks are started and it seems to be stuck
>> > waiting for a container to be allocated forever.
>> >
>> > Any help would be appreciated. Need to do something specific to get the
>> > external yarn shuffle service running in the node manager?
>> >
>> > TIA,
>> > Anders
>>
>>
>>
>> --
>> - Tsuyoshi
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>

Mime
View raw message