From user-return-23225-apmail-spark-user-archive=spark.apache.org@spark.apache.org Mon Dec 29 09:59:00 2014 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B552010B37 for ; Mon, 29 Dec 2014 09:59:00 +0000 (UTC) Received: (qmail 73938 invoked by uid 500); 29 Dec 2014 09:58:59 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 73862 invoked by uid 500); 29 Dec 2014 09:58:58 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 73852 invoked by uid 99); 29 Dec 2014 09:58:58 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Dec 2014 09:58:58 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arpteg@spotify.com designates 209.85.218.49 as permitted sender) Received: from [209.85.218.49] (HELO mail-oi0-f49.google.com) (209.85.218.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Dec 2014 09:58:33 +0000 Received: by mail-oi0-f49.google.com with SMTP id a141so27840669oig.8 for ; Mon, 29 Dec 2014 01:58:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=spotify.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=/OuWBQW1u2lHi1SW+5fz7QLs1knRKpKlNaa0qVFN8Jc=; b=mpPA8LnEPxeOt5aWjXv+uuCNegHddLv04PUE5RnzrtQgpQXrCWJfUVsWpDfFuV7AC1 zf7B0RqCocf4k40MftfZQ80bxyrCkVgMj0pO68a/iobcrZNabFS9HQcGBhchmjiCgMOe f0Fh1TBWwzqTzkQubHU2gx0mfHWZ21A0Qp9PI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=/OuWBQW1u2lHi1SW+5fz7QLs1knRKpKlNaa0qVFN8Jc=; b=m2evG4N0Qml5kkSw9j54WRcB/wW9v8wdR9mx8+T0SDQEx/2CVXiBX7uQQ8XJsoddsM cwRXd8acjoYGloTRKzaXewtNbvRKIdirZomw0ej81FHVBSpUVSx3mqfBc0pmI+eOeJsP eP2+eIB+V5goBE/96S8ouCmtMajlTNck2QMqRBd5gYf1/U93IJhm25dSJjzq9B9U+Ha1 pTdhOyZVUgDTouJZY4igh/VYdGafukAaaYJtNN6tUD2XuAQY9+UNOLuEgXhugdSg9KK6 PzyOtPF/0jK4TjHCX9qRXLbQ795n+1vVDicBhi8zFcrxQFCZ5sAZSVOBadh7QKUTxORr gcJQ== X-Gm-Message-State: ALoCoQlVqP/XsKecMgOEDaWQegqY79HBJj/eKl7sIwCpjKmCazlJXjweK8s/OElf2mFdoKvuiyoh X-Received: by 10.202.228.9 with SMTP id b9mr30928830oih.40.1419847111748; Mon, 29 Dec 2014 01:58:31 -0800 (PST) MIME-Version: 1.0 Received: by 10.76.125.232 with HTTP; Mon, 29 Dec 2014 01:58:01 -0800 (PST) In-Reply-To: References: From: Anders Arpteg Date: Mon, 29 Dec 2014 10:58:01 +0100 Message-ID: Subject: Re: Dynamic Allocation in Spark 1.2.0 To: Shixiong Zhu Cc: Tsuyoshi OZAWA , "user@spark.incubator.apache.org" Content-Type: multipart/alternative; boundary=001a11413860bb4460050b57ea65 X-Virus-Checked: Checked by ClamAV on apache.org --001a11413860bb4460050b57ea65 Content-Type: text/plain; charset=UTF-8 Thanks Tsuyoshi and Shixiong for the info. Awesome with more documentation about the feature! Was afraid that the node manager needed reconfiguration (and restart). Any idea of how much resources will the shuffle service take on the node manager? In a multi-tenant Hadoop cluster environment, it would be undesirable to have a Spark-specific long running service taking up resources from other types of jobs on the cluster. Thanks again, Anders On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu wrote: > I encountered the following issue when enabling dynamicAllocation. You may > want to take a look at it. > > https://issues.apache.org/jira/browse/SPARK-4951 > > Best Regards, > Shixiong Zhu > > 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA : > >> Hi Anders, >> >> I faced the same issue as you mentioned. Yes, you need to install >> spark shuffle plugin for YARN. Please check following PRs which add >> doc to enable dynamicAllocation: >> >> https://github.com/apache/spark/pull/3731 >> https://github.com/apache/spark/pull/3757 >> >> I could run Spark on YARN with dynamicAllocation by following the >> instructions described in the docs. >> >> Thanks, >> - Tsuyoshi >> >> On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg >> wrote: >> > Hey, >> > >> > Tried to get the new spark.dynamicAllocation.enabled feature working on >> Yarn >> > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following >> > settings: >> > >> > conf >> > .set("spark.dynamicAllocation.enabled", "true") >> > .set("spark.shuffle.service.enabled", "true") >> > .set("spark.dynamicAllocation.minExecutors", "10") >> > .set("spark.dynamicAllocation.maxExecutors", "700") >> > >> > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but >> > with the settings above, it will start the app and the first job is >> listed >> > in the web ui. However, no tasks are started and it seems to be stuck >> > waiting for a container to be allocated forever. >> > >> > Any help would be appreciated. Need to do something specific to get the >> > external yarn shuffle service running in the node manager? >> > >> > TIA, >> > Anders >> >> >> >> -- >> - Tsuyoshi >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org >> For additional commands, e-mail: user-help@spark.apache.org >> >> > --001a11413860bb4460050b57ea65 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks=C2=A0Tsuyoshi and=C2=A0Shixiong for the info. Aweso= me with more documentation about the feature!

Was afraid= that the node manager needed reconfiguration (and restart). Any idea of ho= w much resources will the shuffle service take on the node manager? In a mu= lti-tenant Hadoop cluster environment, it would be undesirable to have a Sp= ark-specific long running service taking up resources from other types of j= obs on the cluster.

Thanks again,
Anders= =C2=A0=C2=A0

On Sun, Dec 28, 2014 at 8:08 AM, Shixiong Zhu <= ;zsxwing@gmail.com> wrote:

Best Regards,

Shixiong Zhu=

2014-12-28 2:07 GMT+0= 8:00 Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>:
Hi Anders,<= br>
I faced the same issue as you mentioned. Yes, you need to install
spark shuffle plugin for YARN. Please check following PRs which add
doc to enable dynamicAllocation:

htt= ps://github.com/apache/spark/pull/3731
htt= ps://github.com/apache/spark/pull/3757

I could run Spark on YARN with dynamicAllocation by following the
instructions described in the docs.

Thanks,
- Tsuyoshi

On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg <arpteg@spotify.com> wrote:
> Hey,
>
> Tried to get the new spark.dynamicAllocation.enabled feature working o= n Yarn
> (Hadoop 2.2), but am unsuccessful so far. I've tested with the fol= lowing
> settings:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0conf
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.en= abled", "true")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.shuffle.service.enab= led", "true")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.mi= nExecutors", "10")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.ma= xExecutors", "700")
>
> The app works fine on Spark 1.2 if dynamicAllocation is not enabled, b= ut
> with the settings above, it will start the app and the first job is li= sted
> in the web ui. However, no tasks are started and it seems to be stuck<= br> > waiting for a container to be allocated forever.
>
> Any help would be appreciated. Need to do something specific to get th= e
> external yarn shuffle service running in the node manager?
>
> TIA,
> Anders



--
- Tsuyoshi

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org



--001a11413860bb4460050b57ea65--