From user-return-23169-apmail-spark-user-archive=spark.apache.org@spark.apache.org Sun Dec 28 07:09:11 2014 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 245A0CFE9 for ; Sun, 28 Dec 2014 07:09:10 +0000 (UTC) Received: (qmail 69649 invoked by uid 500); 28 Dec 2014 07:09:05 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 69576 invoked by uid 500); 28 Dec 2014 07:09:04 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 69566 invoked by uid 99); 28 Dec 2014 07:09:04 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Dec 2014 07:09:04 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of zsxwing@gmail.com designates 209.85.213.175 as permitted sender) Received: from [209.85.213.175] (HELO mail-ig0-f175.google.com) (209.85.213.175) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 28 Dec 2014 07:08:39 +0000 Received: by mail-ig0-f175.google.com with SMTP id h15so10163446igd.8 for ; Sat, 27 Dec 2014 23:08:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=2PHbqz7A0gH/VCFda4h/incaEN6HR6U/KaOZfIci52E=; b=JAeX4S9WAXm8ZAWHwinXdampqTiEWCxzQTxYQzovhT3FT2F2S3XtHLm91k+g5fK3nJ P4u2A9PQSJNqWLn0cEzpXp9vFDDO1cJb4zWLBnmLNeBOX9wB64lP3JGqnPMmVUlGJu57 4+b80dSk5DfupIqUn1gv2y2ERPeGZf2x+IGySKecr+LzvMatQLB48q7N91u60m/wUPff z8r1Vtvj7S0H4Dy3mgRr5fPZAvJuKfKAYYUVzO7EjCvopmhgCAtN/A+hYvcSVH0sFrxw ISJuuzBXDpXVSMtoyKKMrTxlcSOUJ+muLWzjUo2XMBPWPeLw3QiveUkhU1JvhA22Ch/o /soQ== MIME-Version: 1.0 X-Received: by 10.107.154.198 with SMTP id c189mr45743721ioe.68.1419750518070; Sat, 27 Dec 2014 23:08:38 -0800 (PST) Received: by 10.107.165.203 with HTTP; Sat, 27 Dec 2014 23:08:38 -0800 (PST) In-Reply-To: References: Date: Sun, 28 Dec 2014 15:08:38 +0800 Message-ID: Subject: Re: Dynamic Allocation in Spark 1.2.0 From: Shixiong Zhu To: Anders Arpteg , Tsuyoshi OZAWA Cc: "user@spark.incubator.apache.org" Content-Type: multipart/alternative; boundary=001a1140e8944ca670050b416de5 X-Virus-Checked: Checked by ClamAV on apache.org --001a1140e8944ca670050b416de5 Content-Type: text/plain; charset=UTF-8 I encountered the following issue when enabling dynamicAllocation. You may want to take a look at it. https://issues.apache.org/jira/browse/SPARK-4951 Best Regards, Shixiong Zhu 2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA : > Hi Anders, > > I faced the same issue as you mentioned. Yes, you need to install > spark shuffle plugin for YARN. Please check following PRs which add > doc to enable dynamicAllocation: > > https://github.com/apache/spark/pull/3731 > https://github.com/apache/spark/pull/3757 > > I could run Spark on YARN with dynamicAllocation by following the > instructions described in the docs. > > Thanks, > - Tsuyoshi > > On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg > wrote: > > Hey, > > > > Tried to get the new spark.dynamicAllocation.enabled feature working on > Yarn > > (Hadoop 2.2), but am unsuccessful so far. I've tested with the following > > settings: > > > > conf > > .set("spark.dynamicAllocation.enabled", "true") > > .set("spark.shuffle.service.enabled", "true") > > .set("spark.dynamicAllocation.minExecutors", "10") > > .set("spark.dynamicAllocation.maxExecutors", "700") > > > > The app works fine on Spark 1.2 if dynamicAllocation is not enabled, but > > with the settings above, it will start the app and the first job is > listed > > in the web ui. However, no tasks are started and it seems to be stuck > > waiting for a container to be allocated forever. > > > > Any help would be appreciated. Need to do something specific to get the > > external yarn shuffle service running in the node manager? > > > > TIA, > > Anders > > > > -- > - Tsuyoshi > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org > For additional commands, e-mail: user-help@spark.apache.org > > --001a1140e8944ca670050b416de5 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I encountered the following issue when enabling=C2=A0dynam= icAllocation. You may want to take a look at it.


Best R= egards,

Shixiong Zhu

2014-12-28 2:07 GMT+08:00 Tsuyoshi OZAWA <ozawa.tsuyoshi@gmail.com>:
Hi Anders,

I faced the same issue as you mentioned. Yes, you need to install
spark shuffle plugin for YARN. Please check following PRs which add
doc to enable dynamicAllocation:

htt= ps://github.com/apache/spark/pull/3731
htt= ps://github.com/apache/spark/pull/3757

I could run Spark on YARN with dynamicAllocation by following the
instructions described in the docs.

Thanks,
- Tsuyoshi

On Sat, Dec 27, 2014 at 11:06 PM, Anders Arpteg <arpteg@spotify.com> wrote:
> Hey,
>
> Tried to get the new spark.dynamicAllocation.enabled feature working o= n Yarn
> (Hadoop 2.2), but am unsuccessful so far. I've tested with the fol= lowing
> settings:
>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0conf
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.en= abled", "true")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.shuffle.service.enab= led", "true")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.mi= nExecutors", "10")
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0.set("spark.dynamicAllocation.ma= xExecutors", "700")
>
> The app works fine on Spark 1.2 if dynamicAllocation is not enabled, b= ut
> with the settings above, it will start the app and the first job is li= sted
> in the web ui. However, no tasks are started and it seems to be stuck<= br> > waiting for a container to be allocated forever.
>
> Any help would be appreciated. Need to do something specific to get th= e
> external yarn shuffle service running in the node manager?
>
> TIA,
> Anders



--
- Tsuyoshi

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


--001a1140e8944ca670050b416de5--