From user-return-59703-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Jul 20 03:20:53 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A745919C9F for ; Wed, 20 Jul 2016 03:20:53 +0000 (UTC) Received: (qmail 60024 invoked by uid 500); 20 Jul 2016 03:20:49 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 59916 invoked by uid 500); 20 Jul 2016 03:20:48 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 59906 invoked by uid 99); 20 Jul 2016 03:20:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Jul 2016 03:20:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 1CBD2C04FD for ; Wed, 20 Jul 2016 03:20:48 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.299 X-Spam-Level: * X-Spam-Status: No, score=1.299 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=aehrlich-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 63R1BgajvL7w for ; Wed, 20 Jul 2016 03:20:46 +0000 (UTC) Received: from mail-pf0-f174.google.com (mail-pf0-f174.google.com [209.85.192.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B63B75F489 for ; Wed, 20 Jul 2016 03:20:45 +0000 (UTC) Received: by mail-pf0-f174.google.com with SMTP id y134so13989484pfg.0 for ; Tue, 19 Jul 2016 20:20:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=aehrlich-com.20150623.gappssmtp.com; s=20150623; h=mime-version:subject:from:in-reply-to:date:cc:message-id:references :to; bh=ANJMJ0P5tU2bFEDx1fvg4B99vB+tFMOwqQPf512igS0=; b=iF4N9PJ+0av2pcM8/TMAb99nLfgqQw91tUBD4WcXvAm2TfxW0GWemRrSB5fdk4zwKV lHtDELj/M9rBZbn0X4tmOpTe7G6pqoqlpCr4t3+JA1+ma7fY76Vn2JmFDci1OttepjGh 4fXL3H0qjAbmngvbdMoDFfhhbteUUvyGp8sUnJMqn4WsYK5JyPnAMS8NxgL3oNArzFiC eZFh6DGkEa255AlJaFZBjbFCxX3/FfvEGuTZ02YkV3g6ID3JiW7I+1kv/wMBSp6Ss9Vs U2ol4zTaFWXFn4xwzyg1Uo5MUdSepoPBhw5Z8uemnNWLVkznTVVp6C8Vf5iRtl5Pz380 lBOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ANJMJ0P5tU2bFEDx1fvg4B99vB+tFMOwqQPf512igS0=; b=SKQZE1dMDXSKFjOYvJ+sPoIYEkyMjUciYX+wLde+nK1o4TLOIDUFuXKxShD6IMZGze H9rsV7GRdWHCTt1o4RcADc/xT2HxhMcXTUiZbArVtsOlHq/BMMMBil1LskpjdMd4grkV AvpvwfaApa3K3ZJiXTvWW48TRum8By/DpK7cghrRsOnlVrqiG9EsaqH/z12NqjUsSTaM +I+yvYq7+3LzgjRmh6OcT3CJn0xvqsz4M5jGUfD9vVa1fgmCbjCkPeo2Ciysn32YqNMZ l6kSSRj/EuHm2jXSoFtQJB6AejBqSXzrwgXEBJjMOtPoJw3iZAA8MUw2SsHaQdswtDM5 Phqg== X-Gm-Message-State: ALyK8tJfKC80Z06jfN77qDC2Ph8YqYpC581CkBHbQ2U45jjofrikj1ACAlEct+Ma7KGVYA== X-Received: by 10.98.83.1 with SMTP id h1mr60762099pfb.78.1468984844573; Tue, 19 Jul 2016 20:20:44 -0700 (PDT) Received: from [10.1.1.13] ([198.8.80.158]) by smtp.gmail.com with ESMTPSA id b68sm464120pfg.85.2016.07.19.20.20.43 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 19 Jul 2016 20:20:44 -0700 (PDT) Content-Type: multipart/alternative; boundary="Apple-Mail=_0987EF89-C7B9-4E5D-9A97-892E9AA472B6" Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Heavy Stage Concentration - Ends With Failure From: Andrew Ehrlich In-Reply-To: Date: Tue, 19 Jul 2016 20:20:42 -0700 Cc: Aaron Jackson , user Message-Id: <7C919479-BC89-47B5-931C-DC161D2265F0@aehrlich.com> References: To: Kuchekar X-Mailer: Apple Mail (2.3124) --Apple-Mail=_0987EF89-C7B9-4E5D-9A97-892E9AA472B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Yea this is a good suggestion; also check 25th percentile, median, and = 75th percentile to see how skewed the input data is. If you find that the RDD=E2=80=99s partitions are skewed you can solve = it either by changing the partitioner when you read the files like = already suggested, or call repartition() on the RDD before the = bottleneck to redistribute the data amongst the partitions by executing = a shuffle. > On Jul 19, 2016, at 6:19 PM, Kuchekar = wrote: >=20 > Hi, >=20 > Can you check if the RDD is partitioned correctly with correct = partition number (if you are manually setting the partition value.) . = Try using Hash partitioner while reading the files. >=20 > One way you can debug is by checking the number of records that = executor has compared to others in the Stage tab of the Spark UI. >=20 > Kuchekar, Nilesh >=20 > On Tue, Jul 19, 2016 at 8:16 PM, Aaron Jackson > wrote: > Hi, >=20 > I have a cluster with 15 nodes of which 5 are HDFS nodes. I kick off = a job that creates some 120 stages. Eventually, the active and pending = stages reduce down to a small bottleneck and it never fails... the tasks = associated with the 10 (or so) running tasks are always allocated to the = same executor on the same host. >=20 > Sooner or later, it runs out of memory ... or some other resource. It = falls over and then they tasks are reallocated to another executor. >=20 > Why do we see such heavy concentration of tasks onto a single executor = when other executors are free? Were the tasks assigned to an executor = when the job was decomposed into stages? >=20 --Apple-Mail=_0987EF89-C7B9-4E5D-9A97-892E9AA472B6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Yea this is a good suggestion; also check 25th percentile, = median, and 75th percentile to see how skewed the input data is.

If you find that the = RDD=E2=80=99s partitions are skewed you can solve it either by changing = the partitioner when you read the files like already suggested, or call = repartition(<int>) on the RDD before the bottleneck to = redistribute the data amongst the partitions by executing a = shuffle.

On = Jul 19, 2016, at 6:19 PM, Kuchekar <kuchekar.nilesh@gmail.com> wrote:

Hi,

Can you check if the RDD is partitioned correctly with = correct partition number (if you are manually setting the partition = value.) . Try using Hash partitioner while reading the files.

One way you can debug is = by checking the number of records that executor has compared to others = in the Stage tab of the Spark UI.

Kuchekar, = Nilesh

On Tue, Jul 19, 2016 at 8:16 = PM, Aaron Jackson <ajackson@pobox.com> wrote:
Hi,

I = have a cluster with 15 nodes of which 5 are HDFS nodes.  I kick off = a job that creates some 120 stages.  Eventually, the active and = pending stages reduce down to a small bottleneck and it never fails... = the tasks associated with the 10 (or so) running tasks are always = allocated to the same executor on the same host.

Sooner or later, it runs out of memory = ... or some other resource.  It falls over and then they tasks are = reallocated to another executor.

Why do we see such heavy concentration = of tasks onto a single executor when other executors are free?  = Were the tasks assigned to an executor when the job was decomposed into = stages?


= --Apple-Mail=_0987EF89-C7B9-4E5D-9A97-892E9AA472B6--