From dev-return-15757-apmail-spark-dev-archive=spark.apache.org@spark.apache.org Tue Nov 3 21:13:30 2015 Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id B346C18C8D for ; Tue, 3 Nov 2015 21:13:30 +0000 (UTC) Received: (qmail 46915 invoked by uid 500); 3 Nov 2015 21:13:28 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 46811 invoked by uid 500); 3 Nov 2015 21:13:28 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 46794 invoked by uid 99); 3 Nov 2015 21:13:28 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2015 21:13:28 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2EB92C070B for ; Tue, 3 Nov 2015 21:13:28 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.9 X-Spam-Level: ** X-Spam-Status: No, score=2.9 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id Gwl6gQ91nFjj for ; Tue, 3 Nov 2015 21:13:22 +0000 (UTC) Received: from mail-qg0-f42.google.com (mail-qg0-f42.google.com [209.85.192.42]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 68F8842ABE for ; Tue, 3 Nov 2015 21:13:22 +0000 (UTC) Received: by qgem9 with SMTP id m9so24815122qge.1 for ; Tue, 03 Nov 2015 13:13:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=EjMI+W/M9C4beiEZqgPMiNIpU0qaZho1udEYU93nnLk=; b=u36tVZUr9lm79aCZwN+k87eixQ+oi8wEAaZv5vxpd0bZj0APagSc6cJba1Wib0ZvSk WkQk0DlRg0fe5zFL/tczqDsz+4VcsTpMGE9tPekaRPBkInpJY4uaGFZ2VqIB8B//jShM WERpxFYoIMgDnk5fXrFJI0b22y7d7gwM1LZsl0NKoSknVJ2bwLxpoUj4d4EfPIVsjT9b JCaQwhHh0xxAz2wNX1EKKnr8qFTI+Qb2giJslM9EzQN9/sqdT9vrvxpYN8LPfJjJm/1f 7p3h1/Gam4Zb7MuXUWc7LRmhTPJRtyUFIfqz4Z29/JHE78JpIXx7CrM/G+RrhhKgFePE tveQ== X-Received: by 10.140.102.162 with SMTP id w31mr38904684qge.93.1446585202136; Tue, 03 Nov 2015 13:13:22 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Justin Uang Date: Tue, 03 Nov 2015 21:13:12 +0000 Message-ID: Subject: Re: Off-heap storage and dynamic allocation To: Reynold Xin Cc: Ryan Williams , "dev@spark.apache.org" Content-Type: multipart/alternative; boundary=001a11c169b81c7b780523a95d74 --001a11c169b81c7b780523a95d74 Content-Type: text/plain; charset=UTF-8 Thanks for your response. I was worried about #3, vs being able to use the objects directly. #2 seems to be the dealbreaker for my use case right? Even if it I am using tachyon for caching, if an executor is lost, then that partition is lost for the purposes of spark? On Tue, Nov 3, 2015 at 5:53 PM Reynold Xin wrote: > I don't think there is any special handling w.r.t. Tachyon vs in-heap > caching. As a matter of fact, I think the current offheap caching > implementation is pretty bad, because: > > 1. There is no namespace sharing in offheap mode > 2. Similar to 1, you cannot recover the offheap memory once Spark driver > or executor crashes > 3. It requires expensive serialization to go offheap > > It would've been simpler to just treat Tachyon as a normal file system, > and use it that way to at least satisfy 1 and 2, and also substantially > simplify the internals. > > > > > On Tue, Nov 3, 2015 at 7:59 AM, Justin Uang wrote: > >> Yup, but I'm wondering what happens when an executor does get removed, >> but when we're using tachyon. Will the cached data still be available, >> since we're using off-heap storage, so the data isn't stored in the >> executor? >> >> On Tue, Nov 3, 2015 at 4:57 PM Ryan Williams < >> ryan.blake.williams@gmail.com> wrote: >> >>> fwiw, I think that having cached RDD partitions prevents executors from >>> being removed under dynamic allocation by default; see SPARK-8958 >>> . The >>> "spark.dynamicAllocation.cachedExecutorIdleTimeout" config >>> >>> controls this. >>> >>> On Fri, Oct 30, 2015 at 12:14 PM Justin Uang >>> wrote: >>> >>>> Hey guys, >>>> >>>> According to the docs for 1.5.1, when an executor is removed for >>>> dynamic allocation, the cached data is gone. If I use off-heap storage like >>>> tachyon, conceptually there isn't this issue anymore, but is the cached >>>> data still available in practice? This would be great because then we would >>>> be able to set spark.dynamicAllocation.cachedExecutorIdleTimeout to be >>>> quite small. >>>> >>>> ================== >>>> In addition to writing shuffle files, executors also cache data either >>>> on disk or in memory. When an executor is removed, however, all cached data >>>> will no longer be accessible. There is currently not yet a solution for >>>> this in Spark 1.2. In future releases, the cached data may be preserved >>>> through an off-heap storage similar in spirit to how shuffle files are >>>> preserved through the external shuffle service. >>>> ================== >>>> >>> > --001a11c169b81c7b780523a95d74 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Thanks for your response. I was worried about #3, vs being= able to use the objects directly. #2 seems to be the dealbreaker for my us= e case right? Even if it I am using tachyon for caching, if an executor is = lost, then that partition is lost for the purposes of spark?

On Tue, Nov 3, 2015 at 5:53 PM Reyno= ld Xin <rxin@databricks.com&g= t; wrote:
I don= 9;t think there is any special handling w.r.t. Tachyon vs in-heap caching. = As a matter of fact, I think the current offheap caching implementation is = pretty bad, because:

1. There is no namespace sharing in= offheap mode
2. Similar to 1, you cannot recover the offheap mem= ory once Spark driver or executor crashes
3. It requires expensiv= e serialization to go offheap

It would've been= simpler to just treat Tachyon as a normal file system, and use it that way= to at least satisfy 1 and 2, and also substantially simplify the internals= .




On Tue, Nov 3, 2015 at 7:59 AM, Ju= stin Uang <justin.uang@gmail.com> wrote:
Yup, but I'm wondering what happens= when an executor does get removed, but when we're using tachyon. Will = the cached data still be available, since we're using off-heap storage,= so the data isn't stored in the executor?

On Tue, Nov 3, 2015 at 4:57 PM Ryan Will= iams <ryan.blake.williams@gmail.com> wrote:
fwiw, I think that having cached RDD partitions= prevents executors from being removed under dynamic allocation by default;= see SPARK-8958. The "spark.dynam= icAllocation.cachedExecutorIdleTimeout" config controls this.=C2= =A0

On Fri, Oct 30, 20= 15 at 12:14 PM Justin Uang <justin.uang@gmail.com> wrote:
Hey guys,

A= ccording to the docs for 1.5.1, when an executor is removed for dynamic all= ocation, the cached data is gone. If I use off-heap storage like tachyon, c= onceptually there isn't this issue anymore, but is the cached data stil= l available in practice? This would be great because then we would be able = to set spark.dynamicAllocation.cachedExecutorIdleTimeout to be quite small.=

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D
In addition to writing shuffle files, executors also cache data= either on disk or in memory. When an executor is removed, however, all cac= hed data will no longer be accessible. There is currently not yet a solutio= n for this in Spark 1.2. In future releases, the cached data may be preserv= ed through an off-heap storage similar in spirit to how shuffle files are p= reserved through the external shuffle service.
=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

--001a11c169b81c7b780523a95d74--