From user-return-73521-apmail-spark-user-archive=spark.apache.org@spark.apache.org Tue Feb 6 20:00:26 2018 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 07C4517528 for ; Tue, 6 Feb 2018 20:00:26 +0000 (UTC) Received: (qmail 67205 invoked by uid 500); 6 Feb 2018 20:00:19 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 67090 invoked by uid 500); 6 Feb 2018 20:00:19 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 67080 invoked by uid 99); 6 Feb 2018 20:00:19 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Feb 2018 20:00:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 1F1D41A03E3 for ; Tue, 6 Feb 2018 20:00:19 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.215 X-Spam-Level: *** X-Spam-Status: No, score=3.215 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_IMAGE_ONLY_16=1.048, HTML_MESSAGE=2, HTML_SHORT_LINK_IMG_2=0.259, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, T_REMOTE_IMAGE=0.01] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=xactlycorp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id NeqY4A_z-b5s for ; Tue, 6 Feb 2018 20:00:18 +0000 (UTC) Received: from mail-ua0-f176.google.com (mail-ua0-f176.google.com [209.85.217.176]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id CF0C05F181 for ; Tue, 6 Feb 2018 20:00:17 +0000 (UTC) Received: by mail-ua0-f176.google.com with SMTP id f5so1997290ual.8 for ; Tue, 06 Feb 2018 12:00:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=xactlycorp.com; s=corpmail; h=mime-version:from:date:message-id:subject:to; bh=MhJwh/HrVb7GfLJFnfVw8Uz1qRovk1q3yybc1oorEys=; b=FChTmbo8wMnFCz7Ghg1CQJ/2hBuREkMmpD9Cwn2RHJ61dcYF6YnFR4xO456Vf08076 ks1z/PDs8J38zD16LJ45+RYGp2iUgQ8lXXLLw/zXBeR2Whfh9KgjBUu8CTkc1aSy73RP ywAeG57F8mA0fvSyZkcBrLTFOyndrI8xhrWeTk50iIcW0jERNdt376CXSnajfWVv7vdw 7CrC/tau1KX0/TisGBrsUtvXMBrrM5Qsj4B86fqiheXZ4e3ZFmA0LJdh2lyWfo+s5Vd/ hYXvLq9MiubmOUqzXgiXlhY6DpfPvTG112BpKyIGV6ZZv6tfVX+hzc2lDSdNcrKPsPCu OeSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=MhJwh/HrVb7GfLJFnfVw8Uz1qRovk1q3yybc1oorEys=; b=ubsQYGcL3QyqLsTgt/KKcHB8x88rb93boWDwfLi1o9s5KJCTKc222aNyCkWhRZLpAl kkdmERDrgg25RWgBpZXbUuL8H756bz3mhWL2nEX+vGHWcWvAn0vMmD4YmG33xxCOh7L6 Q3JdK7GNZoK5eeuNGhE15MHx6BcGqvxhNisl1XPA8EKevZsM9g1xARrQ8/XFNTcXDDbh D/GX1IbQdgL6ZQYNKcB77g9WL4CmNQbUMPWYCCaeMtg4cj6cBcK2fYLi4ZkI/enxrTIB PDtsvMtMrGoS9jPAY1bQxFAr6NdZMNWdZKfHjeWzW3yVznlSEzpdfvnmPJSoSUC9lCRR Qm1w== X-Gm-Message-State: APf1xPDoQUTikjlrx33htPC7WWU0xnuds4sFqsJ7p/ncbXDiustuLS0Q GCLq3u7V4Qyz70SZwMDXZ2R58fZ6CpYoncX/sgDjBrqM1Dq5gEGOjl40qYOCIqoE5VHI47mOuzl RLeTRcAzomtUlwxSYjYlNfdA= X-Google-Smtp-Source: AH8x225Sh7qmrIukut47RbebseuhLWh897eHflq2myw3y+VY6YKzbZtQG4DiEKkdFSbikUPUVoeZXD+XAfAYq6tAyDM= X-Received: by 10.159.43.4 with SMTP id p4mr3305010uaj.142.1517947216977; Tue, 06 Feb 2018 12:00:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.159.42.138 with HTTP; Tue, 6 Feb 2018 12:00:16 -0800 (PST) From: Nirav Patel Date: Tue, 6 Feb 2018 12:00:16 -0800 Message-ID: Subject: Sharing spark executor pool across multiple long running spark applications To: spark users Content-Type: multipart/alternative; boundary="089e08240ff8a7fdea056490a0f6" --089e08240ff8a7fdea056490a0f6 Content-Type: text/plain; charset="UTF-8" Currently sparkContext and it's executor pool is not shareable. Each spakContext gets its own executor pool for entire life of an application. So what is the best ways to share cluster resources across multiple long running spark applications? Only one I see is spark dynamic allocation but it has high latency when it comes to real-time application. -- [image: What's New with Xactly] --089e08240ff8a7fdea056490a0f6 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Currently sparkContext and it's executor pool is not s= hareable. Each spakContext gets its own executor pool for entire life of an= application. So what is the best ways to share cluster resources across mu= ltiple long running spark applications?=C2=A0

Only one I= see is spark dynamic allocation but it has high latency when it comes to r= eal-time application.=C2=A0




3D"What's

= = =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 --089e08240ff8a7fdea056490a0f6--