From dev-return-12899-apmail-spark-dev-archive=spark.apache.org@spark.apache.org Fri May 1 15:02:28 2015 Return-Path: X-Original-To: apmail-spark-dev-archive@minotaur.apache.org Delivered-To: apmail-spark-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E80DE10E4C for ; Fri, 1 May 2015 15:02:27 +0000 (UTC) Received: (qmail 31006 invoked by uid 500); 1 May 2015 15:02:26 -0000 Delivered-To: apmail-spark-dev-archive@spark.apache.org Received: (qmail 30915 invoked by uid 500); 1 May 2015 15:02:26 -0000 Mailing-List: contact dev-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list dev@spark.apache.org Received: (qmail 30904 invoked by uid 99); 1 May 2015 15:02:26 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2015 15:02:26 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: message received from 54.191.145.13 which is an MX secondary for dev@spark.apache.org) Received: from [54.191.145.13] (HELO mx1-us-west.apache.org) (54.191.145.13) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 01 May 2015 15:02:21 +0000 Received: from smtp1.ugent.be (smtp1.ugent.be [157.193.71.182]) by mx1-us-west.apache.org (ASF Mail Server at mx1-us-west.apache.org) with ESMTP id 357BD24BA8 for ; Fri, 1 May 2015 15:02:00 +0000 (UTC) Received: from localhost (mcheck2.ugent.be [157.193.49.249]) by smtp1.ugent.be (Postfix) with ESMTP id 22E8080BC; Fri, 1 May 2015 17:01:33 +0200 (CEST) X-Virus-Scanned: by UGent DICT Received: from smtp1.ugent.be ([IPv6:::ffff:157.193.71.182]) by localhost (mcheck2.UGent.be [::ffff:157.193.43.11]) (amavisd-new, port 10024) with ESMTP id GICyG8sAGa5R; Fri, 1 May 2015 17:01:32 +0200 (CEST) Received: from [157.193.8.84] (vpnh084.ugent.be [157.193.8.84]) (Authenticated sender: ehiggs) by smtp1.ugent.be (Postfix) with ESMTPSA id 4D6F88035; Fri, 1 May 2015 17:01:32 +0200 (CEST) Message-ID: <5543954B.1060102@ugent.be> Date: Fri, 01 May 2015 17:01:31 +0200 From: Ewan Higgs User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: Stephen Carman , Sree V CC: "dev@spark.apache.org" Subject: Re: Tungsten + Flink References: <554093BA.1040606@ugent.be> <1180297705.1114451.1430352067672.JavaMail.yahoo@mail.yahoo.com> In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Miltered: at jchkm1 with ID 5543954B.000 by Joe's j-chkmail (http://helpdesk.ugent.be/email/)! X-j-chkmail-Enveloppe: 5543954B.000 from vpnh084.ugent.be/vpnh084.ugent.be/157.193.8.84/[157.193.8.84]/ X-j-chkmail-Score: MSGID : 5543954B.000 on smtp1.ugent.be : j-chkmail score : . : R=. U=. O=. B=0.000 -> S=0.000 X-j-chkmail-Status: Ham X-Virus-Checked: Checked by ClamAV on apache.org I don't think it's useful to combine them since they are different projects. But I do think that a lot of work went into Flink's paged memory system built on byte buffers and if collaboration can take place to pop that out into like a memory subsystem library that both Spark and Flink can use then it should raise both ships. If the usage patterns are too different then sure, don't use their work. But it looks pretty generic: https://github.com/apache/flink/tree/master/flink-core/src/main/java/org/apache/flink/core/memory To bring this back into other threads: Flink's memory system uses java.nio - so it requires Java 1.7 afaik. :) -Ewan On 05/01/2015 03:54 PM, Stephen Carman wrote: > I think as long as the two frameworks follow the same paradigm for how their interfaces work it’s fine to have 2 competing frameworks. This way the frameworks have some motivation > to be the best at what they do rather than being the only choice whether you like it or not. They also seem to have some differing opinions about how to do certain things leaving me to believe > that the 2 projects exist mostly because of disagreements on fundamentals about how a system such as this should be built and scaled out. > > I think spark should definitely take what it can from these projects, but otherwise they should remain separate projects going their own way. > > Steve > > >> On Apr 29, 2015, at 8:01 PM, Sree V wrote: >> >> I agree, Ewan. >> We should also look into combining both Flink and Spark into one.This eases the industry adaptation instead. >> >> Thanking you. >> >> With Regards >> Sree >> >> >> On Wednesday, April 29, 2015 3:21 AM, Ewan Higgs wrote: >> >> >> Hi all, >> A quick question about Tungsten. The announcement of the Tungsten >> project is on the back of Hadoop Summit in Brussels where some of the >> Flink devs were giving talks [1] on how Flink manages memory using byte >> arrays and the like to avoid the overhead of all the Java types[2]. Is >> there an opportunity for code reuse here? Spark and Flink may have >> different needs in some respects, but they work fundamentally towards >> the same goal so I imagine there could be come worthwhile collaboration. >> >> -Ewan >> >> [1] http://2015.hadoopsummit.org/brussels/speaker/?speaker=MrtonBalassi >> http://2015.hadoopsummit.org/brussels/speaker/?speaker=AljoschaKrettek >> >> [2] >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741525 >> https://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org >> For additional commands, e-mail: dev-help@spark.apache.org >> >> >> > This e-mail is intended solely for the above-mentioned recipient and it may contain confidential or privileged information. If you have received it in error, please notify us immediately and delete the e-mail. You must not copy, distribute, disclose or take any action in reliance on it. In addition, the contents of an attachment to this e-mail may contain software viruses which could damage your own computer system. While ColdLight Solutions, LLC has taken every reasonable precaution to minimize this risk, we cannot accept liability for any damage which you sustain as a result of software viruses. You should perform your own virus checks before opening the attachment. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org For additional commands, e-mail: dev-help@spark.apache.org