From dev-return-1203-apmail-tez-dev-archive=tez.apache.org@tez.apache.org Fri Nov 7 19:14:20 2014 Return-Path: X-Original-To: apmail-tez-dev-archive@minotaur.apache.org Delivered-To: apmail-tez-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9587817ED1 for ; Fri, 7 Nov 2014 19:14:20 +0000 (UTC) Received: (qmail 41580 invoked by uid 500); 7 Nov 2014 19:14:20 -0000 Delivered-To: apmail-tez-dev-archive@tez.apache.org Received: (qmail 41530 invoked by uid 500); 7 Nov 2014 19:14:20 -0000 Mailing-List: contact dev-help@tez.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@tez.apache.org Delivered-To: mailing list dev@tez.apache.org Received: (qmail 41518 invoked by uid 99); 7 Nov 2014 19:14:20 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Nov 2014 19:14:20 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of acm@hortonworks.com designates 74.125.82.51 as permitted sender) Received: from [74.125.82.51] (HELO mail-wg0-f51.google.com) (74.125.82.51) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Nov 2014 19:14:16 +0000 Received: by mail-wg0-f51.google.com with SMTP id l18so4369522wgh.24 for ; Fri, 07 Nov 2014 11:13:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=fHNFP3fUv4AMQINMi2IqK8jfStYyjuIIof+Ssl6aD8w=; b=IKfjtLZ8zCrRsSTNfxe/QnVkzMg2ZdlYwkZvK9o2H8bV6Ttw0/7VQUvxq9vdU2oyCY 3svXfK9JiEQ5bwbr6f9rtqQZnhBjpFUAFnESB31EF+MyfAYSRuExlSBHj0ggR7LYLdyE Tz0SGYIzGrlwZf6j9dDkeNvc9of3YBgIZp8kWXuLMl8fzpfb5ORPtyQJZRcZunlFbn8L d3PvpvWGZAARY1L70i0r/F7o5O2oVZ4ZW5yjPfYHGgAOAt1VretCQ9/o4SRaeq9pq/ag fFJbZ8rqpE7lvvwsj7s//bTLIuV78RDNeKcrVE0TbsvX1Tnpf+Yls0haZyxmyf867Slr MmNg== X-Gm-Message-State: ALoCoQna+3NqZubppav1t/0BTTlEz6iSMF1MTPAEoYJn6S1HS5iFtE/dOso7FU4/gxy6dhUH30CeLJ1NRtLuCdtaejAgflMkyGLWt2dDQbj+G1WpDdoJ8/Y= MIME-Version: 1.0 X-Received: by 10.180.74.76 with SMTP id r12mr7667160wiv.33.1415387634694; Fri, 07 Nov 2014 11:13:54 -0800 (PST) Received: by 10.194.30.134 with HTTP; Fri, 7 Nov 2014 11:13:54 -0800 (PST) In-Reply-To: References: Date: Fri, 7 Nov 2014 11:13:54 -0800 Message-ID: Subject: Re: Flink on Tez From: Arun Murthy To: dev@tez.apache.org Cc: "dev@flink.incubator.apache.org" Content-Type: multipart/alternative; boundary=f46d043894912fa5b70507499d70 X-Virus-Checked: Checked by ClamAV on apache.org --f46d043894912fa5b70507499d70 Content-Type: text/plain; charset=UTF-8 This is great news! Awesome work everyeone... super excited to see this! Arun On Fri, Nov 7, 2014 at 10:03 AM, Kostas Tzoumas wrote: > Hello Flink and Tez, > > I would like to point you to a first version of Flink running on > Tez. This is a Flink subproject (to be initially contributed > to flink-addons) that allows you to run unmodified Flink programs on > top of Apache Tez. > > You can get the code here: > https://github.com/ktzoumas/incubator-flink/tree/tez_support > > If you want to give it a spin, some basic instructions are here: > > https://github.com/ktzoumas/incubator-flink/tree/tez_support/flink-addons/flink-tez > > > Be warned that this is still work in progress, so you may encounter > bugs, and this has not yet been optimized for performance. > > A few words on how it works and the motivation: > > The programs pass as usual through the Flink compiler and use the > Flink runtime operators (map, reduce, join, etc, including the Flink > facilities for sorting, hashing, etc). Instead of generating a Flink > distributed program (called "JobGraph" in Flink), we can now also > generate a Tez program (called "DAG" in Tez). > > I have been asked why would we want to do that, as Flink has its own > execution engine. Two reasons in my opinion. > > First, Tez follows design choices that are geared towards resource > elasticity, whereas the design choices behind Flink's engine are > geared more towards low latency querying and iterative > processing. Therefoere, the two engines can really complement each > other. Users can run their Flink programs in the engine that fits > better their use case and setup. > > Second, in Flink we have put a lot of effort in separating program > assembly with program execution and architecting the system in layers > (APIs, common API, compiler, data processing runtime, distributed > execution engine). The possibility to swap execution engines is a good > showcase of the benefits of such a layered architecture. > > Of course, trying it out and reporting bugs or contributing is very > welcome! > > Best, > Kostas > -- -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. --f46d043894912fa5b70507499d70--