From user-return-60234-apmail-spark-user-archive=spark.apache.org@spark.apache.org Wed Jul 27 07:03:32 2016 Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BEC3B19D83 for ; Wed, 27 Jul 2016 07:03:30 +0000 (UTC) Received: (qmail 7820 invoked by uid 500); 27 Jul 2016 07:03:26 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 7700 invoked by uid 500); 27 Jul 2016 07:03:26 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 7689 invoked by uid 99); 27 Jul 2016 07:03:26 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 27 Jul 2016 07:03:26 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id CDA71CCBAE for ; Wed, 27 Jul 2016 07:03:25 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.252 X-Spam-Level: ** X-Spam-Status: No, score=2.252 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_SOFTFAIL=0.972] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=equalum-io.20150623.gappssmtp.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id K-ClLq_LyRvQ for ; Wed, 27 Jul 2016 07:03:22 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id CC5F45F400 for ; Wed, 27 Jul 2016 07:03:21 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id q128so199655773wma.1 for ; Wed, 27 Jul 2016 00:03:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=equalum-io.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=a1iydcKALnTUxIcr6gtKg3bTLoxNfElCKzMSfgc3zxs=; b=z7eIiZoJohKMX0NC49wzqhRR/jUy/wDxU0NV3U87+DY9/HjvjfkojZ10e9Edt4ag2q lQR5SD4mmeRgHteEiD/iiLMZnvCe889OpwQkHH1SkVDUsZBTbacp3XOPk25feHtpPVHp s9RjqqhuNdjmPr2rhZmfNhhy7KfI4qD7JY1YzTqJM5fYZXr8dYlz+euZNMqfyzsoZsae 3l20MiY/sYASYlbcQclk7dEyTzcCLM2mjiuOp112DisgIk65wzRSLTaCrATBuVCgeoUb yIeirQhtu8SXsmhaFd7kz95biXpRAe/v7p+lXq1GL+IOohD7jPzHhsN4Orx6HEKuhpTt VQvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=a1iydcKALnTUxIcr6gtKg3bTLoxNfElCKzMSfgc3zxs=; b=gaQLN2Li1hvLkv69sbz85ydiqPRbDk+z6EqtlgrdkdOArtTEEagjmEUZYvgEEUt0RQ IeyUz3qe5zKG9dhnNHVJIapbSQD2lNPZKarmogqOa/26FS7lpY84rENPY7iGXjare5nA e5rDSkokcOPOzZPUCMYCE5a549V7ltrcb0WNb+KELXkVgjz5itAWSB5uMQCPHlPcOS3F fkBh2KNPevBfbLVq7EPtlPepwgSlwb3rauJPUYhCbTKy0gCBlNnEtCRUfHheahAy+CnL 6DJtvOJ1PUF0F+cWcmzkQiwOpxE9eGwvuCJdSrWn0nb+rw5oqndb1ScVbz+T8g7Irf4a iaWw== X-Gm-Message-State: AEkoouvzqPYfTze4Of7NrHgS89NouuiVT5BzE8UeAA8s0MrYifu3esH6Vdv3mXRguAA2R1PCDYzNUEEfXPuZ+0kJ X-Received: by 10.28.4.77 with SMTP id 74mr31342060wme.84.1469603000809; Wed, 27 Jul 2016 00:03:20 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.168.201 with HTTP; Wed, 27 Jul 2016 00:03:20 -0700 (PDT) In-Reply-To: References: From: Ofir Manor Date: Wed, 27 Jul 2016 10:03:20 +0300 Message-ID: Subject: Re: The Future Of DStream To: Chang Chen , user Content-Type: multipart/alternative; boundary=001a1141e954d30e120538989cde --001a1141e954d30e120538989cde Content-Type: text/plain; charset=UTF-8 Structured Streaming in 2.0 is declared as alpha - plenty of bits still missing: http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html I assume that it will be declared stable / GA in a future 2.x release, and then it will co-exist with DStream for quite a while before someone will suggest to start a deprecation process that will eventually lead to its removal... As a user, I guess we will need to apply judgement about when to switch to Structured Streaming - each of us have a different risk/value tradeoff, based on our specific situation... Ofir Manor Co-Founder & CTO | Equalum Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen wrote: > Hi guys > > Structure Stream is coming with spark 2.0, but I noticed that DStream is > still here > > What's the future of the DStream, will it be deprecated and removed > eventually? Or co-existed with Structure Stream forever? > > Thanks > Chang > > --001a1141e954d30e120538989cde Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Structured Streaming in 2.0 is declared as alpha - plenty = of bits still missing:
=C2=A0 =C2=A0 =C2=A0http://spar= k.apache.org/docs/latest/structured-streaming-programming-guide.html
I assume that it will be declared stable / GA in a future 2.x release= , and then it will co-exist with DStream for quite a while before someone w= ill suggest to start a deprecation process that will eventually lead to its= removal...
As a user, I guess we will need to apply judgement ab= out when to switch to Structured Streaming - each of us have a different ri= sk/value tradeoff, based on our specific situation...

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286=C2=A0|=C2=A0Email: ofir.manor@equalum.io


On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen = <baibaichen@gmail.com> wrote:
Hi guys

Structure Stream is comi= ng with spark 2.0, =C2=A0but I noticed that DStream is still here

What's the future of the DStream, will it be deprecated= and removed eventually? Or co-existed with =C2=A0Structure Stream forever?=

Thanks
Chang


--001a1141e954d30e120538989cde--