spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <matei.zaha...@gmail.com>
Subject Re: The Future Of DStream
Date Wed, 27 Jul 2016 07:32:02 GMT
Yup, they will definitely coexist. Structured Streaming is currently alpha and will probably
be complete in the next few releases, but Spark Streaming will continue to exist, because
it gives the user more low-level control. It's similar to DataFrames vs RDDs (RDDs are the
lower-level API for when you want control, while DataFrames do more optimizations automatically
by restricting the computation model).

Matei

> On Jul 27, 2016, at 12:03 AM, Ofir Manor <ofir.manor@equalum.io> wrote:
> 
> Structured Streaming in 2.0 is declared as alpha - plenty of bits still missing:
>      http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
<http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html>
> I assume that it will be declared stable / GA in a future 2.x release, and then it will
co-exist with DStream for quite a while before someone will suggest to start a deprecation
process that will eventually lead to its removal...
> As a user, I guess we will need to apply judgement about when to switch to Structured
Streaming - each of us have a different risk/value tradeoff, based on our specific situation...
> 
> Ofir Manor
> 
> Co-Founder & CTO | Equalum
> 
> 
> Mobile: +972-54-7801286 <tel:%2B972-54-7801286> | Email: ofir.manor@equalum.io
<mailto:ofir.manor@equalum.io>
> On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen <baibaichen@gmail.com <mailto:baibaichen@gmail.com>>
wrote:
> Hi guys
> 
> Structure Stream is coming with spark 2.0,  but I noticed that DStream is still here
> 
> What's the future of the DStream, will it be deprecated and removed eventually? Or co-existed
with  Structure Stream forever?
> 
> Thanks
> Chang
> 
> 


Mime
View raw message