Yup, they will definitely coexist. Structured Streaming is currently alpha and will probably be complete in the next few releases, but Spark Streaming will continue to exist, because it gives the user more low-level control. It's similar to DataFrames vs RDDs (RDDs are the lower-level API for when you want control, while DataFrames do more optimizations automatically by restricting the computation model).

Matei

On Jul 27, 2016, at 12:03 AM, Ofir Manor <ofir.manor@equalum.io> wrote:

Structured Streaming in 2.0 is declared as alpha - plenty of bits still missing:
     http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html
I assume that it will be declared stable / GA in a future 2.x release, and then it will co-exist with DStream for quite a while before someone will suggest to start a deprecation process that will eventually lead to its removal...
As a user, I guess we will need to apply judgement about when to switch to Structured Streaming - each of us have a different risk/value tradeoff, based on our specific situation...

Ofir Manor

Co-Founder & CTO | Equalum


Mobile: +972-54-7801286 | Email: ofir.manor@equalum.io


On Wed, Jul 27, 2016 at 8:02 AM, Chang Chen <baibaichen@gmail.com> wrote:
Hi guys

Structure Stream is coming with spark 2.0,  but I noticed that DStream is still here

What's the future of the DStream, will it be deprecated and removed eventually? Or co-existed with  Structure Stream forever?

Thanks
Chang