spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adline Dsilva <>
Subject Re: Structured Streaming - Can I start using it?
Date Tue, 14 Mar 2017 10:26:53 GMT

On 14 Mar 2017 4:19 p.m., Gaurav Pandya <> wrote:
Thanks a lot Michal & Ofir for your insights.

To Ofir - I have not yet finalized my spark streaming code. it is still work in progress.
Now we have Structured streaming available, so thought to re write it to gain maximum benefit
in future. As of now, there are no specific functional or performance issues Nor I have to
leverage any new API. This is just considering future aspects.


On Tue, Mar 14, 2017 at 1:05 PM, Ofir Manor <<>>
To add to what Michael said, my experience was that Structured Streaming in 2.0 was half-baked
/ alpha, but in 2.1 it is significantly more robust. Also a lot of its "missing functionality"
were not available in Spark Streaming either way.
HOWEVER, you mentioned that you think about rewriting your existing spark streaming code...
May I ask why do you need a rewrite? Do you have a specific functional or performance issues?
Some specific new use case or a specific new API you want to leverage?
Changing an existing, working solution has its costs, both in dev time and ops time (changes
to monitoring, troubleshooting etc), so I think you should know what you want to achieve here
and ask / prototype if current release fits it.

Ofir Manor

Co-Founder & CTO | Equalum

Mobile: +972-54-7801286<tel:%2B972-54-7801286> | Email:<>

On Mon, Mar 13, 2017 at 9:45 PM, Michael Armbrust <<>>
I think its very very unlikely that it will get withdrawn.  The primary reason that the APIs
are still marked experimental is that we like to have several releases before committing to
interface stability (in particular the interfaces to write custom sources and sinks are likely
to evolve).  Also, there are currently quite a few limitations in the types of queries that
we can run (i.e. multiple aggregations are disallowed, we don't support stream-stream joins
yet).  In these cases though, we explicitly say its not supported when you try to start your

For the use cases that are supported in 2.1 though (streaming ETL, event time aggregation,
etc) I'll say that we have been using it in production for several months and we have customers
doing the same.

On Mon, Mar 13, 2017 at 11:21 AM, Gaurav1809 <<>>
I read in spark documentation that Structured Streaming is still ALPHA in
Spark 2.1 and the APIs are still experimental. Shall I use it to re write my
existing spark streaming code? Looks like it is not yet production ready.
What happens if Structured Streaming project gets withdrawn?

View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe e-mail:<>


This e-mail (including any attachments) is for the addressee(s) only and may be confidential,
especially as regards personal data. If you are not the intended recipient, please note that
any dealing, review, distribution, printing, copying or use of this e-mail is strictly prohibited.
If you have received this email in error, please notify the sender immediately and delete
the original message (including any attachments).

MIMOS Berhad is a research and development institution under the purview of the Malaysian
Ministry of Science, Technology and Innovation. Opinions, conclusions and other information
in this e-mail that do not relate to the official business of MIMOS Berhad and/or its subsidiaries
shall be understood as neither given nor endorsed by MIMOS Berhad and/or its subsidiaries
and neither MIMOS Berhad nor its subsidiaries accepts responsibility for the same. All liability
arising from or in connection with computer viruses and/or corrupted e-mails is excluded to
the fullest extent permitted by law.
View raw message