spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tomasz Gawęda <tomasz.gaw...@outlook.com>
Subject Real time streaming in Spark
Date Mon, 29 Aug 2016 20:13:07 GMT
Hi everyone,


I wonder if there are plans to implement real time streaming in Spark. I see that in Spark
2.0 Trigger can have more implementations than ProcessingTime.


In my opinion Real Time streaming (so reaction on every event - like continous queries in
Apache Ignite) will be very useful and will fill gap that is currently in Spark. Now, if we
must implement both real-time streaming and batch jobs, the streaming must be done in other
frameworks as Spark allows us only to process event in Micro Batches. Matei Zaharia wrote
in Databricks blog about  Continuous Applications [1], in my opinion adding EventTrigger will
be next big step to Continuous Applications.


What do you think about it? Are there any plans to implement such event-based trigger? Of
course I can help with implementation, however I'm just starting learning Spark internals
and it will take a while before I would be able to write something.


Pozdrawiam / Best regards,

Tomek


[1] https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html

[https://databricks.com/wp-content/uploads/2016/07/spark-2-continuous-apps-OG.png]<https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html>

Continuous Applications: Evolving Streaming in Apache Spark 2.0<https://databricks.com/blog/2016/07/28/continuous-applications-evolving-streaming-in-apache-spark-2-0.html>
databricks.com
Apache Spark 2.0 lays the foundation for Continuous Applications, a simplified and unified
way to write end-to-end streaming applications that reacts to data in real-time.



Mime
View raw message