spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernhard Schäfer <bschae...@inovex.de>
Subject 24/7 Spark Streaming on YARN in Production
Date Mon, 02 Jan 2017 07:29:33 GMT
Two weeks ago I have published a blogpost about our experiences running 24/7
Spark Streaming applications on YARN in production: 
https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/
<https://www.inovex.de/blog/247-spark-streaming-on-yarn-in-production/> 
Amongst others it contains a reference spark-submit command covering
subtleties such as YARN, backpressure and spark.locality.wait configuration.
Maybe this helps some people not getting stuck on the same issues we faced.
Although the post is not targeted at Structured Streaming, the majority of
configurations still hold true.
Feedback is appreciated.



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/24-7-Spark-Streaming-on-YARN-in-Production-tp28265.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message