spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jianshi Huang <jianshi.hu...@gmail.com>
Subject Spark streaming update/restart gracefully
Date Mon, 27 Oct 2014 05:56:56 GMT
Looks like currently solution to update spark-stream jars/configurations is
to

1) save current Kafka offsets somewhere (say zookeeper)
2) shutdown the cluster and restart
3) connect to Kafka with previously saved offset

Assuming we're reading from Kafka which provides nice persistence and
read-from-offset feature.

Is there a better solution? Is it possible to reload jars dynamically?
Resize cluster size dynamically (SPARK-3174)?

A reasonable ask is to have a strategy to update or restart executors
gracefully. Something like https://issues.apache.org/jira/browse/STORM-167

Any suggestion?

-- 
Jianshi Huang

LinkedIn: jianshi
Twitter: @jshuang
Github & Blog: http://huangjs.github.com/

Mime
View raw message