spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashic Mahtab <as...@live.com>
Subject Can multiple streaming apps use the same checkpoint directory?
Date Tue, 20 Jan 2015 11:58:07 GMT
Hi,
For client mode spark submits of applications, we can do the following:

def createStreamingContext() = {
...
 val sc = new SparkContext(conf)
 // Create a StreamingContext with a 1 second batch size
 val ssc = new StreamingContext(sc, Seconds(1))
}
...
val ssc = StreamingContext.getOrCreate(checkPointdir, createStreamingContext _)

If the driver goes down, and we restart it, it'll pick up where it left off. My question is
if multiple streaming apps are submitted through the same machine, can the share the same
check point directory, or does each have to have its own?

Thanks,
Ashic.
 		 	   		  
Mime
View raw message