spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From liuwei <stupi...@126.com>
Subject spark.scheduler.pool seems not working in spark streaming
Date Wed, 30 Jul 2014 08:43:42 GMT
In my spark streaming program, I set scheduler pool, just as follows:

val myFairSchedulerFile = “xxx.xml”
val myStreamingPool = “xxx”

System.setProperty(“spark.scheduler.allocation.file”, myFairSchedulerFile)
val conf = new SparkConf()
val ssc = new StreamingContext(conf, batchInterval)
ssc.sparkContext.setLocalProperty(“spark.scheduler.pool”, myStreamingPool)
….
ssc.start()
ssc.awaitTermination()

I submit my spark streaming job in my spark cluster, and I found stage’s pool name is “default”,
it seem ssc.sparkContext.setLocalProperty(“spark.scheduler.pool”, myStreamingPool) not
work.

Mime
View raw message