spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haopu Wang" <HW...@qilinsoft.com>
Subject concurrent jobs
Date Fri, 18 Jul 2014 10:06:26 GMT
By looking at the code of JobScheduler, I find a parameter of below:

 

  private val numConcurrentJobs = ssc.conf.getInt("spark.streaming.concurrentJobs", 1)

  private val jobExecutor = Executors.newFixedThreadPool(numConcurrentJobs)

 

Does that mean each App can have only one active stage?

 

In my psydo-code below:

 

         S1 = viewDStream.forEach( collect() )..

         S2 = viewDStream.forEach( collect() )..

 

There should be two “collect()” jobs for each batch interval, right? Are they running
in parallel?

 

Thank you!

Mime
View raw message