spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haopu Wang" <>
Subject concurrent jobs
Date Fri, 18 Jul 2014 10:06:26 GMT
By looking at the code of JobScheduler, I find a parameter of below:


  private val numConcurrentJobs = ssc.conf.getInt("spark.streaming.concurrentJobs", 1)

  private val jobExecutor = Executors.newFixedThreadPool(numConcurrentJobs)


Does that mean each App can have only one active stage?


In my psydo-code below:


         S1 = viewDStream.forEach( collect() )..

         S2 = viewDStream.forEach( collect() )..


There should be two “collect()” jobs for each batch interval, right? Are they running
in parallel?


Thank you!

View raw message