spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Determine number of running executors
Date Mon, 19 Jan 2015 09:34:03 GMT
Hi,

On Sat, Jan 17, 2015 at 3:05 AM, Shuai Zheng <szheng.code@gmail.com> wrote:
>
> Can you share more information about how do you do that? I also have
> similar question about this.
>

Not very proud about it ;-), but here you go:

// find the number of workers available to us.
val _runCmd = scala.util.Properties.propOrElse("sun.java.command", "")
val numCoresRe = ".*--executor-cores ([0-9]+) --num-executors ([0-9]+).*".r
val totalNumCores = _runCmd match {
  case numCoresRe(coresPerExecutor, numExecutors) =>
    coresPerExecutor.toInt * numExecutors.toInt
  case _ =>
    0
}
if (totalNumCores > 0)
  logger.debug("total number of cores: " + totalNumCores)
else
  logger.warn("could not extract number of cores from run command: " + _runCmd)

Tobias

Mime
View raw message