spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dogtail Ray <>
Subject Question about Spark process and thread
Date Sun, 28 Jun 2015 15:32:40 GMT

I was looking at Spark source code, and I found that when launching a
Executor, actually Spark is launching a threadpool; each time the scheduler
launches a task, the executor will launch a thread within the threadpool.

However, I also found that the Spark process always has approximately 40
threads running regardless of my configuration (SPARK_WORKER_CORES,
SPARK_WORKER_INSTANCES, --executor-cores, --total-executor-cores, etc.).
Does it mean Spark will pre-launch 40 threads even before the tasks are
launched? Great thanks!


View raw message