spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangyu Li <yisky...@gmail.com>
Subject Spark uses more threads than specified in local[n]
Date Thu, 05 Apr 2018 01:41:35 GMT
Hi,

I am running a job in local mode, configured with local[1] for the sake of
the example. The timeline view in Spark UI is as follows:




It shows that there are actually two threads running, though their
overlapping is very small. To validate this, I also added some change to
Spark's task runner that prints `Thread.currentThread().getId()`, and there
indeed prints out two thread ids.

Is this some behavior from Spark's thread pool to improve performance?

-- 
Sincerely
Xiangyu Li

<yiskylee@gmail.com>

Mime
View raw message