spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xiangyu Li <>
Subject Spark uses more threads than specified in local[n]
Date Thu, 05 Apr 2018 01:41:35 GMT

I am running a job in local mode, configured with local[1] for the sake of
the example. The timeline view in Spark UI is as follows:

It shows that there are actually two threads running, though their
overlapping is very small. To validate this, I also added some change to
Spark's task runner that prints `Thread.currentThread().getId()`, and there
indeed prints out two thread ids.

Is this some behavior from Spark's thread pool to improve performance?

Xiangyu Li


View raw message