spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Melo <>
Subject Re: Driver vs master
Date Mon, 07 Oct 2019 23:45:11 GMT
Hi Amit

On Mon, Oct 7, 2019 at 18:33 Amit Sharma <> wrote:

> Can you please help me understand this. I believe driver programs runs on
> master node

If we are running 4 spark job and driver memory config is 4g then total 16
> 6b would be used of master node.

This depends on what master/deploy mode you're using: if it's "local"
master and "client mode" then yes tasks execute in the same JVM as the
driver. In this case though, the driver JVM uses whatever much space is
allocated for the driver regardless of how many threads you have.

So if we will run more jobs then we need more memory on master. Please
> correct me if I am wrong.

This depends on your application, but in general more threads will require
more memory.

> Thanks
> Amit
It's dark in this basement.

View raw message