spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amit Sharma <resolve...@gmail.com>
Subject Re: Driver vs master
Date Tue, 08 Oct 2019 00:20:52 GMT
Thanks Andrew but I am asking specific to driver memory not about executors
memory. We have just one master and if each jobs driver.memory=4g and
master nodes total memory is 16gb then we can not execute more than 4 jobs
at a time.

On Monday, October 7, 2019, Andrew Melo <andrew.melo@gmail.com> wrote:

> Hi Amit
>
> On Mon, Oct 7, 2019 at 18:33 Amit Sharma <resolve123@gmail.com> wrote:
>
>> Can you please help me understand this. I believe driver programs runs on
>> master node
>
> If we are running 4 spark job and driver memory config is 4g then total 16
>> 6b would be used of master node.
>
>
> This depends on what master/deploy mode you're using: if it's "local"
> master and "client mode" then yes tasks execute in the same JVM as the
> driver. In this case though, the driver JVM uses whatever much space is
> allocated for the driver regardless of how many threads you have.
>
>
> So if we will run more jobs then we need more memory on master. Please
>> correct me if I am wrong.
>>
>
> This depends on your application, but in general more threads will require
> more memory.
>
>
>
>>
>> Thanks
>> Amit
>>
> --
> It's dark in this basement.
>

Mime
View raw message