spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: How to set Spark executor memory?
Date Mon, 16 Mar 2015 08:36:18 GMT
By default spark.executor.memory is set to 512m, I'm assuming since you are
submiting the job using spark-submit and it is not able to override the
value since you are running in local mode. Can you try it without using
spark-submit as a standalone project?

Thanks
Best Regards

On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshen84@gmail.com> wrote:

> I set it in code, not by configuration. I submit my jar file to local. I
> am working in my developer environment.
>
> On Mon, 16 Mar 2015 18:28 Akhil Das <akhil@sigmoidanalytics.com> wrote:
>
>> How are you setting it? and how are you submitting the job?
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have set spark.executor.memory to 2048m, and in the UI "Environment"
>>> page, I can see this value has been set correctly. But in the "Executors"
>>> page, I saw there's only 1 executor and its memory is 265.4MB. Very strange
>>> value. why not 256MB, or just as what I set?
>>>
>>> What am I missing here?
>>>
>>>
>>> Thanks,
>>> David
>>>
>>>
>>

Mime
View raw message