spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <ak...@sigmoidanalytics.com>
Subject Re: How to set Spark executor memory?
Date Mon, 16 Mar 2015 08:59:03 GMT
How much memory are you having on your machine? I think default value is
0.6 of the spark.executor.memory as you can see from here
<http://spark.apache.org/docs/1.2.1/configuration.html#execution-behavior>.

Thanks
Best Regards

On Mon, Mar 16, 2015 at 2:26 PM, Xi Shen <davidshen84@gmail.com> wrote:

> Hi Akhil,
>
> Yes, you are right. If I ran the program from IDE as a normal java
> program, the executor's memory is increased...but not to 2048m, it is set
> to 6.7GB...Looks like there's some formula to calculate this value.
>
>
> Thanks,
> David
>
>
> On Mon, Mar 16, 2015 at 7:36 PM Akhil Das <akhil@sigmoidanalytics.com>
> wrote:
>
>> By default spark.executor.memory is set to 512m, I'm assuming since you
>> are submiting the job using spark-submit and it is not able to override the
>> value since you are running in local mode. Can you try it without using
>> spark-submit as a standalone project?
>>
>> Thanks
>> Best Regards
>>
>> On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshen84@gmail.com> wrote:
>>
>>> I set it in code, not by configuration. I submit my jar file to local. I
>>> am working in my developer environment.
>>>
>>> On Mon, 16 Mar 2015 18:28 Akhil Das <akhil@sigmoidanalytics.com> wrote:
>>>
>>>> How are you setting it? and how are you submitting the job?
>>>>
>>>> Thanks
>>>> Best Regards
>>>>
>>>> On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshen84@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have set spark.executor.memory to 2048m, and in the UI "Environment"
>>>>> page, I can see this value has been set correctly. But in the "Executors"
>>>>> page, I saw there's only 1 executor and its memory is 265.4MB. Very strange
>>>>> value. why not 256MB, or just as what I set?
>>>>>
>>>>> What am I missing here?
>>>>>
>>>>>
>>>>> Thanks,
>>>>> David
>>>>>
>>>>>
>>>>
>>

Mime
View raw message