I set "spark.executor.memory" to "2048m". If the executor storage memory is 0.6 of executor memory, it should be 2g * 0.6 = 1.2g.

My machine has 56GB memory, and 0.6 of that should be 33.6G...I hate math xD


On Mon, Mar 16, 2015 at 7:59 PM Akhil Das <akhil@sigmoidanalytics.com> wrote:
How much memory are you having on your machine? I think default value is 0.6 of the spark.executor.memory as you can see from here.

Thanks
Best Regards

On Mon, Mar 16, 2015 at 2:26 PM, Xi Shen <davidshen84@gmail.com> wrote:
Hi Akhil,

Yes, you are right. If I ran the program from IDE as a normal java program, the executor's memory is increased...but not to 2048m, it is set to 6.7GB...Looks like there's some formula to calculate this value.


Thanks,
David


On Mon, Mar 16, 2015 at 7:36 PM Akhil Das <akhil@sigmoidanalytics.com> wrote:
By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the job using spark-submit and it is not able to override the value since you are running in local mode. Can you try it without using spark-submit as a standalone project?

Thanks
Best Regards

On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <davidshen84@gmail.com> wrote:

I set it in code, not by configuration. I submit my jar file to local. I am working in my developer environment.


On Mon, 16 Mar 2015 18:28 Akhil Das <akhil@sigmoidanalytics.com> wrote:
How are you setting it? and how are you submitting the job?

Thanks
Best Regards

On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <davidshen84@gmail.com> wrote:
Hi,

I have set spark.executor.memory to 2048m, and in the UI "Environment" page, I can see this value has been set correctly. But in the "Executors" page, I saw there's only 1 executor and its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set?

What am I missing here?


Thanks,
David