spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject RE: How to set Spark executor memory?
Date Mon, 16 Mar 2015 13:33:11 GMT
Hi Xi Shen,

You could set the spark.executor.memory in the code itself . new SparkConf()..set("spark.executor.memory",
Or you can try the -- spark.executor.memory 2g while submitting the jar.

Jishnu Prathap

From: Akhil Das []
Sent: Monday, March 16, 2015 2:06 PM
To: Xi Shen
Subject: Re: How to set Spark executor memory?

By default spark.executor.memory is set to 512m, I'm assuming since you are submiting the
job using spark-submit and it is not able to override the value since you are running in local
mode. Can you try it without using spark-submit as a standalone project?

Best Regards

On Mon, Mar 16, 2015 at 1:52 PM, Xi Shen <<>>

I set it in code, not by configuration. I submit my jar file to local. I am working in my
developer environment.

On Mon, 16 Mar 2015 18:28 Akhil Das <<>>
How are you setting it? and how are you submitting the job?

Best Regards

On Mon, Mar 16, 2015 at 12:52 PM, Xi Shen <<>>

I have set spark.executor.memory to 2048m, and in the UI "Environment" page, I can see this
value has been set correctly. But in the "Executors" page, I saw there's only 1 executor and
its memory is 265.4MB. Very strange value. why not 256MB, or just as what I set?

What am I missing here?


The information contained in this electronic message and any attachments to this message are
intended for the exclusive use of the addressee(s) and may contain proprietary, confidential
or privileged information. If you are not the intended recipient, you should not disseminate,
distribute or copy this e-mail. Please notify the sender immediately and destroy all copies
of this message and any attachments. WARNING: Computer viruses can be transmitted via email.
The recipient should check this email and any attachments for the presence of viruses. The
company accepts no liability for any damage caused by any virus transmitted by this email.
View raw message