spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David McWhorter <mcwhor...@ccri.com>
Subject Re: configuring spark.yarn.driver.memoryOverhead on Spark 1.2.0
Date Mon, 12 Jan 2015 16:37:00 GMT
Hi Ganelin, sorry if it wasn't clear from my previous email, but that is 
how I am creating a spark context.  I just didn't write out the lines 
where I create the new SparkConf and SparkContext.  I am also upping the 
driver memory when running.

Thanks,
David

On 01/12/2015 11:12 AM, Ganelin, Ilya wrote:
> There are two related options:
>
> To solve your problem directly try:
> valconf =newSparkConf().set("spark.yarn.driver.memoryOverhead","1024")
> valsc =newSparkContext(conf)
> And the second, which increases the overall memory available on the driver, as part of
your spark-submit script add:
> --driver-memory 2g
> Hope this helps!
>
>
> From: David McWhorter <mcwhorter@ccri.com <mailto:mcwhorter@ccri.com>>
> Date: Monday, January 12, 2015 at 11:01 AM
> To: "user@spark.apache.org <mailto:user@spark.apache.org>" 
> <user@spark.apache.org <mailto:user@spark.apache.org>>
> Subject: configuring spark.yarn.driver.memoryOverhead on Spark 1.2.0
>
> Hi all,
>
> I'm trying to figure out how to set this option: " 
> spark.yarn.driver.memoryOverhead" on Spark 1.2.0.  I found this 
> helpful overview 
> http://apache-spark-user-list.1001560.n3.nabble.com/Stable-spark-streaming-app-td14105.html#a14476,

> which suggests to launch with --spark.yarn.driver.memoryOverhead 1024 
> added to spark-submit.  However, when I do that I get this error:
> Error: Unrecognized option '--spark.yarn.driver.memoryOverhead'.
> Run with --help for usage help or --verbose for debug output
> I have also tried calling 
> sparkConf.set("spark.yarn.driver.memoryOverhead", "1024") on my spark 
> configuration object but I still get "Will allocate AM container, with 
> XXXX MB memory including 384 MB overhead" when launching.  I'm running 
> in yarn-cluster mode.
>
> Any help or tips would be appreciated.
>
> Thanks,
> David
> -- 
>
> David McWhorter
> Software Engineer
> Commonwealth Computer Research, Inc.
> 1422 Sachem Place, Unit #1
> Charlottesville, VA 22901
> mcwhorter@ccri.com  | 434.299.0090x204
>
> ------------------------------------------------------------------------
>
> The information contained in this e-mail is confidential and/or 
> proprietary to Capital One and/or its affiliates. The information 
> transmitted herewith is intended only for use by the individual or 
> entity to which it is addressed.  If the reader of this message is not 
> the intended recipient, you are hereby notified that any review, 
> retransmission, dissemination, distribution, copying or other use of, 
> or taking of any action in reliance upon this information is strictly 
> prohibited. If you have received this communication in error, please 
> contact the sender and delete the material from your computer.
>

-- 

David McWhorter
Software Engineer
Commonwealth Computer Research, Inc.
1422 Sachem Place, Unit #1
Charlottesville, VA 22901
mcwhorter@ccri.com | 434.299.0090x204


Mime
View raw message