spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Landa <metalo...@gmail.com>
Subject Re: Spark Standalone - Failing to pass extra java options to the driver in cluster mode
Date Tue, 20 Aug 2019 05:28:53 GMT
Thanks Jungtaek Lim,
I upgraded the cluster to 2.4.3 and it worked fine.

Thanks,
Alex

On Mon, Aug 19, 2019 at 10:01 PM Jungtaek Lim <kabhwan@gmail.com> wrote:

> Hi Alex,
>
> you seem to hit SPARK-26606 [1] which has been fixed in 2.4.1. Could you
> try it out with latest version?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 1. https://issues.apache.org/jira/browse/SPARK-26606
>
> On Tue, Aug 20, 2019 at 3:43 AM Alex Landa <metaloink@gmail.com> wrote:
>
>> Hi,
>>
>> We are using Spark Standalone 2.4.0 in production and publishing our
>> Scala app using cluster mode.
>> I saw that extra java options passed to the driver don't actually pass.
>> A submit example:
>> *spark-submit --deploy-mode cluster --master spark://<master ip>:7077
>> --driver-memory 512mb --conf
>> "spark.driver.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError" --class
>> App  app.jar *
>>
>> Doesn't pass *-XX:+HeapDumpOnOutOfMemoryError *as a JVM argument, but
>> pass instead
>> *-Dspark.driver.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError*I
>> created a test app for it:
>>
>> val spark = SparkSession.builder()
>>   .master("local")
>>   .appName("testApp").getOrCreate()
>> import spark.implicits._
>>
>> // get a RuntimeMXBean reference
>> val runtimeMxBean = ManagementFactory.getRuntimeMXBean
>>
>> // get the jvm's input arguments as a list of strings
>> val listOfArguments = runtimeMxBean.getInputArguments
>>
>> // print the arguments
>> listOfArguments.asScala.foreach(a => println(s"ARG: $a"))
>>
>>
>> I see that for client mode I get :
>> ARG: -XX:+HeapDumpOnOutOfMemoryError
>> while in cluster mode I get:
>> ARG: -Dspark.driver.extraJavaOptions=-XX:+HeapDumpOnOutOfMemoryError
>>
>> Would appreciate your help how to work around this issue.
>> Thanks,
>> Alex
>>
>>
>
> --
> Name : Jungtaek Lim
> Blog : http://medium.com/@heartsavior
> Twitter : http://twitter.com/heartsavior
> LinkedIn : http://www.linkedin.com/in/heartsavior
>

Mime
View raw message