spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Re: Spark - configuration setting doesn't work
Date Sun, 27 Oct 2019 05:27:08 GMT
Could someone please help me.

On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <chetan.opensource@gmail.com>
wrote:

> Hi Users,
>
> I am setting spark configuration in below way;
>
> val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()
>
> spark.conf.set("spark.speculation", "false")
> spark.conf.set("spark.broadcast.compress", "true")
> spark.conf.set("spark.sql.broadcastTimeout", "36000")
> spark.conf.set("spark.network.timeout", "2500s")
> spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
> spark.conf.set("spark.driver.memory", "10g")
> spark.conf.set("spark.executor.memory", "10g")
>
> import spark.implicits._
>
>
> and submitting spark job with spark - submit. but none of the above configuration is
>
> getting reflected to the job, I have checked at Spark-UI.
>
> I know setting up like this while creation of spark object, it's working well.
>
>
> val spark = SparkSession.builder().appName(APP_NAME)
>   .config("spark.network.timeout", "1500s")
>   .config("spark.broadcast.compress", "true")
>   .config("spark.sql.broadcastTimeout", "36000")
>   .getOrCreate()
>
> import spark.implicits._
>
>
> Can someone please throw light?
>
>

Mime
View raw message