spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Spark - configuration setting doesn't work
Date Thu, 17 Oct 2019 13:59:12 GMT
Hi Users,

I am setting spark configuration in below way;

val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()

spark.conf.set("spark.speculation", "false")
spark.conf.set("spark.broadcast.compress", "true")
spark.conf.set("spark.sql.broadcastTimeout", "36000")
spark.conf.set("spark.network.timeout", "2500s")
spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
spark.conf.set("spark.driver.memory", "10g")
spark.conf.set("spark.executor.memory", "10g")

import spark.implicits._


and submitting spark job with spark - submit. but none of the above
configuration is

getting reflected to the job, I have checked at Spark-UI.

I know setting up like this while creation of spark object, it's working well.


val spark = SparkSession.builder().appName(APP_NAME)
  .config("spark.network.timeout", "1500s")
  .config("spark.broadcast.compress", "true")
  .config("spark.sql.broadcastTimeout", "36000")
  .getOrCreate()

import spark.implicits._


Can someone please throw light?

Mime
View raw message