spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Khatri <chetan.opensou...@gmail.com>
Subject Re: Spark - configuration setting doesn't work
Date Tue, 29 Oct 2019 07:33:47 GMT
Ok, thanks. I wanted to confirm that.

On Sun, Oct 27, 2019 at 12:55 PM hemant singh <hemant2184@gmail.com> wrote:

> You should add the configurations while creating the session, I don’t
> think you can override it once the session is created. Few are though.
>
> Thanks,
> Hemant
>
> On Sun, 27 Oct 2019 at 11:02 AM, Chetan Khatri <
> chetan.opensource@gmail.com> wrote:
>
>> Could someone please help me.
>>
>> On Thu, Oct 17, 2019 at 7:29 PM Chetan Khatri <
>> chetan.opensource@gmail.com> wrote:
>>
>>> Hi Users,
>>>
>>> I am setting spark configuration in below way;
>>>
>>> val spark = SparkSession.builder().appName(APP_NAME).getOrCreate()
>>>
>>> spark.conf.set("spark.speculation", "false")
>>> spark.conf.set("spark.broadcast.compress", "true")
>>> spark.conf.set("spark.sql.broadcastTimeout", "36000")
>>> spark.conf.set("spark.network.timeout", "2500s")
>>> spark.conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
>>> spark.conf.set("spark.driver.memory", "10g")
>>> spark.conf.set("spark.executor.memory", "10g")
>>>
>>> import spark.implicits._
>>>
>>>
>>> and submitting spark job with spark - submit. but none of the above configuration
is
>>>
>>> getting reflected to the job, I have checked at Spark-UI.
>>>
>>> I know setting up like this while creation of spark object, it's working well.
>>>
>>>
>>> val spark = SparkSession.builder().appName(APP_NAME)
>>>   .config("spark.network.timeout", "1500s")
>>>   .config("spark.broadcast.compress", "true")
>>>   .config("spark.sql.broadcastTimeout", "36000")
>>>   .getOrCreate()
>>>
>>> import spark.implicits._
>>>
>>>
>>> Can someone please throw light?
>>>
>>>

Mime
View raw message