spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rishi Shah <rishishah.s...@gmail.com>
Subject Re: Databricks - number of executors, shuffle.partitions etc
Date Thu, 16 May 2019 10:56:38 GMT
Thanks Ayan, I wasn't aware of such user group specifically for databricks.
Thanks for the input, much appreciated!

On Wed, May 15, 2019 at 10:07 PM ayan guha <guha.ayan@gmail.com> wrote:

> Well its a databricks question so better be asked in their forum.
>
> You can set up cluster level params when you create new cluster or add
> them later. Go to cluster page, ipen one cluster, expand additional config
> section and add your param there as key value pair separated by space.
>
> On Thu, 16 May 2019 at 11:46 am, Rishi Shah <rishishah.star@gmail.com>
> wrote:
>
>> Hi All,
>>
>> Any idea?
>>
>> Thanks,
>> -Rishi
>>
>> On Tue, May 14, 2019 at 11:52 PM Rishi Shah <rishishah.star@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> How can we set spark conf parameter in databricks notebook? My cluster
>>> doesn't take into account any spark.conf.set properties... it creates 8
>>> worker nodes (dat executors) but doesn't honor the supplied conf
>>> parameters. Any idea?
>>>
>>> --
>>> Regards,
>>>
>>> Rishi Shah
>>>
>>
>>
>> --
>> Regards,
>>
>> Rishi Shah
>>
> --
> Best Regards,
> Ayan Guha
>


-- 
Regards,

Rishi Shah

Mime
View raw message