spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From N B <nb.nos...@gmail.com>
Subject Re: Broadcast variables can be rebroadcast?
Date Sat, 16 May 2015 07:01:10 GMT
Thanks Ayan. Can we rebroadcast after updating in the driver?

Thanks
NB.


On Fri, May 15, 2015 at 6:40 PM, ayan guha <guha.ayan@gmail.com> wrote:

> Hi
>
> broadcast variables are shipped for the first time it is accessed in a
> transformation to the executors used by the transformation. It will NOT
> updated subsequently, even if the value has changed. However, a new value
> will be shipped to any new executor comes into play after the value has
> changed. This way, changing value of broadcast variable is not a good idea
> as it can create inconsistency within cluster. From documentatins:
>
>  In addition, the object v should not be modified after it is broadcast
> in order to ensure that all nodes get the same value of the broadcast
> variable
>
>
> On Sat, May 16, 2015 at 10:39 AM, N B <nb.nospam@gmail.com> wrote:
>
>> Thanks Ilya. Does one have to call broadcast again once the underlying
>> data is updated in order to get the changes visible on all nodes?
>>
>> Thanks
>> NB
>>
>>
>> On Fri, May 15, 2015 at 5:29 PM, Ilya Ganelin <ilganeli@gmail.com> wrote:
>>
>>> The broadcast variable is like a pointer. If the underlying data changes
>>> then the changes will be visible throughout the cluster.
>>> On Fri, May 15, 2015 at 5:18 PM NB <nb.nospam@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> Once a broadcast variable is created using sparkContext.broadcast(),
>>>> can it
>>>> ever be updated again? The use case is for something like the underlying
>>>> lookup data changing over time.
>>>>
>>>> Thanks
>>>> NB
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Broadcast-variables-can-be-rebroadcast-tp22908.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>
>
>
> --
> Best Regards,
> Ayan Guha
>

Mime
View raw message