spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: Closing over a var with changing value in Streaming application
Date Wed, 21 Jan 2015 12:32:39 GMT
Hi,

On Wed, Jan 21, 2015 at 9:13 PM, Bob Tiernay <btiernay@hotmail.com> wrote:

> Maybe I'm misunderstanding something here, but couldn't this be done with
> broadcast variables? I there is the following caveat from the docs:
>
> "In addition, the object v should not be modified after it is broadcast
> in order to ensure that all nodes get the same value of the broadcast
> variable (e.g. if the variable is shipped to a new node later)"
>

Well, I think I need a modifiable state ("modifiable" = "changes once per
interval") that stores the number of total items seen so far in the
lifetime of my application, and I need this number on each executor. Since
this number changes after every interval processed, I think broadcast
variables are probably not appropriate in this case.

Thanks
Tobias

Mime
View raw message