spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <>
Subject Re: Closing over a var with changing value in Streaming application
Date Wed, 21 Jan 2015 12:32:39 GMT

On Wed, Jan 21, 2015 at 9:13 PM, Bob Tiernay <> wrote:

> Maybe I'm misunderstanding something here, but couldn't this be done with
> broadcast variables? I there is the following caveat from the docs:
> "In addition, the object v should not be modified after it is broadcast
> in order to ensure that all nodes get the same value of the broadcast
> variable (e.g. if the variable is shipped to a new node later)"

Well, I think I need a modifiable state ("modifiable" = "changes once per
interval") that stores the number of total items seen so far in the
lifetime of my application, and I need this number on each executor. Since
this number changes after every interval processed, I think broadcast
variables are probably not appropriate in this case.


View raw message