On Wed, Jan 21, 2015 at 9:13 PM, Bob Tiernay <btiernay@hotmail.com> wrote:
Maybe I'm misunderstanding something here, but couldn't this be done with broadcast variables? I there is the following caveat from the docs: 

"In addition, the object v should not be modified after it is broadcast in order to ensure that all nodes get the same value of the broadcast variable (e.g. if the variable is shipped to a new node later)"

Well, I think I need a modifiable state ("modifiable" = "changes once per interval") that stores the number of total items seen so far in the lifetime of my application, and I need this number on each executor. Since this number changes after every interval processed, I think broadcast variables are probably not appropriate in this case.