spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Areg Baghdasaryan (BLOOMBERG/ 731 LEX -)" <>
Subject How does the Spark Accumulator work under the covers?
Date Fri, 10 Oct 2014 14:37:34 GMT
I was wondering on what does the Spark accumulator do under the covers. I’ve implemented
my own associative addInPlace function for the accumulator, where is this function being run?
Let’s say you call something like => sum += x) is “sum” being accumulated
locally in any way, for each element or partition or node? Is “sum” a broadcast variable?
Or does it only exist on the driver node? How does the driver node get access to the “sum”?
View raw message