spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Akhil Das <>
Subject Re: Accumulator map
Date Sun, 07 Jun 2015 15:57:17 GMT
​Another approach would be to use a zookeeper. If you have zookeeper
running somewhere in the cluster you can simply create a path like
*/dynamic-list*​ in it and then write objects/values to it, you can even
create/access nested objects.

Best Regards

On Fri, Jun 5, 2015 at 7:06 PM, Cosmin Cătălin Sanda <> wrote:

> Hi,
> I am trying to gather some statistics from an RDD using accumulators.
> Essentially, I am counting how many times specific segments appear in each
> row of the RDD. This works fine and I get the expected results, but the
> problem is that each time I add a new segment to look for, I have to
> explicitly create an Accumulator for it and explicitly use the Accumulator
> in the foreach method.
> Is there a way to use a dynamic list of Accumulators in Spark? I want to
> control the segments from a single place in the code and the accumulators
> to be dynamically created and used based on the metrics list.
> BR,
> ------------------------------------
> *Cosmin Catalin SANDA*
> Software Systems Engineer
> Phone: +

View raw message