spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cosmin Cătălin Sanda <cosmincata...@gmail.com>
Subject Accumulator map
Date Fri, 05 Jun 2015 13:36:06 GMT
Hi,

I am trying to gather some statistics from an RDD using accumulators.
Essentially, I am counting how many times specific segments appear in each
row of the RDD. This works fine and I get the expected results, but the
problem is that each time I add a new segment to look for, I have to
explicitly create an Accumulator for it and explicitly use the Accumulator
in the foreach method.

Is there a way to use a dynamic list of Accumulators in Spark? I want to
control the segments from a single place in the code and the accumulators
to be dynamically created and used based on the metrics list.

BR,
------------------------------------
*Cosmin Catalin SANDA*
Software Systems Engineer
Phone: +45.27.30.60.35

Mime
View raw message