spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tarun Kumar <ta...@sumologic.com>
Subject Accumulators not available in Task.taskMetrics
Date Thu, 05 Oct 2017 08:14:12 GMT
Hi, I registered an accumulator in driver via
sparkContext.register(myCustomAccumulator, "accumulator-name"). But this
accumulator is not available in task.metrics.accumulators() list.
Accumulator is not visible in spark UI as well.

Does spark need different configuration to make accumulator visible to
tasks?

My task tries to access accumulator via following two ways, it doesn't seem
to find the accumulator:

First way:

AccumulatorContext.lookForAccumulatorByName("accumulator-name").

  map(accum => {
    accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
  })


Second way:

taskContext.taskMetrics().accumulators().
  filter(_.name == Some("accumulator-name")).

  map(accum => {
    accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
  })


Thanks

Tarun

Mime
View raw message