spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tarun Kumar <ta...@sumologic.com>
Subject Re: Accumulators not available in Task.taskMetrics
Date Fri, 06 Oct 2017 04:53:51 GMT
Any response on this one? Thanks in advance!

On Thu, Oct 5, 2017 at 1:44 PM, Tarun Kumar <tarun@sumologic.com> wrote:

> Hi, I registered an accumulator in driver via sparkContext.register(myCustomAccumulator,
> "accumulator-name"). But this accumulator is not available in task.metrics.accumulators()
> list. Accumulator is not visible in spark UI as well.
>
> Does spark need different configuration to make accumulator visible to
> tasks?
>
> My task tries to access accumulator via following two ways, it doesn't
> seem to find the accumulator:
>
> First way:
>
> AccumulatorContext.lookForAccumulatorByName("accumulator-name").
>
>   map(accum => {
>     accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
>   })
>
>
> Second way:
>
> taskContext.taskMetrics().accumulators().
>   filter(_.name == Some("accumulator-name")).
>
>   map(accum => {
>     accum.asInstanceOf[MyCustomAccumulator].add(*k, v*))
>   })
>
>
> Thanks
>
> Tarun
>
>

Mime
View raw message