crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Created] (CRUNCH-558) Add name to Spark Accumulators
Date Thu, 03 Sep 2015 12:35:45 GMT
Micah Whitacre created CRUNCH-558:

             Summary: Add name to Spark Accumulators
                 Key: CRUNCH-558
             Project: Crunch
          Issue Type: Improvement
          Components: Spark
            Reporter: Micah Whitacre

It was brought up on the mailing list that our Crunch counters are not showing up on the Spark
webui possibly because they are not named.

We are currently testing a few capabilities using Spark and one thing we noticed in Spark
is they don't list any user defined accumulators on web UI. 

On MapReduce I would imagine counters being displayed on the job page, however on a SparkPipeline
I was only able to pull counter information from PipelineResult#getStageResult(). 

I think the reason these accumulators are not visible on web UI is because crunch does not
name these accumulators. Spark expects an accumulator to have a name to be visible on the
(accumulator API with Name)

I would like to know if it's possible in crunch to name these accumulators so they are available
in web UI. This will give us an experience where users can monitor/watch accumulators from
web UI to obtain key information about their jobs. 

This message was sent by Atlassian JIRA

View raw message