crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <>
Subject [jira] [Resolved] (CRUNCH-579) Support Counters from Custom RecordWriters
Date Tue, 19 Apr 2016 20:37:25 GMT


Micah Whitacre resolved CRUNCH-579.
       Resolution: Fixed
         Assignee: Micah Whitacre  (was: Josh Wills)
    Fix Version/s: 0.14.0

Change has been merged to master.  Thanks for the testing help.

> Support Counters from Custom RecordWriters
> ------------------------------------------
>                 Key: CRUNCH-579
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Micah Whitacre
>            Assignee: Micah Whitacre
>             Fix For: 0.14.0
>         Attachments: CRUNCH-579.patch
> A consumer mentioned this on the mailing list:
> {quote}
> So we wrote a custom Crunch Target to write data to a particular location which included
providing RecordWriters, OutputFormat, OutCommitters and such. Now we  wanted to add Counters
to get a count of how much data our reducers are writing. A most obvious design was to use
the TaskAttemptContext that gets passed to the RecordWriter and use the getCounter() method
on it to manipulate the counters. However, that did not work as expected and even though the
counters were getting incremented, they did not show up on the Resource Manager UI. On further
investigation I found that, cruch mangles this context object to add in a named output via
the CrunchOutputs class[1] which basically makes the counters useless within the recordwriter
> Would it be a feasible enhancement for the CrunchOutputs to pass in the original base
context object along with the modified one with named outputs so that it can be used for counters?
Any other suggestions are most welcome as well.
> {quote}

This message was sent by Atlassian JIRA

View raw message