hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10090) Jobtracker metrics not updated properly after execution of a mapreduce job
Date Fri, 15 Nov 2013 20:09:21 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13824041#comment-13824041

Luke Lu commented on HADOOP-10090:

I now recalled some hesitation of extra lock on source:

* It could adversely affect the application performance buy holding the source lock while
doing a snapshot. Currently source have a choice on whether and how the snapshot should be
synchronized or not depending on the nature of the metrics involved. In many cases, source
is implemented by a real object the has application locking logic. Holding a lock doing a
potentially large snapshot (many metrics) _could_ increase lock contention significantly.
* Locking far away from the object is consider an anti-pattern that makes it hard to reason
about locking by looking at the source only.

bq. do you think we should backport YARN-1043? Looks like an incompatible change so not sure
whether we want it back to 1.0 line.

Always update all should be a compatible change, semantic wise, besides extra objects for
non-changing metrics.  It seems that simply backporting the one line change (all is true always)
from YARN-1043 (and keeping the test) is less risky to change the locking mechanisms.

> Jobtracker metrics not updated properly after execution of a mapreduce job
> --------------------------------------------------------------------------
>                 Key: HADOOP-10090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 1.2.1
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: HADOOP-10090.branch-1.2.patch, HADOOP-10090.branch-1.patch, OneBoxRepro.png
> After executing a wordcount mapreduce sample job, jobtracker metrics are not updated
properly. Often times the response from the jobtracker has higher number of job_completed
than job_submitted (for example 8 jobs completed and 7 jobs submitted). 
> Issue reported by Toma Paunovic.

This message was sent by Atlassian JIRA

View raw message