hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Mitic (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10090) Jobtracker metrics not updated properly after execution of a mapreduce job
Date Tue, 12 Nov 2013 23:39:18 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820662#comment-13820662

Ivan Mitic commented on HADOOP-10090:

Thanks a lot Chris and Luke for commenting!

bq. How about #3: we only initialize and update the JMX cache when JMX is first used and stops
updating after a period inactivity and reinitialize and update JMX cache upon activity. Initialize/reinitialize
is a "dense" update, while "update" means the "sparse" update with the current lastRecs mechanisms.
I think #3 is should be a fairly straightforward patch and more flexible than #1 and #2.
Luke, I'm not sure where #3 differs from #2. #2 basically says that each time JMX is queried,
if TTL has exceeded, we update the internal cache and return the values. If TTL is not exceeded,
we return the cached values. There are no updates in scenarios where users do not query JMX
(metrics system timer thread does not trigger a cache update). We just have to make sure that
when JMX is indeed queried, the next time the timer thread runs, we poll for all metric values
(not just the ones that have changed -- I think this is the current behavior as well).

bq. Thinking about this some more, I'm in favor of Ivan's proposal #2. This would have the
side effect of lower cache hit rate/higher frequency of pulling metrics from their sources.
From what I can tell, this is acceptable, because JMX queries haven't been a source of bottlenecks
or high load. (Others, please comment if you know otherwise.)
Thanks Chris. I am also leaning in this direction, assuming that there aren't any gotchas
that I did not know about :)

> Jobtracker metrics not updated properly after execution of a mapreduce job
> --------------------------------------------------------------------------
>                 Key: HADOOP-10090
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10090
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: metrics
>    Affects Versions: 1.2.1
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: OneBoxRepro.png
> After executing a wordcount mapreduce sample job, jobtracker metrics are not updated
properly. Often times the response from the jobtracker has higher number of job_completed
than job_submitted (for example 8 jobs completed and 7 jobs submitted). 
> Issue reported by Toma Paunovic.

This message was sent by Atlassian JIRA

View raw message