hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-4705) Historyserver links expire before the history data does
Date Thu, 04 Oct 2012 21:33:47 GMT
Jason Lowe created MAPREDUCE-4705:

             Summary: Historyserver links expire before the history data does
                 Key: MAPREDUCE-4705
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4705
             Project: Hadoop Map/Reduce
          Issue Type: Bug
          Components: jobhistoryserver, mrv2
    Affects Versions: 0.23.3
            Reporter: Jason Lowe
            Priority: Critical

The historyserver can serve up links to jobs that become useless well before the job history
files are purged.  For example on a large, heavily used cluster we can end up rotating through
the maximum number of jobs the historyserver can track fairly quickly.  If a user was investigating
an issue with a job using a saved historyserver URL, that URL can become useless because the
historyserver has forgotten about the job even though the history files are still sitting
in HDFS.

We can tell the historyserver to keep track of more jobs by increasing {{mapreduce.jobhistory.joblist.cache.size}},
but this has a direct impact on the responsiveness of the main historyserver page since it
serves up all the entries to the client at once.  It looks like Hadoop 1.x avoided this issue
by encoding the history file location into the URLs served up by the historyserver, so it
didn't have to track a mapping between job ID and history file location.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message