hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-3788) [Gridmix] Investigate if Gridmix can be made YARN aware
Date Thu, 02 Feb 2012 06:37:53 GMT
[Gridmix] Investigate if Gridmix can be made YARN aware

                 Key: MAPREDUCE-3788
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3788
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
          Components: contrib/gridmix
    Affects Versions: 0.24.0
            Reporter: Amar Kamat
            Assignee: Ravi Gummadi
             Fix For: 0.23.1, 0.24.0

Gridmix was written keeping in mind the monolithic JobTracker. Calls to the single JobTracker
were throttled to avoid excess load. Also, polling was faster in JobTracker as the job statuses
were cached even if the job was complete. In the YARN world, the situation is slightly different.
To make Gridmix scalable and really a YARN scale-benchmarking tool, Gridmix should be enhanced.
Some directions worth investigating are:
1. Investigate if Gridmix can cache the AM handles and poll the AM directly for map/reduce
task progress. 
2. Can the job monitor be made multi-threaded? Each thread can poll a bunch of AMs.
3. Check if there are better ways for getting job progress updates and get away with the busy-waiting
logic in Gridmix.
4. Can Gridmix be made container aware. The definition of cluster load should be container

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message