hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (MAPREDUCE-452) tasktracker checkpointing capability
Date Thu, 17 Jul 2014 17:13:06 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Allen Wittenauer resolved MAPREDUCE-452.

    Resolution: Fixed

Marking this as fixed since YARN provides this capability.

> tasktracker checkpointing capability
> ------------------------------------
>                 Key: MAPREDUCE-452
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-452
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>            Reporter: Pete Wyckoff
>            Priority: Minor
> This relates to allowing a resource manager (e.g., hadoop on demand) to grow and (rarely)
shrink jobs on the fly.
> Growing is already supported. Shrinking could be done in 2 ways - (1) consider the machine
dead and allow speculative execution to take care of it or (2) moving the existing map outputs
from that machine somewhere else (another machine, dfs) - "task tracker checkpointing" 
> In the case of IO only intensive jobs,  checkpointing the tasktracker doesn't do much
for you.  But, in the case of CPU or other scarce resource (e.g., a DB or Webpage cache...),
the checkpointing could be very useful.  The question is how often is this the case and how

This message was sent by Atlassian JIRA

View raw message