hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ari Rabkin (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-9) mapred.local.dir temp dir. space allocation limited by smallest area
Date Fri, 05 Nov 2010 05:03:23 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928474#action_12928474

Ari Rabkin commented on HADOOP-9:

Paul -- I'm confused. Are you sure this is the issue you meant to comment on?  The patch on
this Jira wasn't yours. 

> mapred.local.dir  temp dir. space allocation limited by smallest area
> ---------------------------------------------------------------------
>                 Key: HADOOP-9
>                 URL: https://issues.apache.org/jira/browse/HADOOP-9
>             Project: Hadoop Common
>          Issue Type: Bug
>         Environment: all
>            Reporter: Paul Baclace
>            Assignee: Ari Rabkin
>            Priority: Minor
>             Fix For: 0.19.0
>         Attachments: hadoop9.patch
> When mapred.local.dir is used to specify multiple  temp dir. areas, space allocation
limited by smallest area because the temp dir. selection algorithm is "round robin starting
from a randomish point".   When round robin is used with approximately constant sized chunks,
the smallest area runs out of space first, and this is a fatal error. 
> Workaround: only list local fs dirs in mapred.local.dir with similarly-sized available
> I wrote a patch to JobConf (currenly being tested) which uses df to check available space
(once a minute or less often) and then uses an efficient roulette selection to do allocation
weighted by magnitude of available space. 

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message