hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12979) IOE in S3a: ${hadoop.tmp.dir}/s3a not configured
Date Wed, 30 Mar 2016 14:47:25 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15218072#comment-15218072

Steve Loughran commented on HADOOP-12979:

The problem is the branch take if there's no s3 buffer dir defined. the else clause is broken;
it's looking for a config option which isn't there.
    if (conf.get(BUFFER_DIR, null) != null) {
      lDirAlloc = new LocalDirAllocator(BUFFER_DIR);
    } else {
      lDirAlloc = new LocalDirAllocator("${hadoop.tmp.dir}/s3a");   // HERE

The fix should be to set {{BUFFER_DIR}} to the full path desired, create the {{LocalDirAllocator(BUFFER_DIR)}}
from the (possibly enhanced) config

> IOE in S3a:  ${hadoop.tmp.dir}/s3a not configured
> -------------------------------------------------
>                 Key: HADOOP-12979
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12979
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
> Running some spark s3a tests trigger an NPE in Hadoop <=2/7l IOE in 2.8 saying "${hadoop.tmp.dir}/s3a
not configured".
> That's correct: there is no configuration option on the conf called "${hadoop.tmp.dir}/s3a
". There may be one called {{hadoop.tmp.dir}}, however.
> Essentially s3a is sending the wrong config option down, if it can't find {{fs.s3a.buffer.dir}}

This message was sent by Atlassian JIRA

View raw message