flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1139) HadoopOutputFormat fails for DOP > 1
Date Fri, 17 Oct 2014 11:44:33 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14174961#comment-14174961

ASF GitHub Bot commented on FLINK-1139:

Github user fhueske commented on the pull request:

    The problem with this issue is, that you don't know what is happening inside of the {{jobCommit()}}
    For Hadoop FileOutputFormats it only removes the temp directory but other formats might
have important logic in there. So removing that call is not a good solution.
    I am currently working on another solution that adds a hook to the JobManager which is
called after all output tasks finished. Will open a PR soon.

> HadoopOutputFormat fails for DOP > 1
> ------------------------------------
>                 Key: FLINK-1139
>                 URL: https://issues.apache.org/jira/browse/FLINK-1139
>             Project: Flink
>          Issue Type: Bug
>          Components: Hadoop Compatibility
>            Reporter: Fabian Hueske
> HadoopOutputFormat fails for DOP > 1. The reason is that job finalization (removal
of the temp directory) is done after the first task finishes.
> All other tasks will fail in that moment, since they cannot continue to write their data
to the temp directory.

This message was sent by Atlassian JIRA

View raw message