beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-57) FileBasedSink should write intermediate output to a different folder
Date Thu, 03 Nov 2016 19:47:58 GMT

    [ https://issues.apache.org/jira/browse/BEAM-57?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15634012#comment-15634012
] 

ASF GitHub Bot commented on BEAM-57:
------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-beam/pull/1050


> FileBasedSink should write intermediate output to a different folder
> --------------------------------------------------------------------
>
>                 Key: BEAM-57
>                 URL: https://issues.apache.org/jira/browse/BEAM-57
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>
> Currently, FileBasedSink writes intermediate files and final output files to the same
folder.  Since deletion of intermediate files is prone to eventual consistency issues with
GCS,S3,other systems and users may use globs in subsequent pipelines, intermediate files should
be written to a different folder.
> This may be a subdirectory or a pipeline-defined temp directory (with probably some additional
subdirectories for this source).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message