flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8406) BucketingSink does not detect hadoop file systems
Date Mon, 22 Jan 2018 14:04:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334292#comment-16334292

ASF GitHub Bot commented on FLINK-8406:

Github user StephanEwen commented on the issue:

    @zentol FYI
    Problem with this is that the tests pass even before the fix, because the fallback path
works. The meiling list thread misses some info as to why the fallback path does not work
for them. The second exception postes is a red herring, it comes from a dependency messup
that shows through child-first classloading.

> BucketingSink does not detect hadoop file systems
> -------------------------------------------------
>                 Key: FLINK-8406
>                 URL: https://issues.apache.org/jira/browse/FLINK-8406
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystem
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Chesnay Schepler
>            Assignee: Stephan Ewen
>            Priority: Blocker
>             Fix For: 1.5.0, 1.4.1
> In {{BucketingSink#createHadoopFileSystem}} one can find this piece of code:
> {code}
> final org.apache.flink.core.fs.FileSystem flinkFs =
>  	org.apache.flink.core.fs.FileSystem.get(path.toUri());
> final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
> 	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
> 	: null;
> {code}
> {{FileSystem#get()}} wraps the created {{FileSystem}} in a {{SafetyNetWrapperFileSystem}},
resulting in the instanceof check to categorically fail.
> We may want to replace the {{get()}} call with {{getUnguardedFileSystem()}}. We should
also look for other occurrences of similar instanceof checks.
> According to a thread on the mailing list this causes the BucketingSink to be unusable.

This message was sent by Atlassian JIRA

View raw message