flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8406) BucketingSink does not detect hadoop file systems
Date Mon, 22 Jan 2018 12:39:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16334208#comment-16334208

ASF GitHub Bot commented on FLINK-8406:

GitHub user StephanEwen opened a pull request:


    [FLINK-8406] [bucketing sink] Fix proper access of Hadoop File Systems

    ## What is the purpose of the change
    Fixes the access to Hadoop file systems when initializing the BucketingSink.
    ## Verifying this change
    The PR adds a unit test for the problem.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (yes / **no)**
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes
/ **no)**
      - The serializers: (yes / **no** / don't know)
      - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes / **no** / don't know)
      - The S3 file system connector: (yes / **no** / don't know)
    ## Documentation
      - Does this pull request introduce a new feature? (yes / **no)**
      - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StephanEwen/incubator-flink bucket_fix

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #5330
commit 1a0736281710fc3710df3410c83b769d51d8cb07
Author: Stephan Ewen <sewen@...>
Date:   2018-01-22T12:32:09Z

    [FLINK-8406] [bucketing sink] Fix proper access of Hadoop File Systems


> BucketingSink does not detect hadoop file systems
> -------------------------------------------------
>                 Key: FLINK-8406
>                 URL: https://issues.apache.org/jira/browse/FLINK-8406
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystem
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Chesnay Schepler
>            Assignee: Stephan Ewen
>            Priority: Blocker
>             Fix For: 1.5.0, 1.4.1
> In {{BucketingSink#createHadoopFileSystem}} one can find this piece of code:
> {code}
> final org.apache.flink.core.fs.FileSystem flinkFs =
>  	org.apache.flink.core.fs.FileSystem.get(path.toUri());
> final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
> 	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
> 	: null;
> {code}
> {{FileSystem#get()}} wraps the created {{FileSystem}} in a {{SafetyNetWrapperFileSystem}},
resulting in the instanceof check to categorically fail.
> We may want to replace the {{get()}} call with {{getUnguardedFileSystem()}}. We should
also look for other occurrences of similar instanceof checks.
> According to a thread on the mailing list this causes the BucketingSink to be unusable.

This message was sent by Atlassian JIRA

View raw message