flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chesnay Schepler (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-8406) BucketingSink does not detect hadoop file systems
Date Mon, 22 Jan 2018 12:45:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chesnay Schepler updated FLINK-8406:
------------------------------------
    Description: 
In {{BucketingSink#createHadoopFileSystem}} one can find this piece of code:

{code}

final org.apache.flink.core.fs.FileSystem flinkFs =
 	org.apache.flink.core.fs.FileSystem.get(path.toUri());
final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
	: null;
{code}

{{FileSystem#get()}} wraps the created {{FileSystem}} in a {{SafetyNetWrapperFileSystem}},
resulting in the instanceof check to categorically fail.

We may want to replace the {{get()}} call with {{getUnguardedFileSystem()}}. We should also
look for other occurrences of similar instanceof checks.

According to a thread on the mailing list this causes the BucketingSink to be unusable. http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/BucketingSink-broken-in-flink-1-4-0-td17710.html

  was:
In {{BucketingSink#createHadoopFileSystem}} one can find this piece of code:

{code}

final org.apache.flink.core.fs.FileSystem flinkFs =
 	org.apache.flink.core.fs.FileSystem.get(path.toUri());
final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
	: null;
{code}

{{FileSystem#get()}} wraps the created {{FileSystem}} in a {{SafetyNetWrapperFileSystem}},
resulting in the instanceof check to categorically fail.

We may want to replace the {{get()}} call with {{getUnguardedFileSystem()}}. We should also
look for other occurrences of similar instanceof checks.

According to a thread on the mailing list this causes the BucketingSink to be unusable.


> BucketingSink does not detect hadoop file systems
> -------------------------------------------------
>
>                 Key: FLINK-8406
>                 URL: https://issues.apache.org/jira/browse/FLINK-8406
>             Project: Flink
>          Issue Type: Bug
>          Components: FileSystem
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Chesnay Schepler
>            Assignee: Stephan Ewen
>            Priority: Blocker
>             Fix For: 1.5.0, 1.4.1
>
>
> In {{BucketingSink#createHadoopFileSystem}} one can find this piece of code:
> {code}
> final org.apache.flink.core.fs.FileSystem flinkFs =
>  	org.apache.flink.core.fs.FileSystem.get(path.toUri());
> final FileSystem hadoopFs = (flinkFs instanceof HadoopFileSystem)
> 	? ((HadoopFileSystem) flinkFs).getHadoopFileSystem()
> 	: null;
> {code}
> {{FileSystem#get()}} wraps the created {{FileSystem}} in a {{SafetyNetWrapperFileSystem}},
resulting in the instanceof check to categorically fail.
> We may want to replace the {{get()}} call with {{getUnguardedFileSystem()}}. We should
also look for other occurrences of similar instanceof checks.
> According to a thread on the mailing list this causes the BucketingSink to be unusable.
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/BucketingSink-broken-in-flink-1-4-0-td17710.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message