flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Piotr Nowojski (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-15586) BucketingSink is ignoring plugins when trying to re-instantiate the HadoopFileSystem
Date Thu, 16 Jan 2020 09:10:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17016727#comment-17016727
] 

Piotr Nowojski commented on FLINK-15586:
----------------------------------------

Removing the release blocker status for 1.10. It's true that {{BucketingSink}} doesn't work
with {{FileSystem}}s coming from plugins, but currently the only {{FileSystem}} which is unshaded
(so have to use be loaded via plugins) are presto and hadoop S3 file systems, which have never
worked with {{BucketingSink}} in the first place - so there is no regression compared to 1.9.

In most cases {{StreamingFileSink}} can be used instead of {{BucketingSink}} and {{BucketingSink}}
will be probably removed in 1.11, so probably we can ignore this bug. 

> BucketingSink is ignoring plugins when trying to re-instantiate the HadoopFileSystem
> ------------------------------------------------------------------------------------
>
>                 Key: FLINK-15586
>                 URL: https://issues.apache.org/jira/browse/FLINK-15586
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / FileSystem
>    Affects Versions: 1.10.0
>            Reporter: Piotr Nowojski
>            Priority: Major
>             Fix For: 1.11.0
>
>
> {{BucketingSink#createHadoopFileSystem}} first is loading {{org.apache.flink.core.fs.FileSystem}}
through the plugin class loader (correctly), but later in the "re-instantiate" branch it's
ignoring the plugin class loader/{{PluginManager}} classes altogether, which means it will
be looking for non existing classes in the parent class loader.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message