flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3677) FileInputFormat: Allow to specify include/exclude file name patterns
Date Sat, 18 Jun 2016 09:40:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337682#comment-15337682

ASF GitHub Bot commented on FLINK-3677:

Github user zentol commented on the issue:

    yes i think so.
    I have 2 more issues with this PR.
    For one the documentation isn't in the correct place i believe. The config.md file specifies
configuration parameters for Flink as a system. These parameters are set in the flink-conf.yaml,
and are not passed to the InputFormats configure() method. I believe the keys belong into
the DataSource section of the Batch Guide, as users have to set the keys using the DataSource#withParameters()
    Which brings me to the second issue: The configure() method is a somewhat antiquated way
for users to configure IO-Formats. We now usually use additional arguments in the readTextFile()
methods, or additional configuration methods like in readCsvFile(). That said, we can't modify
the methods as the ExEnv is @Public, and we can't add another 3-4 variants of readTextFile.
We also can't change the return type of readTextFile from DataSource to a more useful TextReader,
again because it's @Public.
    I'm myself not sure whether it really is an issue, and if so how to resolve it Thus I
would like others to weigh in on this one before proceeding further.

> FileInputFormat: Allow to specify include/exclude file name patterns
> --------------------------------------------------------------------
>                 Key: FLINK-3677
>                 URL: https://issues.apache.org/jira/browse/FLINK-3677
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.0
>            Reporter: Maximilian Michels
>            Assignee: Ivan Mushketyk
>            Priority: Minor
>              Labels: starter
> It would be nice to be able to specify a regular expression to filter files.

This message was sent by Atlassian JIRA

View raw message