flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3777) Add open and close methods to manage IF lifecycle
Date Tue, 07 Jun 2016 11:44:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15318333#comment-15318333

Stephan Ewen commented on FLINK-3777:

I understand that these allow initialization across input splits. My feeling is that if you
need to initialize something a million times per input format, you are probably not creating
the splits correctly.
You will still have overhead for the communication of the splits, complexity and potential
memory issues on the JobManager for millions of splits.

The right fix for that would be to create the splits more correctly, rather than add workaround
methods to make the per-split work in the input format less, at the cost of making the API
harder to maintain.

> Add open and close methods to manage IF lifecycle
> -------------------------------------------------
>                 Key: FLINK-3777
>                 URL: https://issues.apache.org/jira/browse/FLINK-3777
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.0.1
>            Reporter: Flavio Pompermaier
>            Assignee: Flavio Pompermaier
>              Labels: inputformat, lifecycle
> At the moment the opening and closing of an inputFormat are not managed, although open()
could be (improperly IMHO) simulated by configure().
> This limits the possibility to reuse expensive resources (like database connections)
and manage their release. 
> Probably the best option would be to add 2 methods (i.e. openInputformat() and closeInputFormat()
) to RichInputFormat*
> * NOTE: the best option from a "semantic" point of view would be to rename the current
open() and close() to openSplit() and closeSplit() respectively while using open() and close()
methods for the IF lifecycle management, but this would cause a backward compatibility issue...

This message was sent by Atlassian JIRA

View raw message