flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mxm <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Date Tue, 06 Oct 2015 18:10:38 GMT
Github user mxm commented on the pull request:

    @sachingoel0101 You're right. The detached YARN execution mode is broken for jobs which
use eager execution and build other jobs using the results. However, the detached YARN execution
could work if the ApplicationMaster executed the program. Essentially, this means shifting
the Client to the YARN cluster.
    @StephanEwen The way Sachin was going to handle the eager execution during detached mode
is to throw an Exception. However, the current code doesn't handle all cases. It is in the
nature of an interactive session that multiple jobs can be submitted. Results from submitted
jobs can be reused not only via count/collect but also using the returned JobExecutionResult.
Therefore, it is very difficult to deal with all these cases. IMHO we can only support non-interactive
detached job submissions.
    All in all, I think this is a nice idea but implementing it has too much overhead. The
implications of the changes are not quite clear. I pledge to change this to "Won't Fix" until
we think about a proper implementation. In the meantime, we should fix the detached YARN mode
or remove it.

If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.

View raw message