flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sachingoel0101 <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Date Tue, 06 Oct 2015 11:10:11 GMT
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145824055
  
    Yes. I agree. It started out as a simple issue to support detached jobs but gotten a bit
messy.
    1. In detached mode, whether the program is run in interactive mode or normal mode, if
there is a call to an eager execution function, it will lead to an exception. [As of now.]
    2. This is where the problem started to appear more difficult. Since the accumulator addition
is entirely handled by Runtime Context, there can be no check whether the job is in detached
mode. Of course, having to store this information in two places[`ExecutionEnvironment` and
`RuntimeContext`] is poor design. 
    If the job is being run in detached mode, the accumulators are lost and user can never
access them or use them.
    But disallowing accumulators will effectively break most of existing programs. So that's
bad choice.
    Eager execution functions use just accumulators actually, so, to maintain harmony, we
have to maintain them too.
    If we just consider the first commit of this PR which adds support for executing detached
jobs, on a eager call, the main program throws an exception, because the job execution result
doesn't contain any accumulators.
    ```java
    this.flatMap(new Utils.CollectHelper<T>(id, serializer)).name("collect()")
    	.output(new DiscardingOutputFormat<T>()).name("collect() sink");
    JobExecutionResult res = getExecutionEnvironment().execute();
    ```
    I'm really not sure how to handle accumulators in detached programs. What we can do is:
    In `JobExecutionResult`, whenever an accumulator is requested, we check for the runtime
to be -1, which indicates job was exited in detached mode. Display  a warning to the user
on the Cli itself and indicate that the results of accumulators are available on the web frontend
[which will require writing them to a file on the job manager]. 
    One more thing: Since the detached mode is available on Yarn, it must throw the exception
too in case of `collect` or `print` calls. Is this the case? I can't check on a yarn cluster
myself.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message