flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sachingoel0101 <...@git.apache.org>
Subject [GitHub] flink pull request: [FLINK-2797][cli] Add support for running jobs...
Date Tue, 06 Oct 2015 19:26:44 GMT
Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145972139
  
    I agree. We need to have a proper handling of eager execution, and why exactly it shouldn't
be allowed. The client exits is not a very good explanation from the user viewpoint, and it
might end up breaking programs.
    Further, the collect call is used in other API methods as well, for example, in Gelly.
Detached mode in the current version will lead to failure of every such code.
    
    One question: What exactly is the significance of a detached submission and what does
the user expect when they submit the job in detached mode? From what I understand, it should
mean that the user should be able to shut down the machine on which the client is present.
They should be able to monitor the output for however long they want, stop and resume at a
later point. I'm drawing the analogy from how the `screen` program works on linux systems.

    According to this understanding of mine, the following organization would make sense:
                                                            __ __ __ __ __ __ __ __ __ __
                                                           |  ExecutorClient(invokes main)
 |
                                                           |   [Output available on web  
     |
    User(command line/ webclient) ->  |    if detached mode. Otherwise |
    [Actor system receives updates     |    sent to user client]                  |
    if not in detached mode otherwise |   [Compute job graph and         |
    shuts down after submitting]          |       submit to Job Manager]       |  -> Task
Managers
                                                           |                             
                     |
                                                           |             Job Manager     
           |
                                                           |__ __ __ __ __ __ __ __ __ __
 |
                                                              Job Manager actor system
    
    However, if the executor program is in the job manager actor system, what happens in case
of a failure? Can we recover the state of the main thread, its memory space etc. on a different
leader?
    
    I hope that added something to a discussion that is now bound to happen on detached mode.
:)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message