flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2797) CLI: Missing option to submit jobs in detached mode
Date Tue, 06 Oct 2015 19:27:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945621#comment-14945621
] 

ASF GitHub Bot commented on FLINK-2797:
---------------------------------------

Github user sachingoel0101 commented on the pull request:

    https://github.com/apache/flink/pull/1214#issuecomment-145972139
  
    I agree. We need to have a proper handling of eager execution, and why exactly it shouldn't
be allowed. The client exits is not a very good explanation from the user viewpoint, and it
might end up breaking programs.
    Further, the collect call is used in other API methods as well, for example, in Gelly.
Detached mode in the current version will lead to failure of every such code.
    
    One question: What exactly is the significance of a detached submission and what does
the user expect when they submit the job in detached mode? From what I understand, it should
mean that the user should be able to shut down the machine on which the client is present.
They should be able to monitor the output for however long they want, stop and resume at a
later point. I'm drawing the analogy from how the `screen` program works on linux systems.

    According to this understanding of mine, the following organization would make sense:
                                                            __ __ __ __ __ __ __ __ __ __
                                                           |  ExecutorClient(invokes main)
 |
                                                           |   [Output available on web  
     |
    User(command line/ webclient) ->  |    if detached mode. Otherwise |
    [Actor system receives updates     |    sent to user client]                  |
    if not in detached mode otherwise |   [Compute job graph and         |
    shuts down after submitting]          |       submit to Job Manager]       |  -> Task
Managers
                                                           |                             
                     |
                                                           |             Job Manager     
           |
                                                           |__ __ __ __ __ __ __ __ __ __
 |
                                                              Job Manager actor system
    
    However, if the executor program is in the job manager actor system, what happens in case
of a failure? Can we recover the state of the main thread, its memory space etc. on a different
leader?
    
    I hope that added something to a discussion that is now bound to happen on detached mode.
:)



> CLI: Missing option to submit jobs in detached mode
> ---------------------------------------------------
>
>                 Key: FLINK-2797
>                 URL: https://issues.apache.org/jira/browse/FLINK-2797
>             Project: Flink
>          Issue Type: Bug
>          Components: Command-line client
>    Affects Versions: 0.9, 0.10
>            Reporter: Maximilian Michels
>            Assignee: Sachin Goel
>             Fix For: 0.10
>
>
> Jobs can only be submitted in detached mode using YARN but not on a standalone installation.
This has been requested by users who want to submit a job, get the job id, and later query
its status.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message