flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3347) TaskManager (or its ActorSystem) need to restart in case they notice quarantine
Date Thu, 27 Oct 2016 12:02:58 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15611669#comment-15611669
] 

ASF GitHub Bot commented on FLINK-3347:
---------------------------------------

Github user StephanEwen commented on the issue:

    https://github.com/apache/flink/pull/2696
  
    That's a very good addition, we need something like that.
    
    After an offline discussion with @tillrohrmann we came to the following conclusion:
    
    There is a tricky problem with that pure appraoch: When the JobManager fails, all TaskManagers
will "quarantine" that JobManager's actor system after they detected the failure. That means
they exit and restart. Effectively, a JobManager failure results in all TaskManagers restarting.
    That is a bit heavy.
    
    Instead, we'll adjust this to do the following:
      - TaskManagers must not watch the JobManager via Akka. That way, JobManager failures
do not cause any quarantining on the TaskManager side.
      - The JobManager keeps watching the TaskManagers via Akka, so TaskManager failures (false
positives) still result in TaskManager quarantine, which means the TaskManager need to restart
when a TM-JM link breaks
    
    How do TaskManagers then detect JobManager failure?
      - TaskManagers send heartbeats to the JobManager anyways (accumulators, in the future
task status reconciliation). The TaskManagers use that to detect JobManager failures.
      - In high availability setups, TaskManagers notice JobManager failure also via ZooKeeper
      - In addition, in flip-6 the resource manager tells TaskManagers about JobManager container
failures



> TaskManager (or its ActorSystem) need to restart in case they notice quarantine
> -------------------------------------------------------------------------------
>
>                 Key: FLINK-3347
>                 URL: https://issues.apache.org/jira/browse/FLINK-3347
>             Project: Flink
>          Issue Type: Improvement
>          Components: TaskManager
>    Affects Versions: 0.10.1
>            Reporter: Stephan Ewen
>            Assignee: Till Rohrmann
>            Priority: Critical
>             Fix For: 1.0.0, 1.2.0, 1.1.4
>
>
> There are cases where Akka quarantines remote actor systems. In that case, no further
communication is possible with that actor system unless one of the two actor systems is restarted.
> The result is that a TaskManager is up and available, but cannot register at the JobManager
(Akka refuses connection because of the quarantined state), making the TaskManager a useless
process.
> I suggest to let the TaskManager restart itself once it notices that either it quarantined
the JobManager, or the JobManager quarantined it.
> It is possible to recognize that by listening to certain events in the actor system event
stream: http://stackoverflow.com/questions/32471088/akka-cluster-detecting-quarantined-state



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message