flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7486) flink-mesos: Support for adding unique attribute / group_by attribute constraints
Date Wed, 20 Sep 2017 18:39:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16173650#comment-16173650
] 

ASF GitHub Bot commented on FLINK-7486:
---------------------------------------

Github user EronWright commented on a diff in the pull request:

    https://github.com/apache/flink/pull/4628#discussion_r140051191
  
    --- Diff: flink-mesos/src/main/java/org/apache/flink/mesos/runtime/clusterframework/MesosFlinkResourceManager.java
---
    @@ -663,6 +666,7 @@ private void taskTerminated(Protos.TaskID taskID, Protos.TaskStatus
status) {
     	// ------------------------------------------------------------------------
     
     	private LaunchableMesosWorker createLaunchableMesosWorker(Protos.TaskID taskID) {
    +		setCoTaskGetter();
    --- End diff --
    
    I believe that this approach is violating an Akka rule by passing a direct reference to
the RM object to the launch coordinator.   I understand the challenge to be that Fenzo's `BalancedHostAttrConstraint`
expects a `coTasksGetter` function ([ref](http://netflix.github.io/Fenzo/fenzo-core/com/netflix/fenzo/plugins/BalancedHostAttrConstraint.html#BalancedHostAttrConstraint-com.netflix.fenzo.functions.Func1-java.lang.String-int-)).
  But that function cannot keep a reference to the RM object.  
    
     An imperfect solution would be to pass the `taskIds` to the `LaunchableMesosWorker` constructor;
imperfect because the list wouldn't be dynamic.
    
    In truth we want all tasks to be balanced and so the getter is actually overkill.  If
you look at the implementation of `BalancedHostAttrConstraint` ([ref](https://github.com/Netflix/Fenzo/blob/master/fenzo-core/src/main/java/com/netflix/fenzo/plugins/BalancedHostAttrConstraint.java))
it uses the function output to lookup tasks in the `TaskTrackerState`.   A custom constraint
could simply use all tasks in `TaskTrackerState`  without need for a `coTasksGetter`.
    
    Once we get this sorted out, some of the code should be moved to a utility class.


> flink-mesos: Support for adding unique attribute / group_by attribute constraints
> ---------------------------------------------------------------------------------
>
>                 Key: FLINK-7486
>                 URL: https://issues.apache.org/jira/browse/FLINK-7486
>             Project: Flink
>          Issue Type: Improvement
>          Components: Mesos
>    Affects Versions: 1.3.2
>            Reporter: Bhumika Bayani
>            Assignee: Bhumika Bayani
>
> In our setup, we have multiple mesos-workers. Inspite of this, flink application master
most of the times ends up spawning all task-managers on same mesos-worker.
> We intend to ensure HA of task managers. We would like to make sure each task-manager
is running on different mesos-worker as well as such mesos-worker which does not share the
AZ attribute with earlier task manager instances.
> Netflix-fenzo supports adding UniqueHostAttribute and BalancedHostAttribute contraints.
Flink-mesos should also enable us to add these kind of constraints.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message