flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "YufeiLiu (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-12122) Spread out tasks evenly across all available registered TaskManagers
Date Thu, 09 Jan 2020 10:25:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-12122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011666#comment-17011666
] 

YufeiLiu commented on FLINK-12122:
----------------------------------

[~trohrmann] I noticed that this feature is evenly across available "registered" TaskManagers,
so it's might not work with on yarn mode? 
I write a test, each container has 2 slot : 
{code:java}
source.setParallelism(2).map().setParallelism(4).addsink().setParallelism(2)
{code}
In first time schedule, all source subtask are in one TM, but sink subtasks are spread out.
ResourceManager allocate container after request slot,  so this config doesn't work with source
operator. 
Is ther any way to spread out all tasks on yarn?

> Spread out tasks evenly across all available registered TaskManagers
> --------------------------------------------------------------------
>
>                 Key: FLINK-12122
>                 URL: https://issues.apache.org/jira/browse/FLINK-12122
>             Project: Flink
>          Issue Type: Sub-task
>          Components: Runtime / Coordination
>    Affects Versions: 1.6.4, 1.7.2, 1.8.0
>            Reporter: Till Rohrmann
>            Assignee: Till Rohrmann
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 1.9.2, 1.10.0
>
>         Attachments: image-2019-05-21-12-28-29-538.png, image-2019-05-21-13-02-50-251.png
>
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> With Flip-6, we changed the default behaviour how slots are assigned to {{TaskManages}}.
Instead of evenly spreading it out over all registered {{TaskManagers}}, we randomly pick
slots from {{TaskManagers}} with a tendency to first fill up a TM before using another one.
This is a regression wrt the pre Flip-6 code.
> I suggest to change the behaviour so that we try to evenly distribute slots across all
available {{TaskManagers}} by considering how many of their slots are already allocated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message