flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1003) Spread out scheduling strategy
Date Sat, 18 Jun 2016 10:41:05 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337744#comment-15337744

ASF GitHub Bot commented on FLINK-1003:

GitHub user gyfora opened a pull request:


    [FLINK-1003] [WIP] Spread out scheduling of tasks

    This is a working progress PR with the core functionality implemented but no tests yet.
    As this is a highly critical part of the system I would like to get some initial feedback
before proceeding to write / change a huge amount of tests :)
    About the functionality:
    This is an adaptation of https://github.com/apache/flink/pull/60 to the current flink
scheduler. Instead of preferring local instances when scheduling new task slots the new scheduling
strategy allows users to balance the load on the different task managers.
    Every time a new task needs to be scheduled the scheduler considers all instances that
satisfy the scheduling constraints (has available nodes + locality constraints) and picks
the one with the smallest load. Load is calculated by the percentage of task slots occupied
in a given task manager.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/gyfora/flink scheduling

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2129
commit f895fd71f392482cf0a50e32dc637f7885995c4e
Author: Gyula Fora <gyfora@apache.org>
Date:   2016-06-18T10:19:13Z

    [FLINK-1003] Spread out scheduling of tasks


> Spread out scheduling strategy
> ------------------------------
>                 Key: FLINK-1003
>                 URL: https://issues.apache.org/jira/browse/FLINK-1003
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Till Rohrmann
>            Assignee: Gyula Fora
> Currently the Flink scheduler tries to fill one instance completely before the tasks
are deployed to another instance. This is a good behaviour in multi-user and multi-job scenarios
but it wastes resources if one wants to use the complete cluster. Therefore, another scheduling
strategy where the load among the different instances is kept balanced might be useful. This
spread out strategy will deploy the tasks such that the overall work is equally distributed.

This message was sent by Atlassian JIRA

View raw message