flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2834) Global round-robin for temporary directories
Date Tue, 20 Oct 2015 16:43:27 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965367#comment-14965367
] 

ASF GitHub Bot commented on FLINK-2834:
---------------------------------------

GitHub user greghogan opened a pull request:

    https://github.com/apache/flink/pull/1272

    [FLINK-2834] Global round-robin for temporary directories

    Multiple TaskManager filesystems can be used by configuring multiple temporary directories.
    This patch changes the process of spilling files from a per-operator round-robin to a
global
    round-robin such that each directory is written to in turn across all operators, reducing
    unbalanced I/O due to bunching.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/greghogan/flink 2834_global_round_robin_for_temporary_directories

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/flink/pull/1272.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1272
    
----
commit 4af4332bfdf63112e940f706a6b8fedc0b7cff5b
Author: Greg Hogan <code@greghogan.com>
Date:   2015-10-20T14:47:58Z

    [FLINK-2834] Global round-robin for temporary directories
    
    Multiple TaskManager filesystems can be used by configuring multiple temporary directories.
    This patch changes the process of spilling files from a per-operator round-robin to a
global
    round-robin such that each directory is written to in turn across all operators, reducing
    unbalanced I/O due to bunching.

----


> Global round-robin for temporary directories
> --------------------------------------------
>
>                 Key: FLINK-2834
>                 URL: https://issues.apache.org/jira/browse/FLINK-2834
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>    Affects Versions: 0.10
>            Reporter: Greg Hogan
>            Assignee: Greg Hogan
>            Priority: Minor
>
> Currently each {{FileIOChannel}} performs a local round-robin among temporary directories.
Initially every writer will be concurrently spilling to the same first directory, then the
same second directory, etc. Even when evenly dispersed this effect will naturally occur due
to the bunching effect.
> One solution is to use a global counter when selecting the temporary path.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message