flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1094) Simplify input split assignments
Date Wed, 17 Sep 2014 03:33:33 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14136717#comment-14136717

ASF GitHub Bot commented on FLINK-1094:

GitHub user StephanEwen opened a pull request:


    Major Rework of JobManager scheduling classes

    The new design separates scheduler, instance manager, and execution graph cleanly.
      - Job graphs can be incrementally constructed
      - The execution graph is a slim asynchronous state machine
      - The scheduler picks instances and slots on demand and uses sharing groups / co-location
constraints to implement resource sharing and co location
      - The management graph is removed, timestamps for state transitions are attached to
the execution graph 
    This pull request fixes also the following issues
     - FLINK-1094 (simplify input split assignment)
     - FLINK-1030 (Cleanly separate the Instance Management and Resource Assignment)
     - FLINK-1029 (Attach slot information to scheduling/assignment information in Scheduler)
     - FLINK-989 (improve not enough slots SchedulingException message)
     - FLINK-897 (Bug in scheduler for job graph instance sharing)
     - FLINK-820 (Support for unconnected data flows - runtime parts)
     - FLINK-625 (Add a fail(Exception) method to the job graph, to report problems detected
in RPC calls)
     - FLINK-230 (Job Cancellation does not work properly: "Cannot find execution graph to
job ID")
     - FLINK-165 (rework Nephele Task Hierarchy)
     - FLINK-79 (Startup time of jobs)
     - FLINK-15 (Rework Nephele Execution Graph State Machine)
     - FLINK-13 (Bug in Nephele Scheduler)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StephanEwen/incubator-flink jm_rework

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #122
commit b970b384f32ef2700df1ee66e208b13864cfeaf6
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-15T17:08:48Z

    [FLINK-989] Improve error message when not finding enough task slots.

commit 213d84ba411beea1e1d5a4e0a10c76b259b98615
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-16T14:09:14Z

    [FLINK-1030] Refactor and clean up instance managers.

commit 29ad0e56c60bddf45e79275b75de35288e9a23df
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-18T02:22:00Z

    [FLINK-1094] Reworked, improved, and testes split assigners

commit c9df0e791773a0b5d6149120176fa17e41cc75ea
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-06-22T17:05:02Z

    Unify all job vertices to one type (rather than dedicated input/output types)

commit a823010f9c05b8fd15a8ffb5621cf25a9902924a
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-06-23T17:44:18Z

    Make IDs immutable and serializable.

commit 9281c06915cb91e4a47ce6c6263a175a7a05766a
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-06-30T15:12:10Z

    Stubs for intermediate data set and related classes

commit 407b8bc1483044eb14488c7145c5d9c186adea41
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-20T11:11:23Z

    Redesign Scheduler from pre-assignment to more flexible schedule-on-demand model

commit 1a38c2be2aac004f4e07a20727c03be8f70f223f
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-21T17:09:26Z

    Adapt RPC to support primitive types as parameters and return values.

commit d40c5683b41f14f677e7ac9ca161852e63700a2e
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-07-25T01:09:49Z

    Redesign Scheduler part 2

commit b6821068322d1cfa7e4d7de503c5faad88a7bc96
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-06-30T19:16:15Z

    Refactor job graph construction to incremental attachment based

commit fbfd3146788665d5424e91460d99893e2503da16
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-08T21:29:42Z

    Remove management graph and simplify historic job status

commit 8fe71097c93e0a237a3478476ae02c3b4b0e1f3f
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-09T19:12:20Z

    Introduce execution attempts at execution vertex.
    Add tests for job event classes

commit 8ea25284b969aeb1c2edcf83ed7c7a345c49135f
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-11T05:18:56Z

    Finalize ExecutionGraph state machine and calls

commit af905ff505528dd686b7a63dc9acda92041db9b9
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-11T14:31:50Z

    Adjust ExecutionGraph state machine to TaskManager's failing model (direct transitions
to canceled)

commit 5813241a54981984e88a6e91e40c0b23a2bd69d8
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-12T12:57:54Z

    Adjusted job graph generator to new job graph classes

commit 13ebcfb9974724d76b288c8531641c7fdfff5cd9
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-12T16:00:08Z

    Adjust test logging for new execution graph tests to logback framework

commit bda267ce9aacd597a710adc5ebe5d1126aea0481
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-14T19:23:28Z

    Add proper locality of scheduling tracking to scheduler. Add local scheduling to slot
sharing groups.

commit af921858c03b61dfa70d2c8a74e21498c3eeb086
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-14T22:39:00Z

    Add options strict co-location constraints to scheduler

commit 90ca25ddaed7f8b1edff4ee1f0f7304a1b4e8ddc
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T00:43:18Z

    More graceful failing/errors/logging when canceling in early job stages

commit d61cb78106476dd986440a397d7ac915de577cec
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T01:15:33Z

    Fix buffer leak in TaskManager / test tasks

commit 16a9fddb083ca180321a4daff1fbae1a748bf5f5
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T01:45:01Z

    Fix logging in EventCollector
    Fix comparisons (null pointer safe) in JobManagerITCase

commit 2f0d7b042cb066d405921ba499d7ab0b741c1af7
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T14:51:12Z

    Better error messages at TaskManager startup and registration

commit 2fff545f8b1398e8d19209d036dee02a945983ca
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T14:52:48Z

    Fix race at TaskManager registration during startup

commit c8c64ef7c461e683ed78c45e95fca083eb0ab9bd
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T14:54:14Z

    Fix serializability for SlotSharingGroup

commit 6f53653a17717b986b7a0131036ed189a279b33d
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T14:54:38Z

    Adjust tests to new JobGraphModel

commit b9c03d5df5477f621802a938bcdb618d42acab87
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-15T19:15:07Z

    Add co-location-constraints as a special case of slot-shared scheduling

commit 372b2ac2bd572da37f5b1146b36d19f881db5da8
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-16T17:48:02Z

    Fix bug in topological sort

commit dc58aefb3a1003fbfef86f28864bb787e76ee64e
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-16T22:02:19Z

    Port streaming package to new JobGraph API and adjust all runtime-level tests

commit 62d9b3c089c87703229746c5ebe2056d42e796d0
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-16T22:03:00Z

    Improved distribution of IDs. Previous implementation lost bits due to double-to-long
multiplication and rounding.

commit 7d01f8640634af0d7caa67070e6cabca6dcb91d7
Author: Stephan Ewen <sewen@apache.org>
Date:   2014-09-16T22:04:28Z

    Fix failure exception message report back to client


> Simplify input split assignments
> --------------------------------
>                 Key: FLINK-1094
>                 URL: https://issues.apache.org/jira/browse/FLINK-1094
>             Project: Flink
>          Issue Type: Improvement
>          Components: JobManager
>            Reporter: Stephan Ewen
>            Assignee: Stephan Ewen
>            Priority: Minor
> The input split assigner is currently one shared instances across jobs and inputs. The
mapping between splits, split type, and assigner is very inflexible, as it requires changing
an internal class in the runtime project.
> The following changes will make it simpler and more efficient:
>  - Attach the split assigner to the job vertex (the ExecutionJobVertex) for the task
that consumes the input splits.
>  - Move the input split assigner interfaces into the core project, as well as the simple
base implementations of the assigners (default and locality aware)
>  - Let input split producers (such as input formats) create their own assigners.

This message was sent by Atlassian JIRA

View raw message