flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1019) Rework RPC service
Date Sat, 11 Oct 2014 18:41:34 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14168295#comment-14168295

ASF GitHub Bot commented on FLINK-1019:

GitHub user tillrohrmann opened a pull request:


    [FLINK-1019] Implementation of akka based RPC system

    Replaced the old Nephele RPC service with akka based system. Thus, several components
are now implemented as actors. This includes the JobManager, TaskManager, MemoryArchivist,
JobClient. The legacy RPC service and the corresponding protocols are removed. 
    Replaced also the execution service of the ExecutionGraph by akka's futures to unify the
    Removed the LocalInstanceManager whose task is now handled by the InstanceManager. The
responsibility to create local task managers is now delegated to the FlinkMiniCluster. 
    The EventCollector was removed and the respective event classes. The events are now directly
sent to the respective listeners. 
    Moved the resources of the WebInfoServer and the WebInterfaceServer to the resource folders
of the corresponding projects. As a consequence these resources are bundled with the jars
and directly served from them by Jetty. 
    The yarn client was adapted to communicate with the actors. The former ApplicationMaster
is combined with the JobManager to simplify the system. The uber-jar is now created with maven's
shading plugin.
    Since this is a big change I would be happy if another pair of eyes could take a look
at it.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tillrohrmann/incubator-flink akka_scala

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #149
commit 98d15ca6380263c980d34038f9772f49abb4a5ba
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-03T12:53:21Z

    Added configuration constants for akka's actor system.

commit 68f096e8017b5bb5b345d20b26fcba5103b2a451
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-04T09:32:59Z

    Reworked the NepheleMiniCluster logic
    added further communication logic to jobmanager and taskmanager.

commit 963709a9c6194bd54cd2dd5429643c8c91cce8a7
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-05T09:17:16Z

    Reworked the EventCollector and the ArchiveListener as actors.
    Replacing the EventCollector and the MemoryArchivist.
    Finished EventCollector and adjusted ExecutionGraph and ExecutionVertex to register actors
as listeners.

commit da838966c8f34ef4f90dd0422cf65c90275b88aa
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-11T15:47:28Z

    Reworked the TaskManager.
    Changed RuntimeEnvironment and TaskInputSplitProvider to work with ActorRefs.

commit 509a7770ae9c13528d12f9d2be44541ad085a0af
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-12T12:48:56Z

    Implemented JobManagerProfiler and TaskManagerProfiler as actors.
    Renamed NepheleMiniCluster to FlinkMiniCluster.

commit ab86da20b0fa6f7e2559dd6e641b80a172641beb
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-12T15:46:02Z

    Removed proxis from JobClient and CliFrontend.

commit 1cf8230c6ccf6371fe4e15785672e1ede91d6313
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-15T12:07:42Z

    Removed old RPC service.

commit 545c5ff0eb7eceeb850a930e8e3db9908bb8ec26
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-18T15:09:31Z

    Ported the JobManagerITCase to actor implementation.

commit d70f71a24ba36d79e78355815629afef6abde1f0
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-19T12:48:04Z

    Updated all JobManager and TaskManager relevant test cases to work with actors.

commit 2e7bff1f4309b83bf31cb27072a24f925038222b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-22T14:14:03Z

    Removed legacy protocol classes.

commit 19ac6da337ed952ccd0a88a30942b962b1b3791b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-23T15:28:39Z

    Updated license headers of scala files to conform to scala-checkstyle.

commit b984abf455becf31d221af94bf3a1144696cda4b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-09-26T15:27:35Z

    Removed execution service from execution graph and replaced by akka's futures.

commit a3e630249579deeba397641c70f19a6ad1c85262
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-02T09:19:52Z

    Replaced the JobClient by an actor.

commit 3823d091e70dc59ce9c86ab81b145d7a20a9df3c
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-05T20:58:49Z

    Removed old java implementations of the JobManager, TaskManager, JobClient, EventCollector,
TaskOperationResult and MemoryArchivist.

commit 6da8db9542346a6bb89d68ac2e7ba74c364fe06a
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-06T12:45:37Z

    Adapted Webserver to work with actors.

commit aa49195bbe7d292b00a2fffa7bcdf6e668fb1227
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-06T13:55:11Z

    Added the webserver to the job manager.

commit 749a0a8537cc86420bd8b1172f9fbbb9048e1dc9
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-07T15:00:56Z

    Removed old events. Adapted webserver communication so that it can now talk to the actor

commit 538e1aa4c5fd04f4f2e7f9d98bcd94b9989bc4f0
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-08T10:10:25Z

    Added AKKA_LOG_LEVEL config constant.

commit 6ad2e36a409a48d7c6a7ef0b71b6409d011a8a89
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-08T12:47:43Z

    Added scala docs to actor messages.

commit f4d940d749731bd6999df44b5494c701e2a3d985
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-08T13:48:15Z

    Removed listen parameter from SubmitJobDetached message.

commit 5b1b33c5af5ab4608d7c6d0fcc592ac80da646e9
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-09T17:24:39Z

    Reworked Yarn client: Actor based communication.

commit 027829d7d17549c2421d2e0db9fd4b08ddb5751b
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-10T13:50:27Z

    Moved WebInfoServer and WebInterfaceServer resources into resource folders of respective
projects. Jetty uses the jar as base directory.

commit 7bd5b1a34c7507dfcc136a54ffdb9234f806f0a7
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-10T23:08:42Z

    Implemented proper shutdown of yarn containers if the system is shutdown via the yarn

commit d8e307cbf710c60406b270a08b102624c5145643
Author: Till Rohrmann <trohrmann@apache.org>
Date:   2014-10-11T18:37:59Z

    Resolved scala checkstyle issues.


> Rework RPC service
> ------------------
>                 Key: FLINK-1019
>                 URL: https://issues.apache.org/jira/browse/FLINK-1019
>             Project: Flink
>          Issue Type: Improvement
>          Components: Distributed Runtime
>            Reporter: Ufuk Celebi
> There is work going on to improve the RPC service by using [Akka|akka.io]. I couldn't
find a issue for it.
> Could one of the two people working on it ([~StephanEwen] and [~asteriosk]) please give
an overview of the changes and a status update?

This message was sent by Atlassian JIRA

View raw message