flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-951) Reworking of Iteration Synchronization, Accumulators and Aggregators
Date Thu, 07 May 2015 09:50:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14532355#comment-14532355

ASF GitHub Bot commented on FLINK-951:

Github user rmetzger commented on the pull request:

    Hey @markus-h, sorry for not reacting to your pull request (other changes also sometimes
need a lot of time until they are merged)
    I think @StephanEwen would be the best reviewer for this pull request. He is currently
at Strata in London. Maybe he has time to take a closer look here once he is back.

> Reworking of Iteration Synchronization, Accumulators and Aggregators
> --------------------------------------------------------------------
>                 Key: FLINK-951
>                 URL: https://issues.apache.org/jira/browse/FLINK-951
>             Project: Flink
>          Issue Type: Improvement
>          Components: Iterations, Optimizer
>    Affects Versions: 0.9
>            Reporter: Markus Holzemer
>            Assignee: Markus Holzemer
>              Labels: refactoring
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> I just realized that there is no real Jira issue for the task I am currently working
> I am currently reworking a few things regarding Iteration Synchronization, Accumulators
and Aggregators. Currently the synchronization at the end of one superstep is done through
channel events. That makes it hard to track the current status of iterations. That is why
I am changing this synchronization to use RPC calls with the JobManager, so that the JobManager
manages the current status of all iterations.
> Currently we use Accumulators outside of iterations and Aggregators inside of iterations.
Both have a similiar function, but a bit different interfaces and handling. I want to unify
these two concepts. I propose that we stick in the future to Accumulators only. Aggregators
therefore are removed and Accumulators are extended to cover the usecases Aggregators were
used fore before. The switch to RPC for iterations makes it possible to also send the current
Accumulator values at the end of each superstep, so that the JobManager (and thereby the webinterface)
will be able to print intermediate accumulation results.

This message was sent by Atlassian JIRA

View raw message