flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-1675) Rework Accumulators
Date Wed, 06 May 2015 13:36:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530555#comment-14530555
] 

Robert Metzger commented on FLINK-1675:
---------------------------------------

(3) will also resolve the issue that accumulators are only reported for successfully finished
jobs. We should also have accumulators for failed and cancelled jobs.

Streaming jobs working on infinite source will almost always be cancelled, so the accumulator
results won't be accessible.

> Rework Accumulators
> -------------------
>
>                 Key: FLINK-1675
>                 URL: https://issues.apache.org/jira/browse/FLINK-1675
>             Project: Flink
>          Issue Type: Bug
>          Components: JobManager, TaskManager
>    Affects Versions: 0.9
>            Reporter: Stephan Ewen
>             Fix For: 0.9
>
>
> The accumulators need an overhaul to address various issues:
> 1.  User defined Accumulator classes crash the client, because it is not using the user
code classloader to decode the received message.
> 2.  They should be attached to the ExecutionGraph, not the dedicated AccumulatorManager.
That makes them accessible also for archived execution graphs.
> 3.  Accumulators should be sent periodically, as part of the heart beat that sends metrics.
This allows them to be updated in real time
> 4. Accumulators should be stored fine grained (per executionvertex, or per execution)
and the final value should be on computed by merging all involved ones. This allows users
to access the per-subtask accumulators, which is often interesting.
> 5. Accumulators should subsume the aggregators by allowing to be "versioned" with a superstep.
The versioned ones should be redistributed to the cluster after each superstep.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message