beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-648) Persist and restore Aggergator values in case of recovery from failure
Date Tue, 20 Sep 2016 18:59:20 GMT
Amit Sela created BEAM-648:
------------------------------

             Summary: Persist and restore Aggergator values in case of recovery from failure
                 Key: BEAM-648
                 URL: https://issues.apache.org/jira/browse/BEAM-648
             Project: Beam
          Issue Type: Bug
          Components: runner-spark
            Reporter: Amit Sela
            Assignee: Amit Sela


While Aggregators are fault-tolerant (pending https://github.com/apache/incubator-beam/pull/909),
they will recover into their initial value. 

The SparkRunner should persist the values to durable storage and recover from the persisted
state.

One way to go would be to persist via Spark's listeners, for example "onBatchCompleted".

For more details see SPARK-5206.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message