spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-4026) Write ahead log to synchronously write received data to HDFS and recover on driver failure
Date Tue, 21 Oct 2014 01:56:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tathagata Das updated SPARK-4026:
---------------------------------
    Priority: Critical  (was: Major)

> Write ahead log to synchronously write received data to HDFS and recover on driver failure
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-4026
>                 URL: https://issues.apache.org/jira/browse/SPARK-4026
>             Project: Spark
>          Issue Type: Sub-task
>          Components: Streaming
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>            Priority: Critical
>
> As part of the effort to avoid data loss on Spark Streaming driver failure, we want to
implement a write ahead log that can write received data to HDFS. This allows the received
data to be persist across driver failures. So when the streaming driver is restarted, it can
find and reprocess all the data that were received but not processed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message