spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-4026) Write ahead log to synchronously write received data to HDFS and recover on driver failure
Date Tue, 21 Oct 2014 01:56:33 GMT
Tathagata Das created SPARK-4026:
------------------------------------

             Summary: Write ahead log to synchronously write received data to HDFS and recover
on driver failure
                 Key: SPARK-4026
                 URL: https://issues.apache.org/jira/browse/SPARK-4026
             Project: Spark
          Issue Type: Sub-task
          Components: Streaming
            Reporter: Tathagata Das
            Assignee: Tathagata Das


As part of the effort to avoid data loss on Spark Streaming driver failure, we want to implement
a write ahead log that can write received data to HDFS. This allows the received data to be
persist across driver failures. So when the streaming driver is restarted, it can find and
reprocess all the data that were received but not processed. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message