spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <>
Subject [jira] [Issue Comment Deleted] (SPARK-6599) Improve usability of Kinesis-based Spark Streaming
Date Sat, 01 Aug 2015 05:36:04 GMT


Tathagata Das updated SPARK-6599:
    Comment: was deleted

(was: I am updating the title of this JIRA to better reflect the issue associated with this.
In other words, I am reducing the scope this JIRA, and marking it as resolved as all the associated
JIRAs are already reasolved in Spark 1.4. For the reliability related issues, please see JIRA

> Improve usability of Kinesis-based Spark Streaming
> --------------------------------------------------
>                 Key: SPARK-6599
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
> Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver
and driver failures). Using the write ahead logs can mitigate some of the problem, but it
is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most
likely file system to be used in the EC2 environment. Hence, we have to take a different approach
to improving reliability for Kinesis.
> A detailed design doc on how this can be achieved will be added later.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message