spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <>
Subject [jira] [Commented] (SPARK-6599) Improve usability and reliability of Kinesis stream
Date Sat, 01 Aug 2015 06:18:04 GMT


Tathagata Das commented on SPARK-6599:

Apologies to all who received mails because they were watching. I did a clean up of this JIRA
and all the related JIRAs.

> Improve usability and reliability of Kinesis stream
> ---------------------------------------------------
>                 Key: SPARK-6599
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
> Usability improvements: 
> API improvements, AWS SDK upgrades, etc.
> Reliability improvements:
> Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver
and driver failures). Using the write ahead logs can mitigate some of the problem, but it
is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most
likely file system to be used in the EC2 environment. Hence, we have to take a different approach
to improving reliability for Kinesis. See
for more details.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message