spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tathagata Das (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (SPARK-9215) Implement WAL-free Kinesis receiver that give at-least once guarantee
Date Wed, 05 Aug 2015 07:22:05 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tathagata Das resolved SPARK-9215.
----------------------------------
       Resolution: Fixed
    Fix Version/s: 1.5.0

> Implement WAL-free Kinesis receiver that give at-least once guarantee
> ---------------------------------------------------------------------
>
>                 Key: SPARK-9215
>                 URL: https://issues.apache.org/jira/browse/SPARK-9215
>             Project: Spark
>          Issue Type: Improvement
>          Components: Streaming
>    Affects Versions: 1.4.1
>            Reporter: Tathagata Das
>            Assignee: Tathagata Das
>             Fix For: 1.5.0
>
>
> Currently, the KinesisReceiver can loose some data in the case of certain failures (receiver
and driver failures). Using the write ahead logs can mitigate some of the problem, but it
is not ideal because WALs dont work with S3 (eventually consistency, etc.) which is the most
likely file system to be used in the EC2 environment. Hence, we have to take a different approach
to improving reliability for Kinesis.
> Detailed design doc - https://docs.google.com/document/d/1k0dl270EnK7uExrsCE7jYw7PYx0YC935uBcxn3p0f58/edit?usp=sharing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message