spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: it seem like the exactly once feature not work on spark1.4
Date Fri, 17 Jul 2015 10:37:22 GMT
Yes. More information in my talk -
https://www.youtube.com/watch?v=d5UJonrruHk

On Fri, Jul 17, 2015 at 1:15 AM, JoneZhang <joyoungzhang@gmail.com> wrote:

> I see now.
>
> There are three steps in SparkStreaming + Kafka date processing
> 1.Receiving the data
> 2.Transforming the data
> 3.Pushing out the data
>
> SparkStreaming + Kafka only provide an exactly-once guarantee on step 1 & 2
> We need to ensure exactly once on step 3 by myself.
>
> More details see base on
> http://spark.apache.org/docs/latest/streaming-programming-guide.html
> <http://spark.apache.org/docs/latest/streaming-programming-guide.html>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/it-seem-like-the-exactly-once-feature-not-work-on-spark1-4-tp23871p23884.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message