spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From qihong <qc...@pivotal.io>
Subject Re: how to choose right DStream batch interval
Date Fri, 05 Sep 2014 19:09:32 GMT
repost since original msg was marked with "This post has NOT been accepted by
the mailing list yet."

I have some questions regarding DStream batch interval: 

1. if it only take 0.5 second to process the batch 99% of time, but 1% of
batches need 5 seconds to process (due to some random factor or failures),
then what's the right batch interval? 5 seconds (the worst case)? 

2. What will happen to DStream processing if 1 batch took longer than batch
interval? Can Spark recover from that? 

Thanks,
Qihong



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/how-to-choose-right-DStream-batch-interval-tp13578p13579.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message