spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jörn Franke <jornfra...@gmail.com>
Subject Re: How big the spark stream window could be ?
Date Mon, 09 May 2016 13:55:38 GMT
I do not recommend large windows. You can have small windows, store the data and then do the
reports for one hour or one day on stored data. 

> On 09 May 2016, at 05:19, "kramer2009@126.com" <kramer2009@126.com> wrote:
> 
> We have some stream data need to be calculated and considering use spark
> stream to do it.
> 
> We need to generate three kinds of reports. The reports are based on 
> 
> 1. The last 5 minutes data
> 2. The last 1 hour data
> 3. The last 24 hour data
> 
> The frequency of reports is 5 minutes. 
> 
> After reading the docs, the most obvious way to solve this seems to set up a
> spark stream with 5 minutes interval and two window which are 1 hour and 1
> day.
> 
> 
> But I am worrying that if the window is too big for one day and one hour. I
> do not have much experience on spark stream, so what is the window length in
> your environment? 
> 
> Any official docs talking about this?
> 
> 
> 
> 
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-big-the-spark-stream-window-could-be-tp26899.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message