spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: RDD union of a window in Dstream
Date Wed, 21 May 2014 18:05:15 GMT
Are you not just looking for the window() function that creates the
sliding-window RDDs in the first place? That DStreams' RDDs give you
all elements in the sliding window, and you can compute a mean or
variance as you like.

You should be able to do this quite efficiently without recomputing
each time by using reduceByWindow and a running mean / stdev formula.

On Wed, May 21, 2014 at 1:42 PM, Laeeq Ahmed <laeeqspark@yahoo.com> wrote:
> Hi,
>
> I want to do union of all RDDs in each window of DStream. I found
> Dstream.union and haven't seen anything like DStream.windowRDDUnion.
>
> Is there any way around it?
>
> I want to find mean and SD of all values which comes under each sliding
> window for which I need to union all the RDDs in each window. This is not a
> running mean and SD.
>
> Regards,
> Laeeq
>

Mime
View raw message