spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laeeq Ahmed <laeeqsp...@yahoo.com>
Subject Average of each RDD in Stream
Date Wed, 07 May 2014 17:43:37 GMT
Hi,

I use the following code for calculating average. The problem is that the reduce operation
return a DStream here and not a tuple as it normally does without Streaming. So how can we
get the sum and the count from the DStream. Can we cast it to tuple?


val numbers = ssc.textFileStream(args(1))
    val sumandcount = numbers.map(n => (n.toDouble, 1)).reduce{ (a, b) => (a._1 +
b._1, a._2 + b._2) }
    sumandcount.print()


Regards,
Laeeq

Mime
View raw message