spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Laeeq Ahmed <laeeqsp...@yahoo.com.INVALID>
Subject Efficient Top count in each window
Date Thu, 12 Mar 2015 18:06:22 GMT
Hi, 
I have a streaming application where am doing top 10 count in each window which seems slow.
Is there efficient way to do this.
        val counts = keyAndValues.map(x => math.round(x._3.toDouble)).countByValueAndWindow(Seconds(4),
Seconds(4))        val topCounts = counts.repartition(1).map(_.swap).transform(rdd =>
rdd.sortByKey(false)).map(_.swap).mapPartitions(rdd => rdd.take(10))
Regards,
Laeeq

Mime
View raw message