spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Use SparkStreaming to find the max of a dataset?
Date Fri, 08 Aug 2014 00:35:28 GMT
You can do the following.

var globalMax = ...

dstreamOfNumericalType.foreachRDD( rdd => {
     globalMax = math.max(rdd.max, globalMax)
})

globalMax will keep getting updated after every batch

TD



On Thu, Aug 7, 2014 at 5:31 PM, bumble123 <tc1095@att.com> wrote:

> I can't figure out how to use Spark Streaming to find the max of a 5 second
> batch of data and keep updating the max every 5 seconds. How would I do
> this?
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Use-SparkStreaming-to-find-the-max-of-a-dataset-tp11734.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message