spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From salemi <>
Subject Re: Spark Streaming + reduceByWindow(reduceFunc, invReduceFunc, windowDuration, slideDuration
Date Thu, 07 Aug 2014 02:58:06 GMT

The reason I am looking to do it differently is because the latency and
batch processing times are bad about 40 sec. I took the times from the
Streaming UI.

As you suggested I tried the window as below and still the times are bad.
 val dStream = KafkaUtils.createStream(ssc, zkQuorum, group, topicpMap)
      val eventData =",")).map(data =>
Data(data(0), data(1), data(2), data(3), data(4))).window(Minutes(15),
      val result =  eventData.transform((rdd, time) => {
        sql("SELECT count(state) FROM data WHERE state='Active'")
Any suggestions?


View this message in context:
Sent from the Apache Spark User List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message