spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Nadler <jnad...@srcginc.com>
Subject Re: Streaming with Java: Expected ReduceByWindow to Return JavaDStream
Date Mon, 19 Jan 2015 17:03:01 GMT
For anyone who finds this later, looks like Jerry already took care of this
here:  https://issues.apache.org/jira/browse/SPARK-5315

Thanks!

On Sun, Jan 18, 2015 at 10:28 PM, Shao, Saisai <saisai.shao@intel.com>
wrote:

>  Hi Jeff,
>
>
>
> From my understanding it seems more like a bug, since JavaDStreamLike is
> used for Java code, return a Scala DStream is not reasonable. You can fix
> this by submitting a PR, or I can help you to fix this.
>
>
>
> Thanks
>
> Jerry
>
>
>
> *From:* Jeff Nadler [mailto:jnadler@srcginc.com]
> *Sent:* Monday, January 19, 2015 2:04 PM
> *To:* user@spark.apache.org
> *Subject:* Streaming with Java: Expected ReduceByWindow to Return
> JavaDStream
>
>
>
>
>
> Can anyone tell me if my expectations are sane?
>
>
>
> I'm trying to do a reduceByWindow using the 3-arg signature (not providing
> an inverse reduce function):
>
> JavaDStream<whatevs> reducedStream = messages.reduceByWindow((x, y) -> reduce(x,
y), Durations.*seconds*(5), Durations.*seconds*(5));
>
> This isn't building; looks like it's returning DStream not JavaDStream.
>
> From JavaDStreamLike.scala, looks like this sig returns DStream, the 4-arg sig with the
inverse reduce returns JavaDStream.
>
> *def *reduceByWindow(
>     reduceFunc: (T, T) => T,
>     windowDuration: Duration,
>     slideDuration: Duration
>   ): DStream[T] = {
>   dstream.reduceByWindow(reduceFunc, windowDuration, slideDuration)
> }
>
> So I'm just a noob.  Is this a bug or am I missing something?
>
> Thanks!
>
> Jeff Nadler
>
>
>
>

Mime
View raw message