spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From diogo <di...@uken.com>
Subject Re: Streaming window operations not producing output
Date Tue, 04 Nov 2014 15:20:50 GMT
So, to answer my own n00b question, if case anyone ever needs it. You have
to enable checkpointing (by ssc.checkpoint(hdfsPath)). Windowed operations
need to be *checkpointed*, otherwise windows just won't work (and how could
they).

On Tue, Oct 28, 2014 at 10:24 AM, diogo <diogo@uken.com> wrote:

> Hi there, I'm trying to use Window operations on streaming, but everything
> I perform a windowed computation, I stop getting results.
>
> For example:
>
> val wordCounts = pairs.reduceByKey(_ + _)
> wordCounts.print()
>
> Will print the output to the stdout on 'batch duration' interval. Now if I
> replace it with:
>
> val wordCounts = pairs.reduceByKeyAndWindow(_+_, _-_, Seconds(4),
> Seconds(2))
> wordCounts.print()
>
> It will never output. What did I get wrong?
>
> Thanks.
>

Mime
View raw message