spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <tathagata.das1...@gmail.com>
Subject Re: Streaming window operations not producing output
Date Tue, 04 Nov 2014 22:11:09 GMT
Didnt oyu get any errors in the log4j logs, saying that you have to enable
checkpointing?

TD

On Tue, Nov 4, 2014 at 7:20 AM, diogo <diogo@uken.com> wrote:

> So, to answer my own n00b question, if case anyone ever needs it. You have
> to enable checkpointing (by ssc.checkpoint(hdfsPath)). Windowed
> operations need to be *checkpointed*, otherwise windows just won't work
> (and how could they).
>
> On Tue, Oct 28, 2014 at 10:24 AM, diogo <diogo@uken.com> wrote:
>
>> Hi there, I'm trying to use Window operations on streaming, but
>> everything I perform a windowed computation, I stop getting results.
>>
>> For example:
>>
>> val wordCounts = pairs.reduceByKey(_ + _)
>> wordCounts.print()
>>
>> Will print the output to the stdout on 'batch duration' interval. Now if
>> I replace it with:
>>
>> val wordCounts = pairs.reduceByKeyAndWindow(_+_, _-_, Seconds(4),
>> Seconds(2))
>> wordCounts.print()
>>
>> It will never output. What did I get wrong?
>>
>> Thanks.
>>
>
>

Mime
View raw message