spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balakrishnan Narendran <balu.na...@gmail.com>
Subject Re: spark streaming with checkpoint
Date Mon, 26 Jan 2015 06:11:22 GMT
Yeah use streaming to gather the incoming logs and write to log file then
run a spark job evry 5 minutes to process the counts. Got it. Thanks a
lot.!!!!

On 07:07, Mon, 26 Jan 2015 Tobias Pfeiffer <tgp@preferred.jp> wrote:

> Hi,
>
> On Tue, Jan 20, 2015 at 8:16 PM, balu.naren <balu.naren@gmail.com> wrote:
>
>> I am a beginner to spark streaming. So have a basic doubt regarding
>> checkpoints. My use case is to calculate the no of unique users by day. I
>> am using reduce by key and window for this. Where my window duration is 24
>> hours and slide duration is 5 mins.
>>
> Adding to what others said, this feels more like a task for "run a Spark
> job every five minutes using cron" than using the sliding window
> functionality from Spark Streaming.
>
> Tobias
>

Mime
View raw message