spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: QueueStream Does Not Support Checkpointing
Date Fri, 14 Aug 2015 22:35:09 GMT
A hacky workaround is to create a customer InputDStream that creates the
right RDDs based on a function. The TestInputDStream
<https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala#L61>
does something similar for Spark Streaming unit tests.

TD

On Fri, Aug 14, 2015 at 1:04 PM, Asim Jalis <asimjalis@gmail.com> wrote:

> I want to test some Spark Streaming code that is using
> reduceByKeyAndWindow. If I do not enable checkpointing, I get the error:
>
> java.lang.IllegalArgumentException: requirement failed: The checkpoint
>> directory has not been set. Please set it by StreamingContext.checkpoint().
>
>
> But if I enable checkpointing I get
>
> queueStream doesn't support checkpointing
>
>
> Is there a workaround for this?
>
> My goal is to test that the windowing logic in my code is correct. Is
> there a way to disable these strict checks or a different dstream I can use
> that I can populate programmatically and then use for testing?
>
> Thanks.
>
> Asim
>
>

Mime
View raw message