spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asim Jalis <asimja...@gmail.com>
Subject Re: QueueStream Does Not Support Checkpointing
Date Sat, 15 Aug 2015 00:49:03 GMT
Another fix might be to remove the exception that is thrown when windowing
and other stateful operations are used without checkpointing.

On Fri, Aug 14, 2015 at 5:43 PM, Asim Jalis <asimjalis@gmail.com> wrote:

> I feel the real fix here is to remove the exception from QueueInputDStream
> class by reverting the fix of
> https://issues.apache.org/jira/browse/SPARK-8630
>
> I can write another class that is identical to the QueueInputDStream class
> except it does not throw the exception. But this feels like a convoluted
> solution.
>
> Throwing exceptions to forbid behavior in code is risky because it can
> easily break legitimate uses of a class.
>
> Is there a way to reopen https://issues.apache.org/jira/browse/SPARK-8630.
> I have added a comment to it, but I am not sure if that will have that
> effect.
>
> Thanks.
>
> Asim
>
> On Fri, Aug 14, 2015 at 4:03 PM, Holden Karau <holden@pigscanfly.ca>
> wrote:
>
>> I just pushed some code that does this for spark-testing-base (
>> https://github.com/holdenk/spark-testing-base )  (its in master) and
>> will publish an updated artifact with it for tonight.
>>
>> On Fri, Aug 14, 2015 at 3:35 PM, Tathagata Das <tdas@databricks.com>
>> wrote:
>>
>>> A hacky workaround is to create a customer InputDStream that creates the
>>> right RDDs based on a function. The TestInputDStream
>>> <https://github.com/apache/spark/blob/master/streaming/src/test/scala/org/apache/spark/streaming/TestSuiteBase.scala#L61>
>>> does something similar for Spark Streaming unit tests.
>>>
>>> TD
>>>
>>> On Fri, Aug 14, 2015 at 1:04 PM, Asim Jalis <asimjalis@gmail.com> wrote:
>>>
>>>> I want to test some Spark Streaming code that is using
>>>> reduceByKeyAndWindow. If I do not enable checkpointing, I get the error:
>>>>
>>>> java.lang.IllegalArgumentException: requirement failed: The checkpoint
>>>>> directory has not been set. Please set it by StreamingContext.checkpoint().
>>>>
>>>>
>>>> But if I enable checkpointing I get
>>>>
>>>> queueStream doesn't support checkpointing
>>>>
>>>>
>>>> Is there a workaround for this?
>>>>
>>>> My goal is to test that the windowing logic in my code is correct. Is
>>>> there a way to disable these strict checks or a different dstream I can use
>>>> that I can populate programmatically and then use for testing?
>>>>
>>>> Thanks.
>>>>
>>>> Asim
>>>>
>>>>
>>>
>>
>>
>> --
>> Cell : 425-233-8271
>> Twitter: https://twitter.com/holdenkarau
>> Linked In: https://www.linkedin.com/in/holdenkarau
>>
>
>

Mime
View raw message