spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Asim Jalis <>
Subject QueueStream Does Not Support Checkpointing
Date Fri, 14 Aug 2015 20:04:30 GMT
I want to test some Spark Streaming code that is using
reduceByKeyAndWindow. If I do not enable checkpointing, I get the error:

java.lang.IllegalArgumentException: requirement failed: The checkpoint
> directory has not been set. Please set it by StreamingContext.checkpoint().

But if I enable checkpointing I get

queueStream doesn't support checkpointing

Is there a workaround for this?

My goal is to test that the windowing logic in my code is correct. Is there
a way to disable these strict checks or a different dstream I can use that
I can populate programmatically and then use for testing?



View raw message