spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: Spark Checkpointing Bug
Date Thu, 31 Oct 2013 05:46:09 GMT
Hey Craig,

Maybe I'm missing something, but I don't see any evidence that your
code is doing checkpointing.

This error is caused by your job looking for an input block that it
thinks should exist from the receiver, but it can't find it.

What type of input are you using (kafka?). Also, does this happen all
the time or is it non-deterministic?

- Patrick

On Tue, Oct 29, 2013 at 1:51 PM, Craig Vanderborgh
<craigvanderborgh@gmail.com> wrote:
> I have disabled checkpointing in my Spark Streaming application with:
>
> ssc.checkpoint(null)
>
> This application does not need checkpointing, and there is no operation that
> I'm doing (only filter operations, and, importantly, NO WINDOWING) that
> should need checkpointing.  And yet I am getting this:
>
> 2013-10-29 14:47:22 ERROR LocalScheduler - Exception in task 3
> java.lang.Exception: Could not compute split, block input-0-1383076026200
> not found
>         at spark.rdd.BlockRDD.compute(BlockRDD.scala:32)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.rdd.FilteredRDD.compute(FilteredRDD.scala:15)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.rdd.UnionPartition.iterator(UnionRDD.scala:12)
>         at spark.rdd.UnionRDD.compute(UnionRDD.scala:52)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:19)
>         at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
>         at spark.RDD.iterator(RDD.scala:196)
>         at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:127)
>         at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:75)
>         at
> spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:76)
>         at
> spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:49)
>         at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>         at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>         at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>         at java.lang.Thread.run(Thread.java:662)
>
> Why might I be getting this exception?
>
> Thanks,
> Craig Vanderborgh

Mime
View raw message