spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Craig Vanderborgh <craigvanderbo...@gmail.com>
Subject Spark Checkpointing Bug
Date Tue, 29 Oct 2013 20:51:35 GMT
I have disabled checkpointing in my Spark Streaming application with:

ssc.checkpoint(null)

This application does not need checkpointing, and there is no operation
that I'm doing (only filter operations, and, importantly, NO WINDOWING)
that should need checkpointing.  And yet I am getting this:

2013-10-29 14:47:22 ERROR LocalScheduler - Exception in task 3
java.lang.Exception: Could not compute split, block input-0-1383076026200
not found
        at spark.rdd.BlockRDD.compute(BlockRDD.scala:32)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.FilteredRDD.compute(FilteredRDD.scala:15)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.UnionPartition.iterator(UnionRDD.scala:12)
        at spark.rdd.UnionRDD.compute(UnionRDD.scala:52)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:19)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:207)
        at spark.RDD.iterator(RDD.scala:196)
        at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:127)
        at spark.scheduler.ShuffleMapTask.run(ShuffleMapTask.scala:75)
        at
spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:76)
        at
spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:49)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
        at java.util.concurrent.FutureTask.run(FutureTask.java:138)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

Why might I be getting this exception?

Thanks,
Craig Vanderborgh

Mime
View raw message