spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruebenacker, Oliver A" <Oliver.Ruebenac...@altisource.com>
Subject java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -9223372036842471144
Date Tue, 28 Oct 2014 15:04:15 GMT

     Hello,

  I have a Spark app which I run with master "local[3]". When running without any persist
calls, it seems to work fine, but as soon as I add persist calls (at default storage level),
it fails at the first persist call with the message below. Unfortunately, I can't post the
code. Polling the JVM memory stats while the app is running seems to indicate that the JVM
has not yet grown to its maximum size.

  Any advice? Thanks!

     Best, Oliver

14/10/28 10:51:30 INFO storage.MemoryStore: ensureFreeSpace(-9223372036842471144) called with
curMem=1760, maxMem=3523372646
14/10/28 10:51:30 INFO storage.MemoryStore: Block rdd_1_2 stored as values in memory (estimated
size -9223372036842471400.0 B, free -9223372033343709200.0 B)
14/10/28 10:51:30 ERROR executor.Executor: Exception in task 2.0 in stage 0.0 (TID 2)
java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -9223372036842471144
       at scala.Predef$.require(Predef.scala:233)
       at org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
       at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:767)
       at org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:625)
       at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:167)
       at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
       at org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
       at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
       at org.apache.spark.scheduler.Task.run(Task.scala:54)
       at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
       at java.lang.Thread.run(Unknown Source)
14/10/28 10:51:30 INFO scheduler.TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost,
PROCESS_LOCAL, 3961 bytes)
14/10/28 10:51:30 INFO executor.Executor: Running task 3.0 in stage 0.0 (TID 3)
14/10/28 10:51:30 INFO spark.CacheManager: Partition rdd_1_3 not found, computing it
14/10/28 10:51:30 WARN scheduler.TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2, localhost):
java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative: -9223372036842471144
        scala.Predef$.require(Predef.scala:233)
        org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
        org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:767)
        org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:625)
        org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:167)
        org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        java.lang.Thread.run(Unknown Source)
14/10/28 10:51:30 ERROR scheduler.TaskSetManager: Task 2 in stage 0.0 failed 1 times; aborting
job
14/10/28 10:51:30 INFO scheduler.TaskSchedulerImpl: Cancelling stage 0
14/10/28 10:51:30 INFO scheduler.TaskSchedulerImpl: Stage 0 was cancelled
14/10/28 10:51:30 INFO executor.Executor: Executor is trying to kill task 0.0 in stage 0.0
(TID 0)
14/10/28 10:51:30 INFO executor.Executor: Executor is trying to kill task 1.0 in stage 0.0
(TID 1)
14/10/28 10:51:30 INFO executor.Executor: Executor is trying to kill task 3.0 in stage 0.0
(TID 3)
14/10/28 10:51:30 INFO scheduler.DAGScheduler: Failed to run count at XXXXX
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure:
Task 2 in stage 0.0 failed 1 times, most recent failure: Lost task 2.0 in stage 0.0 (TID 2,
localhost): java.lang.IllegalArgumentException: requirement failed: sizeInBytes was negative:
-9223372036842471144
        scala.Predef$.require(Predef.scala:233)
        org.apache.spark.storage.BlockInfo.markReady(BlockInfo.scala:55)
        org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:767)
        org.apache.spark.storage.BlockManager.putArray(BlockManager.scala:625)
        org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:167)
        org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:227)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
        org.apache.spark.scheduler.Task.run(Task.scala:54)
        org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
        java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        java.lang.Thread.run(Unknown Source)
Driver stacktrace:
       at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
       at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
       at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
       at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
       at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
       at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
       at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
       at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
       at scala.Option.foreach(Option.scala:236)
       at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
       at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
       at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
       at akka.actor.ActorCell.invoke(ActorCell.scala:456)
       at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
       at akka.dispatch.Mailbox.run(Mailbox.scala:219)
       at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
       at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
       at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
       at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
       at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)


Oliver Ruebenacker | Solutions Architect

Altisource(tm)
290 Congress St, 7th Floor | Boston, Massachusetts 02210
P: (617) 728-5582 | ext: 275585
Oliver.Ruebenacker@Altisource.com<mailto:Oliver.Ruebenacker@Altisource.com> | www.Altisource.com

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or privilege. Message
transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************

Mime
View raw message