spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc3)
Date Tue, 21 Jan 2014 21:52:21 GMT
This isn't a new RC or a result - I just tried to make an new vote
thread and the mail clients get confused and merge it in with the old
thread :(

On Tue, Jan 21, 2014 at 1:04 PM, Andy Konwinski <andykonwinski@gmail.com> wrote:
> I think we might need to change the subject to [RESULT][VOTE] or something
> so that the apache scripts can tell when a vote thread ends, even if it's
> simply being replaced by another vote thread for rc{n+1}.
>
>
> On Tue, Jan 21, 2014 at 3:15 AM, Patrick Wendell <pwendell@gmail.com> wrote:
>
>> This vote is cancelled in favor of rc4.
>>
>>
>> On Mon, Jan 20, 2014 at 4:05 PM, Ewen Cheslack-Postava <me@ewencp.org>wrote:
>>
>>> I'm using sbt and also tried doubling all the values in the sbt/sbt
>>> script. Still fails during the tests, and leaves a sbt.ForkMain process
>>> running (and eating up 100% CPU...).
>>>
>>> I can break this out into a separate issue/thread, I just figured it
>>> might be relevant since I'd imagine this is a fairly common setup (Macbook
>>> Air).
>>>
>>> -Ewen
>>>
>>>   Reynold Xin <rxin@databricks.com>
>>>  January 20, 2014 10:15 AM
>>> That's a perm gen issue - you need to adjust the perm gem size. In sbt it
>>> should've been set automatically, but I think for Maven, you need to set
>>> the maven opts, which is documented in the build instructions.
>>>
>>>
>>>
>>>   Ewen Cheslack-Postava <ewencp@fastmail.fm>
>>>  January 19, 2014 11:35 PM
>>>  I can't get the tests to run on a Mac, 10.7.5, java -version output:
>>>
>>> java version "1.6.0_65"
>>> Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
>>> Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
>>>
>>> For reference, Spark 0.8.* build and test find on the same configuration.
>>> 0.9.0-rc3 fails *after* PrimitiveVectorSuite, I'm not sure what it's
>>> running at that time since all the tests in PrimitiveVectorSuite seem to
>>> have finished:
>>>
>>> [info] PrimitiveVectorSuite:
>>> [info] - primitive value (4 milliseconds)
>>> [info] - non-primitive value (5 milliseconds)
>>> [info] - ideal growth (4 milliseconds)
>>> [info] - ideal size (5 milliseconds)
>>> [info] - resizing (6 milliseconds)
>>> [ERROR] [01/19/2014 23:16:27.508] [spark-akka.actor.default-dispatcher-4]
>>> [ActorSystem(spark)] exception while executing timer task
>>> org.apache.spark.SparkException: Error sending message to
>>> BlockManagerMaster [message = HeartBeat(BlockManagerId(<driver>, localhost,
>>> 51634, 0))]
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:176)
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:52)
>>>     at org.apache.spark.storage.BlockManager.org
>>> $apache$spark$storage$BlockManager$$heartBeat(BlockManager.scala:97)
>>>     at
>>> org.apache.spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:135)
>>>     at akka.actor.Scheduler$$anon$9.run(Scheduler.scala:80)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anon$3$$anon$2.run(Scheduler.scala:241)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$TaskHolder.run(Scheduler.scala:464)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:281)
>>>     at
>>> akka.actor.LightArrayRevolverScheduler$$anonfun$close$1.apply(Scheduler.scala:280)
>>>     at scala.collection.Iterator$class.foreach(Iterator.scala:727)
>>>     at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
>>>     at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
>>>     at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
>>>     at akka.actor.LightArrayRevolverScheduler.close(Scheduler.scala:279)
>>>     at akka.actor.ActorSystemImpl.stopScheduler(ActorSystem.scala:630)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply$mcV$sp(ActorSystem.scala:582)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$_start$1.apply(ActorSystem.scala:582)
>>>     at akka.actor.ActorSystemImpl$$anon$3.run(ActorSystem.scala:596)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.runNext$1(ActorSystem.scala:750)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply$mcV$sp(ActorSystem.scala:753)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks$$anonfun$run$1.apply(ActorSystem.scala:746)
>>>     at akka.util.ReentrantGuard.withGuard(LockUtil.scala:15)
>>>     at
>>> akka.actor.ActorSystemImpl$TerminationCallbacks.run(ActorSystem.scala:746)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>     at
>>> akka.actor.ActorSystemImpl$$anonfun$terminationCallbacks$1.apply(ActorSystem.scala:593)
>>>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>     at
>>> akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
>>>     at
>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>     at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
>>>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
>>>     at
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>     at
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>> Caused by: akka.pattern.AskTimeoutException:
>>> Recipient[Actor[akka://spark/user/BlockManagerMaster#927284646]] had
>>> already been terminated.
>>>     at akka.pattern.AskableActorRef$.ask$extension(AskSupport.scala:134)
>>>     at
>>> org.apache.spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:161)
>>>     ... 39 more
>>>
>>> [ERROR] [01/19/2014 23:20:03.131]
>>> [sparkWorker2-akka.actor.default-dispatcher-19] [ActorSystem(sparkWorker2)]
>>> Uncaught fatal error from thread
>>> [sparkWorker2-akka.actor.default-dispatcher-19] shutting down ActorSystem
>>> [sparkWorker2]
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> [ERROR] [01/19/2014 23:20:05.226]
>>> [sparkWorker2-akka.actor.default-dispatcher-2] [ActorSystem(sparkWorker2)]
>>> Uncaught fatal error from thread
>>> [sparkWorker2-akka.actor.default-dispatcher-2] shutting down ActorSystem
>>> [sparkWorker2]
>>> java.lang.OutOfMemoryError: PermGen space
>>>
>>> [... snip ... followed by a bunch more like these ]
>>>
>>> It looks like it's just a heap size issue. But since it's leaving a
>>> process running that has -Xmx3g explicitly specified on the command line
>>> I'm surprised it works for everyone else but consistently fails for me.
>>> Almost certainly not a blocker, but some memory setting in the tests might
>>> need adjustment.
>>>
>>> -Ewen
>>>   Henry Saputra <henry.saputra@gmail.com>
>>>  January 19, 2014 10:07 PM
>>> Ah yes, makes sense, thanks!
>>>
>>> - Henry
>>>   Patrick Wendell <pwendell@gmail.com>
>>>  January 19, 2014 10:01 PM
>>> Eventually the notes get posted on the apache website. I attached them
>>> to this e-mail so that people can get a sense of what is in the
>>> release before they vote on it.
>>>   Henry Saputra <henry.saputra@gmail.com>
>>>  January 19, 2014 9:57 PM
>>> Hi Patrick, quick question, where are you planning to add the release
>>> notes?
>>> I dont think it is part of the source, is it?
>>>
>>> - Henry
>>>
>>>
>>

Mime
View raw message