spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <pwend...@gmail.com>
Subject Re: SBT gen-idea doesn't work well after merging SPARK-1776
Date Tue, 15 Jul 2014 06:02:46 GMT
Thanks DB,

Feel free to file sub-jira's under:
https://issues.apache.org/jira/browse/SPARK-2487

I've been importing the Maven build into Intellij, it might be worth
trying that as well to see if it works.

- Patrick

On Mon, Jul 14, 2014 at 4:53 PM, DB Tsai <dbtsai@dbtsai.com> wrote:
> I've a clean clone of spark master repository, and I generated the
> intellij project file by sbt gen-idea as usual. There are two issues
> we have after merging SPARK-1776 (read dependencies from Maven).
>
> 1) After SPARK-1776, sbt gen-idea will download the dependencies from
> internet even those jars are in local cache. Before merging, the
> second time we run gen-idea will not download anything but use the
> jars in cache.
>
> 2) The tests with spark local context can not be run in the intellij.
> It will show the following exception.
>
> The current workaround we've are checking out any snapshot before
> merging to gen-idea, and then switch back to current master. But this
> will not work when the master deviate too much from the latest working
> snapshot.
>
> [ERROR] [07/14/2014 16:27:49.967] [ScalaTest-run] [Remoting] Remoting
> error: [Startup timed out] [
> akka.remote.RemoteTransportException: Startup timed out
> at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
> at akka.remote.Remoting.start(Remoting.scala:191)
> at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:104)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:153)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:132)
> at org.apache.spark.mllib.util.LocalSparkContext$class.beforeAll(LocalSparkContext.scala:29)
> at org.apache.spark.mllib.optimization.LBFGSSuite.beforeAll(LBFGSSuite.scala:27)
> at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> at org.apache.spark.mllib.optimization.LBFGSSuite.beforeAll(LBFGSSuite.scala:27)
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> at org.apache.spark.mllib.optimization.LBFGSSuite.run(LBFGSSuite.scala:27)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
> at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
> at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
> at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
> at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
> at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
> at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
> at org.scalatest.tools.Runner$.run(Runner.scala:883)
> at org.scalatest.tools.Runner.run(Runner.scala)
> at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:141)
> at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:32)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
> Caused by: java.util.concurrent.TimeoutException: Futures timed out
> after [10000 milliseconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at akka.remote.Remoting.start(Remoting.scala:173)
> ... 35 more
> ]
>
> An exception or error caused a run to abort: Futures timed out after
> [10000 milliseconds]
> java.util.concurrent.TimeoutException: Futures timed out after [10000
> milliseconds]
> at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
> at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
> at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:107)
> at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
> at scala.concurrent.Await$.result(package.scala:107)
> at akka.remote.Remoting.start(Remoting.scala:173)
> at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
> at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
> at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
> at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:104)
> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:153)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:117)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:132)
> at org.apache.spark.mllib.util.LocalSparkContext$class.beforeAll(LocalSparkContext.scala:29)
> at org.apache.spark.mllib.optimization.LBFGSSuite.beforeAll(LBFGSSuite.scala:27)
> at org.scalatest.BeforeAndAfterAll$class.beforeAll(BeforeAndAfterAll.scala:187)
> at org.apache.spark.mllib.optimization.LBFGSSuite.beforeAll(LBFGSSuite.scala:27)
> at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:253)
> at org.apache.spark.mllib.optimization.LBFGSSuite.run(LBFGSSuite.scala:27)
> at org.scalatest.tools.SuiteRunner.run(SuiteRunner.scala:55)
> at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2563)
> at org.scalatest.tools.Runner$$anonfun$doRunRunRunDaDoRunRun$3.apply(Runner.scala:2557)
> at scala.collection.immutable.List.foreach(List.scala:318)
> at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:2557)
> at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1044)
> at org.scalatest.tools.Runner$$anonfun$runOptionallyWithPassFailReporter$2.apply(Runner.scala:1043)
> at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:2722)
> at org.scalatest.tools.Runner$.runOptionallyWithPassFailReporter(Runner.scala:1043)
> at org.scalatest.tools.Runner$.run(Runner.scala:883)
> at org.scalatest.tools.Runner.run(Runner.scala)
> at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.runScalaTest2(ScalaTestRunner.java:141)
> at org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner.main(ScalaTestRunner.java:32)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai

Mime
View raw message