spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Evan Chan ...@ooyala.com>
Subject Re: Akka problem when using scala command to launch Spark applications in the current 0.9.0-SNAPSHOT
Date Tue, 24 Dec 2013 18:50:42 GMT
Hi Reynold,

The default, documented methods of starting Spark all use the assembly jar,
and thus java, right?

-Evan



On Fri, Dec 20, 2013 at 11:36 PM, Reynold Xin <rxin@databricks.com> wrote:

> It took me hours to debug a problem yesterday on the latest master branch
> (0.9.0-SNAPSHOT), and I would like to share with the dev list in case
> anybody runs into this Akka problem.
>
> A little background for those of you who haven't followed closely the
> development of Spark and YARN 2.2: YARN 2.2 uses protobuf 2.5, and Akka
> uses an older version of protobuf that is not binary compatible. In order
> to have a single build that is compatible for both YARN 2.2 and pre-2.2
> YARN/Hadoop, we published a special version of Akka that builds with
> protobuf shaded (i.e. using a different package name for the protobuf
> stuff).
>
> However, it turned out Scala 2.10 includes a version of Akka jar in its
> default classpath (look at the lib folder in Scala 2.10 binary
> distribution). If you use the scala command to launch any Spark application
> on the current master branch, there is a pretty high chance that you
> wouldn't be able to create the SparkContext (stack trace at the end of the
> email). The problem is that the Akka packaged with Scala 2.10 takes
> precedence in the classloader over the special Akka version Spark includes.
>
> Before we have a good solution for this, the workaround is to use java to
> launch the application instead of scala. All you need to do is to include
> the right Scala jars (scala-library and scala-compiler) in the classpath.
> Note that the scala command is really just a simple script that calls java
> with the right classpath.
>
>
> Stack trace:
>
> java.lang.NoSuchMethodException:
> akka.remote.RemoteActorRefProvider.<init>(java.lang.String,
> akka.actor.ActorSystem$Settings, akka.event.EventStream,
> akka.actor.Scheduler, akka.actor.DynamicAccess)
> at java.lang.Class.getConstructor0(Class.java:2763)
> at java.lang.Class.getDeclaredConstructor(Class.java:2021)
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$2.apply(DynamicAccess.scala:77)
> at scala.util.Try$.apply(Try.scala:161)
> at
>
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:74)
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at
>
> akka.actor.ReflectiveDynamicAccess$$anonfun$createInstanceFor$3.apply(DynamicAccess.scala:85)
> at scala.util.Success.flatMap(Try.scala:200)
> at
>
> akka.actor.ReflectiveDynamicAccess.createInstanceFor(DynamicAccess.scala:85)
> at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:546)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
> at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
> at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:79)
> at
> org.apache.spark.SparkEnv$.createFromSystemProperties(SparkEnv.scala:120)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:106)
>



-- 
--
Evan Chan
Staff Engineer
ev@ooyala.com  |

<http://www.ooyala.com/>
<http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message