spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shixiong Zhu <zsxw...@gmail.com>
Subject Re: weird bytecode incompatability issue between spark-core jar from mvn repo and official spark prebuilt binary
Date Thu, 18 Dec 2014 09:38:34 GMT
@Rui do you mean the spark-core jar in the maven central repo
are incompatible with the same version of the the official pre-built Spark
binary? That's really weird. I thought they should have used the same codes.

Best Regards,
Shixiong Zhu

2014-12-18 17:22 GMT+08:00 Sean Owen <sowen@cloudera.com>:
>
> Well, it's always a good idea to used matched binary versions. Here it
> is more acutely necessary. You can use a pre built binary -- if you
> use it to compile and also run. Why does it not make sense to publish
> artifacts?
>
> Not sure what you mean about core vs assembly, as the assembly
> contains all of the modules. You don't literally need the same jar
> file.
>
> On Thu, Dec 18, 2014 at 3:20 AM, Sun, Rui <rui.sun@intel.com> wrote:
> > Not using spark-submit. The App directly communicates with the Spark
> cluster
> > in standalone mode.
> >
> >
> >
> > If mark the Spark dependency as 'provided’, then the spark-core .jar
> > elsewhere must be pointe to in CLASSPATH. However, the pre-built Spark
> > binary only has an assembly jar, not having individual module jars. So
> you
> > don’t have a chance to point to a module.jar which is the same binary as
> > that in the pre-built Spark binary.
> >
> >
> >
> > Maybe the Spark distribution should contain not only the assembly jar but
> > also individual module jars. Any opinion?
> >
> >
> >
> > From: Shivaram Venkataraman [mailto:shivaram@eecs.berkeley.edu]
> > Sent: Thursday, December 18, 2014 2:20 AM
> > To: Sean Owen
> > Cc: Sun, Rui; user@spark.apache.org
> > Subject: Re: weird bytecode incompatability issue between spark-core jar
> > from mvn repo and official spark prebuilt binary
> >
> >
> >
> > Just to clarify, are you running the application using spark-submit after
> > packaging with sbt package ? One thing that might help is to mark the
> Spark
> > dependency as 'provided' as then you shouldn't have the Spark classes in
> > your jar.
> >
> >
> >
> > Thanks
> >
> > Shivaram
> >
> >
> >
> > On Wed, Dec 17, 2014 at 4:39 AM, Sean Owen <sowen@cloudera.com> wrote:
> >
> > You should use the same binaries everywhere. The problem here is that
> > anonymous functions get compiled to different names when you build
> > different (potentially) so you actually have one function being called
> > when another function is meant.
> >
> >
> > On Wed, Dec 17, 2014 at 12:07 PM, Sun, Rui <rui.sun@intel.com> wrote:
> >> Hi,
> >>
> >>
> >>
> >> I encountered a weird bytecode incompatability issue between spark-core
> >> jar
> >> from mvn repo and official spark prebuilt binary.
> >>
> >>
> >>
> >> Steps to reproduce:
> >>
> >> 1.     Download the official pre-built Spark binary 1.1.1 at
> >> http://d3kbcqa49mib13.cloudfront.net/spark-1.1.1-bin-hadoop1.tgz
> >>
> >> 2.     Launch the Spark cluster in pseudo cluster mode
> >>
> >> 3.     A small scala APP which calls RDD.saveAsObjectFile()
> >>
> >> scalaVersion := "2.10.4"
> >>
> >>
> >>
> >> libraryDependencies ++= Seq(
> >>
> >>   "org.apache.spark" %% "spark-core" % "1.1.1"
> >>
> >> )
> >>
> >>
> >>
> >> val sc = new SparkContext(args(0), "test") //args[0] is the Spark master
> >> URI
> >>
> >>   val rdd = sc.parallelize(List(1, 2, 3))
> >>
> >>   rdd.saveAsObjectFile("/tmp/mysaoftmp")
> >>
> >>           sc.stop
> >>
> >>
> >>
> >> throws an exception as follows:
> >>
> >> [error] (run-main-0) org.apache.spark.SparkException: Job aborted due to
> >> stage failure: Task 1 in stage 0.0 failed 4 times, most recent failure:
> >> Lost
> >> task 1.3 in stage 0.0 (TID 6, ray-desktop.sh.intel.com):
> >> java.lang.ClassCastException: scala.Tuple2 cannot be cast to
> >> scala.collection.Iterator
> >>
> >> [error]
>  org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
> >>
> >> [error]
>  org.apache.spark.rdd.RDD$$anonfun$13.apply(RDD.scala:596)
> >>
> >> [error]
> >> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> >>
> >> [error]
> >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> >>
> >> [error]         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> >>
> >> [error]
>  org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> >>
> >> [error]
> >> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> >>
> >> [error]         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> >>
> >> [error]
> >> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62)
> >>
> >> [error]         org.apache.spark.scheduler.Task.run(Task.scala:54)
> >>
> >> [error]
> >> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:178)
> >>
> >> [error]
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> >>
> >> [error]
> >>
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >>
> >> [error]         java.lang.Thread.run(Thread.java:701)
> >>
> >>
> >>
> >> After investigation, I found that this is caused by bytecode
> >> incompatibility
> >> issue between RDD.class in spark-core_2.10-1.1.1.jar and the pre-built
> >> spark
> >> assembly respectively.
> >>
> >>
> >>
> >> This issue also happens with spark 1.1.0.
> >>
> >>
> >>
> >> Is there anything wrong in my usage of Spark? Or anything wrong in the
> >> process of deploying Spark module jars to maven repo?
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message