spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From giive chen <thegi...@gmail.com>
Subject Re: Error reading HDFS file using spark 0.9.0 / hadoop 2.2.0 - incompatible protobuf 2.5 and 2.4.1
Date Wed, 26 Mar 2014 00:09:08 GMT
Hi

I am quite beginner in spark and I have similar issue last week. I don't
know if my issue is the same as yours. I found that my program's jar
contain protobuf and when I remove this dependency on my program's pom.xml,
rebuild my program and it works.

Here is how I solved my own issue.

Environment:

Spark 0.9, HDFS (Hadoop 2.3), Scala 2.10. My spark is hadoop 2 HDP2
prebuild version from http://spark.apache.org/downloads.html. I don't build
spark by my own.

Problem :

I use spark 0.9 example folder's word count program to connect my hdfs file
which is build on hadoop 2.3. The running command is "./bin/run-example
org.apache.spark.examples.WordCount"
It show "Caused by: java.lang.VerifyError". I survey a lot on web but
cannot get any workable solution.

How I Solve my issue

I found that if I use spark 0.9's spark-shell and it can connect hdfs file
without this problem. But if I use run-example command, it show
java.lang.VerifyError.
I think the main reason is these two command(spark-shell and run-example)'s
classpath is different.

Run-Example's classpath is $SPARK_HOME
/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar::$SPARK_HOME/conf:$SPARK_HOME/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar

Spark-Home's classpath
is :$SPARK_HOME/conf:$SPARK_HOME/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar

The class path difference is
$SPARK_HOME/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar
and it is build by exmaple program. When I look into  this jar file, I
found that it contain two protobuf which I don't know where it is from. I
remove all dependency from my example pom.xml and left only one
dependncy "spark-core".
I rebuild it and it success.

I don't know if my issue is the same as yours. I hope it can help.

Wisely Chen



On Wed, Mar 26, 2014 at 7:10 AM, Patrick Wendell <pwendell@gmail.com> wrote:

> Starting with Spark 0.9 the protobuf dependency we use is shaded and
> cannot interfere with other protobuf libaries including those in
> Hadoop. Not sure what's going on in this case. Would someone who is
> having this problem post exactly how they are building spark?
>
> - Patrick
>
> On Fri, Mar 21, 2014 at 3:49 PM, Aureliano Buendia <buendia360@gmail.com>
> wrote:
> >
> >
> >
> > On Tue, Mar 18, 2014 at 12:56 PM, Ognen Duzlevski
> > <ognen@plainvanillagames.com> wrote:
> >>
> >>
> >> On 3/18/14, 4:49 AM, dmpour23@gmail.com wrote:
> >>>
> >>> On Sunday, 2 March 2014 19:19:49 UTC+2, Aureliano Buendia  wrote:
> >>>>
> >>>> Is there a reason for spark using the older akka?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Mar 2, 2014 at 1:53 PM, 1esha <alexey.r...@gmail.com>
wrote:
> >>>>
> >>>> The problem is in akka remote. It contains files compiled with 2.4.*.
> >>>> When
> >>>>
> >>>> you run it with 2.5.* in classpath it fails like above.
> >>>>
> >>>>
> >>>>
> >>>> Looks like moving to akka 2.3 will solve this issue. Check this issue
> -
> >>>>
> >>>>
> >>>>
> https://www.assembla.com/spaces/akka/tickets/3154-use-protobuf-version-2-5-0#/activity/ticket
> :
> >>>>
> >>>>
> >>>> Is the solution to exclude the  2.4.*. dependency on protobuf or will
> >>>> thi produce more complications?
> >>
> >> I am not sure I remember what the context was around this but I run
> 0.9.0
> >> with hadoop 2.2.0 just fine.
> >
> >
> > The problem is that spark depends on an older version of akka, which
> depends
> > on an older version of protobuf (2.4).
> >
> > This means people cannot use protobuf 2.5 with spark.
> >
> >>
> >> Ognen
> >
> >
>

Mime
View raw message