spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurvinder Singh <gurvinder.si...@uninett.no>
Subject Re: issue with running example code
Date Thu, 03 Jul 2014 16:49:13 GMT
Just to provide more information on this issue. It seems that SPARK_HOME
environment variable is causing the issue. If I unset the variable in
spark-class script and run in the local mode my code runs fine without
the exception. But if I run with SPARK_HOME, I get the exception
mentioned below. I could run without setting SPARK_HOME but it is not
possible to run in the cluster settings, as this tells where is spark on
worker nodes. E.g. we are using Mesos as cluster manager, thus when set
master to mesos we get the exception as SPARK_HOME is not set.

Just to mention again the pyspark works fine as well as spark-shell,
only when we are running compiled jar it seems SPARK_HOME causes some
java run time issues that we get class cast exception.

Thanks,
Gurvinder
On 07/01/2014 09:28 AM, Gurvinder Singh wrote:
> Hi,
> 
> I am having issue in running scala example code. I have tested and able
> to run successfully python example code, but when I run the scala code I
> get this error
> 
> java.lang.ClassCastException: cannot assign instance of
> org.apache.spark.examples.SparkPi$$anonfun$1 to field
> org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of
> org.apache.spark.rdd.MappedRDD
> 
> I have compiled spark from the github directly and running with the
> command as
> 
> spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar
> --class org.apache.spark.examples.SparkPi 5 --jars
> /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar
> 
> Any suggestions will be helpful.
> 
> Thanks,
> Gurvinder
> 


Mime
View raw message