spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gurvinder Singh <gurvinder.si...@uninett.no>
Subject Re: issue with running example code
Date Fri, 04 Jul 2014 09:16:57 GMT
In the end it turns out that the issue was caused by a config settings
in spark-defaults.conf. After removing this setting

spark.files.userClassPathFirst   true

things are back to normal. Just reporting in case f someone will have
the same issue.

- Gurvinder
On 07/03/2014 06:49 PM, Gurvinder Singh wrote:
> Just to provide more information on this issue. It seems that SPARK_HOME
> environment variable is causing the issue. If I unset the variable in
> spark-class script and run in the local mode my code runs fine without
> the exception. But if I run with SPARK_HOME, I get the exception
> mentioned below. I could run without setting SPARK_HOME but it is not
> possible to run in the cluster settings, as this tells where is spark on
> worker nodes. E.g. we are using Mesos as cluster manager, thus when set
> master to mesos we get the exception as SPARK_HOME is not set.
> 
> Just to mention again the pyspark works fine as well as spark-shell,
> only when we are running compiled jar it seems SPARK_HOME causes some
> java run time issues that we get class cast exception.
> 
> Thanks,
> Gurvinder
> On 07/01/2014 09:28 AM, Gurvinder Singh wrote:
>> Hi,
>>
>> I am having issue in running scala example code. I have tested and able
>> to run successfully python example code, but when I run the scala code I
>> get this error
>>
>> java.lang.ClassCastException: cannot assign instance of
>> org.apache.spark.examples.SparkPi$$anonfun$1 to field
>> org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of
>> org.apache.spark.rdd.MappedRDD
>>
>> I have compiled spark from the github directly and running with the
>> command as
>>
>> spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar
>> --class org.apache.spark.examples.SparkPi 5 --jars
>> /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar
>>
>> Any suggestions will be helpful.
>>
>> Thanks,
>> Gurvinder
>>
> 


Mime
View raw message