spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark throws NoSuchFieldError when testing on cluster mode
Date Sat, 21 Jun 2014 11:06:32 GMT
This inevitably means the run-time classpath includes a different copy
of the same library/class as something in your uber jar and the
different version is taking precedence. Here it's Commons
HttpComponents. Where exactly it's coming from is specific to your
deployment, but that's the issue.

On Sat, Jun 21, 2014 at 9:30 AM, Peng Cheng <pc175@uow.edu.au> wrote:
> I have a Spark application that runs perfectly in local mode with 8 threads,
> but when deployed on a single-node cluster. It gives the following error:
>
> ROR TaskSchedulerImpl: Lost executor 0 on 192.168.42.202: Uncaught exception
> Spark assembly has been built with Hive, including Datanucleus jars on
> classpath
> 14/06/21 04:18:53 ERROR TaskSetManager: Task 2.0:0 failed 3 times; aborting
> job
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 2.0:0 failed 3 times, most recent failure: Exception
> failure in TID 7 on host 192.168.42.202: java.lang.NoSuchFieldError:
> INSTANCE
>         org.apache.http.entity.ContentType.parse(ContentType.java:229)
> ...
>
> This is weird as this error is supposed to be caught by compiler but not jvm
> (unless Spark has changed the content of a class internally, which is
> impossible because the class is in the uber-jar but not closure). Also, I
> can confirm that the class that contains INSTANCE as a property is in the
> uber jar, so there is really no reason for Spark to throw it.
>
> Here is another independent question: I've also encounter several errors
> that only appears in cluster mode, they are hard to fix because I cannot
> debug them. Is there a local cluster simulation mode that can throw all
> errors yet allows me to debug?
>
> Yours Peng
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-throws-NoSuchFieldError-when-testing-on-cluster-mode-tp8064.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

Mime
View raw message