spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Pfeiffer <...@preferred.jp>
Subject Re: different behaviour of the same code
Date Tue, 04 Nov 2014 01:38:45 GMT
Hi,

On Fri, Oct 31, 2014 at 4:31 PM, lieyan <lieyan@yahoo.com> wrote:
>
> The code are here:  LogReg.scala
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/LogReg.scala
> >
>
> Then I click the "Run" button of the IDEA, and I get the following error
> message
> errlog.txt
> <
> http://apache-spark-user-list.1001560.n3.nabble.com/file/n17803/errlog.txt
> >
> .
> But when I export the jar file, and use *spark-submit --class
> net.yanl.spark.LogReg log_reg.jar 150000*. The program works finely.
>

I have not used the spark built-in cluster manager and I don't know how
application jar distribution is done in it. However, it seems to me that
when you use spark-submit, then spark-submit takes care of distributing
your jar file properly to all the cluster nodes, that's why it works fine.
When you run it from your IDE, it seems not to do that, that's why some
classes are not there on all cluster nodes and you run
into ClassNotFoundExceptions. If you change the master to "local[3]"
instead of "spark://master.local:7077" and run it from IDEA, does it work?

Tobias

Mime
View raw message