spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <cjno...@gmail.com>
Subject Re: Submitting spark jobs through yarn-client
Date Sat, 03 Jan 2015 04:35:25 GMT
So looking @ the actual code- I see where it looks like --class 'notused'
--jar null is set on the ClientBase.scala when yarn is being run in client
mode. One thing I noticed is that the jar is being set by trying to grab
the jar's uri from the classpath resources- in this case I think it's
finding the spark-yarn jar instead of spark-assembly so when it tries to
runt the ExecutorLauncher.scala, none of the core classes (like
org.apache.spark.Logging) are going to be available on the classpath.

I hope this is the root of the issue. I'll keep this thread updated with my
findings.

On Fri, Jan 2, 2015 at 5:46 PM, Corey Nolet <cjnolet@gmail.com> wrote:

> .. and looking even further, it looks like the actual command tha'ts
> executed starting up the JVM to run the
> org.apache.spark.deploy.yarn.ExecutorLauncher is passing in "--class
> 'notused' --jar null".
>
> I would assume this isn't expected but I don't see where to set these
> properties or why they aren't making it through.
>
> On Fri, Jan 2, 2015 at 5:02 PM, Corey Nolet <cjnolet@gmail.com> wrote:
>
>> Looking a little closer @ the launch_container.sh file, it appears to be
>> adding a $PWD/__app__.jar to the classpath but there is no __app__.jar in
>> the directory pointed to by PWD. Any ideas?
>>
>> On Fri, Jan 2, 2015 at 4:20 PM, Corey Nolet <cjnolet@gmail.com> wrote:
>>
>>> I'm trying to get a SparkContext going in a web container which is being
>>> submitted through yarn-client. I'm trying two different approaches and both
>>> seem to be resulting in the same error from the yarn nodemanagers:
>>>
>>> 1) I'm newing up a spark context direct, manually adding all the lib
>>> jars from Spark and Hadoop to the setJars() method on the SparkConf.
>>>
>>> 2) I'm using SparkSubmit,main() to pass the classname and jar containing
>>> my code.
>>>
>>>
>>> When yarn tries to create the container, I get an exception in the
>>> driver "Yarn application already ended, might be killed or not able to
>>> launch application master". When I look into the logs for the nodemanager,
>>> I see "NoClassDefFoundError: org/apache/spark/Logging.
>>>
>>> Looking closer @ the contents of the nodemanagers, I see that the spark
>>> yarn jar was renamed to __spark__.jar and placed in the app cache while the
>>> rest of the libraries I specified via setJars() were all placed in the file
>>> cache. Any ideas as to what may be happening? I even tried adding the
>>> spark-core dependency and uber-jarring my own classes so that the
>>> dependencies would be there when Yarn tries to create the container.
>>>
>>
>>
>

Mime
View raw message