spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <>
Subject Re: Submitting spark jobs through yarn-client
Date Fri, 02 Jan 2015 22:46:32 GMT
.. and looking even further, it looks like the actual command tha'ts
executed starting up the JVM to run the
org.apache.spark.deploy.yarn.ExecutorLauncher is passing in "--class
'notused' --jar null".

I would assume this isn't expected but I don't see where to set these
properties or why they aren't making it through.

On Fri, Jan 2, 2015 at 5:02 PM, Corey Nolet <> wrote:

> Looking a little closer @ the file, it appears to be
> adding a $PWD/__app__.jar to the classpath but there is no __app__.jar in
> the directory pointed to by PWD. Any ideas?
> On Fri, Jan 2, 2015 at 4:20 PM, Corey Nolet <> wrote:
>> I'm trying to get a SparkContext going in a web container which is being
>> submitted through yarn-client. I'm trying two different approaches and both
>> seem to be resulting in the same error from the yarn nodemanagers:
>> 1) I'm newing up a spark context direct, manually adding all the lib jars
>> from Spark and Hadoop to the setJars() method on the SparkConf.
>> 2) I'm using SparkSubmit,main() to pass the classname and jar containing
>> my code.
>> When yarn tries to create the container, I get an exception in the driver
>> "Yarn application already ended, might be killed or not able to launch
>> application master". When I look into the logs for the nodemanager, I see
>> "NoClassDefFoundError: org/apache/spark/Logging.
>> Looking closer @ the contents of the nodemanagers, I see that the spark
>> yarn jar was renamed to __spark__.jar and placed in the app cache while the
>> rest of the libraries I specified via setJars() were all placed in the file
>> cache. Any ideas as to what may be happening? I even tried adding the
>> spark-core dependency and uber-jarring my own classes so that the
>> dependencies would be there when Yarn tries to create the container.

View raw message