spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Using an external jar in the driver, in yarn-standalone mode.
Date Wed, 26 Mar 2014 16:22:09 GMT
Andrew,
Spark automatically deploys the jar on the DFS cache if it's included with
the addJars option.  It then still needs to be SparkContext.addJar'd to get
it to the executors.

-Sandy


On Wed, Mar 26, 2014 at 6:14 AM, Julien Carme <julien.carme@gmail.com>wrote:

> Hello Andrew,
>
> Thanks for the tip, I accessed the Classpath Entries on the yarn
> monitoring (in case of yarn it is not localhost:4040 but
> yarn_master:8088//proxy/[application_id]/environment). I saw that my jar
> was actually on the CLASSPATH and was available to my application.
>
> I realized that I could not access my .jar because there were something
> wrong with it, it was only partially transfered to my cluster and was
> therefore not usable. I am confused.
>
> Sorry, and thanks for your help.
>
>
>
> 2014-03-26 1:01 GMT+01:00 Andrew Lee <alee526@hotmail.com>:
>
>  Hi Julien,
>>
>> The ADD_JAR doesn't work in the command line. I checked spark-class, and
>> I couldn't find any Bash shell bringing in the variable ADD_JAR to the
>> CLASSPATH.
>>
>> Were you able to print out the properties and environment variables from
>> the Web GUI?
>>
>> localhost:4040
>>
>> This should give you an idea what is included in the current Spark shell.
>> The bin/spark-shell invokes bin/spark-class, and I don't see ADD_JAR in
>> bin/spark-class as well.
>>
>> Hi Sandy,
>>
>> Does Spark automatically deploy the JAR for you on the DFS cache if Spark
>> is running on cluster mode? I haven't got that far yet to deploy my own
>> one-time JAR for testing. Just setup a local cluster for practice.
>>
>>
>> ------------------------------
>> Date: Tue, 25 Mar 2014 23:13:58 +0100
>> Subject: Re: Using an external jar in the driver, in yarn-standalone mode.
>> From: julien.carme@gmail.com
>> To: user@spark.apache.org
>>
>>
>> Thanks for your answer.
>>
>> I am using
>> bin/spark-class  org.apache.spark.deploy.yarn.Client --jar myjar.jar
>> --class myclass ...
>>
>> myclass in myjar.jar contains a main that initializes a SparkContext in
>> yarn-standalone mode.
>>
>> Then I am using some code that uses myotherjar.jar, but I do not execute
>> it using the spark context or a RDD, so my understanding is that it is not
>> excuted on yarn slaves, only on the yarn master.
>>
>> I found no way to make my code being able to find myotherjar.jar.
>> CLASSPATH is set by Spark (or Yarn?) before being executed on the Yarn
>> Master, it is not set by me. It seems that the idea is to set
>> SPARK_CLASSPATH and/or ADD_JAR and then these jars becomes automatically
>> available in the Yarn Master but it did not work for me.
>>
>> I tried also to use sc.addJar, it did not work either, but anyway it
>> seems clear that this is used for dependancies in the code exectued on the
>> slaves, not on the master. Tell me if I am wrong
>>
>>
>>
>>
>>
>>
>> 2014-03-25 21:11 GMT+01:00 Nathan Kronenfeld <nkronenfeld@oculusinfo.com>
>> :
>>
>> by 'use ... my main program' I presume you mean you have a main function
>> in a class file you want to use as your entry point.
>>
>> SPARK_CLASSPATH, ADD_JAR, etc add your jars in on the master and the
>> workers... but they don't on the client.
>> For that, you're just using ordinary, everyday java/scala - so it just
>> has to be on the normal java classpath.
>>
>> Could that be your issue?
>>
>>           -Nathan
>>
>>
>>
>> On Tue, Mar 25, 2014 at 2:18 PM, Sandy Ryza <sandy.ryza@cloudera.com>wrote:
>>
>> Hi Julien,
>>
>> Have you called SparkContext#addJars?
>>
>> -Sandy
>>
>>
>> On Tue, Mar 25, 2014 at 10:05 AM, Julien Carme <julien.carme@gmail.com>wrote:
>>
>> Hello,
>>
>> I have been struggling for ages to use an external jar in my spark driver
>> program, in yarn-standalone mode. I just want to use in my main program,
>> outside the calls to spark functions, objects that are defined in another
>> jar.
>>
>> I tried to set SPARK_CLASSPATH, ADD_JAR, I tried to use --addJar in the
>> spark-class arguments, I always end up with a "Class not found exception"
>> when I want to use classes defined in my jar.
>>
>> Any ideas?
>>
>> Thanks a lot,
>>
>>
>>
>>
>>
>> --
>> Nathan Kronenfeld
>> Senior Visualization Developer
>> Oculus Info Inc
>> 2 Berkeley Street, Suite 600,
>> Toronto, Ontario M5A 4J5
>> Phone:  +1-416-203-3003 x 238
>> Email:  nkronenfeld@oculusinfo.com
>>
>>
>>
>

Mime
View raw message