spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Or <and...@databricks.com>
Subject Re: Submitting Spark Applications using Spark Submit
Date Sat, 20 Jun 2015 02:19:15 GMT
Hi Raghav,

I'm assuming you're using standalone mode. When using the Spark EC2 scripts
you need to make sure that every machine has the most updated jars. Once
you have built on one of the nodes, you must *rsync* the Spark directory to
the rest of the nodes (see /root/spark-ec2/copy-dir).

That said, I usually build it locally on my laptop and *scp* the assembly
jar to the cluster instead of building it there. The EC2 machines often
take much longer to build for some reason. Also it's cumbersome to set up
proper IDE there.

-Andrew


2015-06-19 19:11 GMT-07:00 Raghav Shankar <raghav0110.cs@gmail.com>:

> Thanks Andrew! Is this all I have to do when using the spark ec2 script to
> setup a spark cluster? It seems to be getting an assembly jar that is not
> from my project(perhaps from a maven repo). Is there a way to make the
> ec2 script use the assembly jar that I created?
>
> Thanks,
> Raghav
>
>
> On Friday, June 19, 2015, Andrew Or <andrew@databricks.com> wrote:
>
>> Hi Raghav,
>>
>> If you want to make changes to Spark and run your application with it,
>> you may follow these steps.
>>
>> 1. git clone git@github.com:apache/spark
>> 2. cd spark; build/mvn clean package -DskipTests [...]
>> 3. make local changes
>> 4. build/mvn package -DskipTests [...] (no need to clean again here)
>> 5. bin/spark-submit --master spark://[...] --class your.main.class
>> your.jar
>>
>> No need to pass in extra --driver-java-options or
>> --driver-extra-classpath as others have suggested. When using spark-submit,
>> the main jar comes from assembly/target/scala_2.10, which is prepared
>> through "mvn package". You just have to make sure that you re-package the
>> assembly jar after each modification.
>>
>> -Andrew
>>
>> 2015-06-18 16:35 GMT-07:00 maxdml <maxdml@cs.duke.edu>:
>>
>>> You can specify the jars of your application to be included with
>>> spark-submit
>>> with the /--jars/ switch.
>>>
>>> Otherwise, are you sure that your newly compiled spark jar assembly is in
>>> assembly/target/scala-2.10/?
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-Spark-Applications-using-Spark-Submit-tp23352p23400.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>

Mime
View raw message