spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonal Goyal <sonalgoy...@gmail.com>
Subject Re: How to use spark-submit
Date Mon, 12 May 2014 06:53:29 GMT
Hi Stephen,

I am using maven shade plugin for creating my uber jar. I have marked spark
dependencies as provided.

Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>




On Mon, May 12, 2014 at 1:04 AM, Stephen Boesch <javadba@gmail.com> wrote:

> HI Sonal,
>     Yes I am working towards that same idea.  How did you go about
> creating the non-spark-jar dependencies ?  The way I am doing it is a
> separate straw-man project that does not include spark but has the external
> third party jars included. Then running sbt compile:managedClasspath and
> reverse engineering the lib jars from it.  That is obviously not ideal.
>
> The maven "run" will be useful for other projects built by maven: i will
> keep in my notes.
>
> AFA sbt run-example, it requires additional libraries to be added for my
> external dependencies.  I tried several items including  ADD_JARS,
>  --driver-class-path  and combinations of extraClassPath. I have deferred
> that ad-hoc approach to finding a systematic one.
>
>
>
>
> 2014-05-08 5:26 GMT-07:00 Sonal Goyal <sonalgoyal4@gmail.com>:
>
> I am creating a jar with only my dependencies and run spark-submit through
>> my project mvn build. I have configured the mvn exec goal to the location
>> of the script. Here is how I have set it up for my app. The mainClass is my
>> driver program, and I am able to send my custom args too. Hope this helps.
>>
>> <plugin>
>> <groupId>org.codehaus.mojo</groupId>
>> <artifactId>exec-maven-plugin</artifactId>
>> <executions>
>> <execution>
>>  <goals>
>> <goal>exec</goal>
>> </goals>
>>  </execution>
>> </executions>
>> <configuration>
>>            <executable>/home/sgoyal/spark/bin/spark-submit</executable>
>>  <arguments>
>> <argument>${jars}</argument>
>>                         <argument>--class</argument>
>>                         <argument>${mainClass}</argument>
>> <argument>--arg</argument>
>>                         <argument>${spark.master}</argument>
>>                         <argument>--arg</argument>
>>                         <argument>${my app arg 1}</argument>
>>                         <argument>--arg</argument>
>>                         <argument>${my arg 2}</argument>
>> </arguments>
>> </configuration>
>> </plugin>
>>
>>
>> Best Regards,
>> Sonal
>> Nube Technologies <http://www.nubetech.co>
>>
>>  <http://in.linkedin.com/in/sonalgoyal>
>>
>>
>>
>>
>> On Wed, May 7, 2014 at 6:57 AM, Tathagata Das <
>> tathagata.das1565@gmail.com> wrote:
>>
>>> Doesnt the run-example script work for you? Also, are you on the latest
>>> commit of branch-1.0 ?
>>>
>>> TD
>>>
>>>
>>> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta <soumya.simanta@gmail.com
>>> > wrote:
>>>
>>>>
>>>>
>>>> Yes, I'm struggling with a similar problem where my class are not found
>>>> on the worker nodes. I'm using 1.0.0_SNAPSHOT.  I would really appreciate
>>>> if someone can provide some documentation on the usage of spark-submit.
>>>>
>>>> Thanks
>>>>
>>>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <javadba@gmail.com>
>>>> wrote:
>>>> >
>>>> >
>>>> > I have a spark streaming application that uses the external streaming
>>>> modules (e.g. kafka, mqtt, ..) as well.  It is not clear how to properly
>>>> invoke the spark-submit script: what are the ---driver-class-path and/or
>>>> -Dspark.executor.extraClassPath parameters required?
>>>> >
>>>> >  For reference, the following error is proving difficult to resolve:
>>>> >
>>>> > java.lang.ClassNotFoundException:
>>>> org.apache.spark.streaming.examples.StreamingExamples
>>>> >
>>>>
>>>
>>>
>>
>

Mime
View raw message