spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sonal Goyal <sonalgoy...@gmail.com>
Subject Re: How to use spark-submit
Date Tue, 13 May 2014 14:26:00 GMT
Hi Stephen,

Sorry I just use plain mvn.

Best Regards,
Sonal
Nube Technologies <http://www.nubetech.co>

<http://in.linkedin.com/in/sonalgoyal>




On Mon, May 12, 2014 at 12:29 PM, Stephen Boesch <javadba@gmail.com> wrote:

> @Sonal - makes sense.  Is the maven shade plugin runnable within sbt ? If
> so would you care to share those build.sbt (or .scala) lines?  If not, are
> you aware of a similar plugin for sbt?
>
>
>
>
> 2014-05-11 23:53 GMT-07:00 Sonal Goyal <sonalgoyal4@gmail.com>:
>
> Hi Stephen,
>>
>> I am using maven shade plugin for creating my uber jar. I have marked
>> spark dependencies as provided.
>>
>> Best Regards,
>> Sonal
>> Nube Technologies <http://www.nubetech.co>
>>
>> <http://in.linkedin.com/in/sonalgoyal>
>>
>>
>>
>>
>> On Mon, May 12, 2014 at 1:04 AM, Stephen Boesch <javadba@gmail.com>wrote:
>>
>>> HI Sonal,
>>>     Yes I am working towards that same idea.  How did you go about
>>> creating the non-spark-jar dependencies ?  The way I am doing it is a
>>> separate straw-man project that does not include spark but has the external
>>> third party jars included. Then running sbt compile:managedClasspath and
>>> reverse engineering the lib jars from it.  That is obviously not ideal.
>>>
>>> The maven "run" will be useful for other projects built by maven: i will
>>> keep in my notes.
>>>
>>> AFA sbt run-example, it requires additional libraries to be added for my
>>> external dependencies.  I tried several items including  ADD_JARS,
>>>  --driver-class-path  and combinations of extraClassPath. I have deferred
>>> that ad-hoc approach to finding a systematic one.
>>>
>>>
>>>
>>>
>>> 2014-05-08 5:26 GMT-07:00 Sonal Goyal <sonalgoyal4@gmail.com>:
>>>
>>> I am creating a jar with only my dependencies and run spark-submit
>>>> through my project mvn build. I have configured the mvn exec goal to the
>>>> location of the script. Here is how I have set it up for my app. The
>>>> mainClass is my driver program, and I am able to send my custom args too.
>>>> Hope this helps.
>>>>
>>>> <plugin>
>>>> <groupId>org.codehaus.mojo</groupId>
>>>> <artifactId>exec-maven-plugin</artifactId>
>>>> <executions>
>>>> <execution>
>>>>  <goals>
>>>> <goal>exec</goal>
>>>> </goals>
>>>>  </execution>
>>>> </executions>
>>>> <configuration>
>>>>            <executable>/home/sgoyal/spark/bin/spark-submit</executable>
>>>>  <arguments>
>>>> <argument>${jars}</argument>
>>>>                         <argument>--class</argument>
>>>>                         <argument>${mainClass}</argument>
>>>> <argument>--arg</argument>
>>>>                         <argument>${spark.master}</argument>
>>>>                         <argument>--arg</argument>
>>>>                         <argument>${my app arg 1}</argument>
>>>>                         <argument>--arg</argument>
>>>>                         <argument>${my arg 2}</argument>
>>>> </arguments>
>>>> </configuration>
>>>> </plugin>
>>>>
>>>>
>>>> Best Regards,
>>>> Sonal
>>>> Nube Technologies <http://www.nubetech.co>
>>>>
>>>>  <http://in.linkedin.com/in/sonalgoyal>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, May 7, 2014 at 6:57 AM, Tathagata Das <
>>>> tathagata.das1565@gmail.com> wrote:
>>>>
>>>>> Doesnt the run-example script work for you? Also, are you on the
>>>>> latest commit of branch-1.0 ?
>>>>>
>>>>> TD
>>>>>
>>>>>
>>>>> On Mon, May 5, 2014 at 7:51 PM, Soumya Simanta <
>>>>> soumya.simanta@gmail.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> Yes, I'm struggling with a similar problem where my class are not
>>>>>> found on the worker nodes. I'm using 1.0.0_SNAPSHOT.  I would really
>>>>>> appreciate if someone can provide some documentation on the usage
of
>>>>>> spark-submit.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> > On May 5, 2014, at 10:24 PM, Stephen Boesch <javadba@gmail.com>
>>>>>> wrote:
>>>>>> >
>>>>>> >
>>>>>> > I have a spark streaming application that uses the external
>>>>>> streaming modules (e.g. kafka, mqtt, ..) as well.  It is not clear
how to
>>>>>> properly invoke the spark-submit script: what are the ---driver-class-path
>>>>>> and/or -Dspark.executor.extraClassPath parameters required?
>>>>>> >
>>>>>> >  For reference, the following error is proving difficult to
resolve:
>>>>>> >
>>>>>> > java.lang.ClassNotFoundException:
>>>>>> org.apache.spark.streaming.examples.StreamingExamples
>>>>>> >
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message