spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nunarob <...@nunahealth.com>
Subject Re: Deploying spark applications from within Eclipse?
Date Fri, 01 Aug 2014 18:42:32 GMT
Thanks much for your help Tobias.

I am trying what you have suggested but am receiving error messages which I
think are generally related to incorrect spark configuration. Will try and
fix those and let you know how it goes.

Cheers,

-Rob


On Wed, Jul 30, 2014 at 6:01 PM, Tobias Pfeiffer [via Apache Spark User
List] <ml-node+s1001560n11007h64@n3.nabble.com> wrote:

> Hi,
>
> if your Spark driver runs in Eclipse, you should be able to get what you
> want. That is, create a main() function that creates a SparkContext with
> the correct master URL of your cluster and execute it with your standard
> Eclipse method. For example, I am using Mesos and the following works fine:
>
>         val conf = new SparkConf()
>           .setMaster("zk://HOSTNAME:2181/mesos")
>           .setAppName("My Application")
>           .setJars("/path/to/my-application-assembly-1.0.jar" :: Nil)
>           .set("spark.executor.uri",
> "/path/to/spark-1.0.0-bin-hadoop2.tgz")
>           .set("spark.mesos.coarse", "true")
>         val sc = new SparkContext(conf)
>
> However, if you change something in the cluster node processing code, you
> will have to redeploy the my-application-assembly-1.0.jar file.
>
> Tobias
>
>
> On Thu, Jul 31, 2014 at 9:20 AM, nunarob <[hidden email]
> <http://user/SendEmail.jtp?type=node&node=11007&i=0>> wrote:
>
>> Hi all,
>>
>> I'm a new Spark user and I'm interested in deploying java-based Spark
>> applications directly from Eclipse. Specifically, I'd like to write a
>> Spark/java application, have it directly submitted to a spark cluster, and
>> then see the output of the spark execution directly in the eclipse
>> console.
>>
>> Following the quick start guide, I'm able to package up a spark
>> application
>> using maven from within eclipse and then able to manually submit the jar
>> using the spark-submit script. Is there anyway to submit the packaged jar
>> to
>> the spark cluster directly from within eclipse, and to see the output in
>> the
>> eclipse console?
>>
>> Apologies if this is documented somewhere - still learning about the
>> entire
>> spark ecosystem.
>>
>> Thanks!
>>
>> -Rob
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>
>
>
> ------------------------------
>  If you reply to this email, your message will be added to the discussion
> below:
>
> http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11007.html
>  To unsubscribe from Deploying spark applications from within Eclipse?, click
> here
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=11001&code=cm9iQG51bmFoZWFsdGguY29tfDExMDAxfC00ODcwNDM5MTI=>
> .
> NAML
> <http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Deploying-spark-applications-from-within-Eclipse-tp11001p11190.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Mime
View raw message