spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nikaash Puri <nikaashp...@gmail.com>
Subject Re: IDE suitable for Spark
Date Tue, 07 Apr 2020 17:32:10 GMT
I think as long as you set the master in the code as the correct cluster URL, everything should
work as expected. So you should be able to place breakpoints in IntelliJ as if running in
local mode.

Get Outlook for iOS<https://aka.ms/o0ukef>
________________________________
From: Pat Ferrel <pat@occamsmachete.com>
Sent: Tuesday, April 7, 2020 10:58:39 PM
To: Luiz Camargo <camargho@gmail.com>; Dennis Suhari <d.suhari@icloud.com.invalid>
Cc: zahidr1000@gmail.com <zahidr1000@gmail.com>; user@spark.apache.org <user@spark.apache.org>;
yeikel valdes <email@yeikel.com>
Subject: Re: IDE suitable for Spark

IntelliJ Scala works well when debugging master=local. Has anyone used it for remote/cluster
debugging? I’ve heard it is possible...


From: Luiz Camargo <camargho@gmail.com><mailto:camargho@gmail.com>
Reply: Luiz Camargo <camargho@gmail.com><mailto:camargho@gmail.com>
Date: April 7, 2020 at 10:26:35 AM
To: Dennis Suhari <d.suhari@icloud.com.invalid><mailto:d.suhari@icloud.com.invalid>
Cc: yeikel valdes <email@yeikel.com><mailto:email@yeikel.com>, zahidr1000@gmail.com
<zahidr1000@gmail.com><mailto:zahidr1000@gmail.com>, user@spark.apache.org<user@spark.apache.org><mailto:user@spark.apache.org>
Subject:  Re: IDE suitable for Spark

I have used IntelliJ Spark/Scala with the sbt tool

On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari <d.suhari@icloud.com.invalid> wrote:
We are using Pycharm resp. R Studio with Spark libraries to submit Spark Jobs.

Von meinem iPhone gesendet

Am 07.04.2020 um 18:10 schrieb yeikel valdes <email@yeikel.com<mailto:email@yeikel.com>>:



Zeppelin is not an IDE but a notebook.  It is helpful to experiment but it is missing a lot
of the features that we expect from an IDE.

Thanks for sharing though.

---- On Tue, 07 Apr 2020 04:45:33 -0400 zahidr1000@gmail.com<mailto:zahidr1000@gmail.com>wrote
----

When I first logged on I asked if there was a suitable IDE for Spark.
I did get a couple of responses.
Thanks.

I did actually find one which is suitable IDE for spark.
That is  Apache Zeppelin.

One of many reasons it is suitable for Apache Spark is.
The  up and running Stage which involves typing bin/zeppelin-daemon.sh start
Go to browser and type http://localhost:8080
That's it!

Then to
Hit the ground running
There are also ready to go Apache Spark examples
showing off the type of functionality one will be using in real life production.

Zeppelin comes with  embedded Apache Spark  and scala as default interpreter with 20 + interpreters.
I have gone on to discover there are a number of other advantages for real time production
environment with Zeppelin offered up by other Apache Products.

Backbutton.co.uk<http://Backbutton.co.uk>
¯_(ツ)_/¯
♡۶Java♡۶RMI ♡۶
Make Use Method {MUM}
makeuse.org<http://makeuse.org>
<http://www.backbutton.co.uk>



--


Prof. Luiz Camargo
Educador - Computação

Mime
View raw message