spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Som Lima <somplastic...@gmail.com>
Subject Re: IDE suitable for Spark : Monitoring & Debugging Spark Jobs
Date Tue, 07 Apr 2020 21:01:28 GMT
The definitive guide
Chapter 18:
Monitoring and Debugging

"This chapter covers the key details you need to monitor and debug your
Spark Applications.  To do this , we will walk through the spark UI with an
example query designed to help you understand how to trace your  own jobs
through the executions life cycle. The example we'll look at will also help
you to understand  how to debug your jobs and where errors are likely to
occur."








On Tue, 7 Apr 2020, 18:28 Pat Ferrel, <pat@occamsmachete.com> wrote:

> IntelliJ Scala works well when debugging master=local. Has anyone used it
> for remote/cluster debugging? I’ve heard it is possible...
>
>
> From: Luiz Camargo <camargho@gmail.com> <camargho@gmail.com>
> Reply: Luiz Camargo <camargho@gmail.com> <camargho@gmail.com>
> Date: April 7, 2020 at 10:26:35 AM
> To: Dennis Suhari <d.suhari@icloud.com.invalid>
> <d.suhari@icloud.com.invalid>
> Cc: yeikel valdes <email@yeikel.com> <email@yeikel.com>,
> zahidr1000@gmail.com <zahidr1000@gmail.com> <zahidr1000@gmail.com>,
> user@spark.apache.org <user@spark.apache.org> <user@spark.apache.org>
> Subject:  Re: IDE suitable for Spark
>
> I have used IntelliJ Spark/Scala with the sbt tool
>
> On Tue, Apr 7, 2020 at 1:18 PM Dennis Suhari <d.suhari@icloud.com.invalid>
> wrote:
>
>> We are using Pycharm resp. R Studio with Spark libraries to submit Spark
>> Jobs.
>>
>> Von meinem iPhone gesendet
>>
>> Am 07.04.2020 um 18:10 schrieb yeikel valdes <email@yeikel.com>:
>>
>> 
>>
>> Zeppelin is not an IDE but a notebook.  It is helpful to experiment but
>> it is missing a lot of the features that we expect from an IDE.
>>
>> Thanks for sharing though.
>>
>> ---- On Tue, 07 Apr 2020 04:45:33 -0400 * zahidr1000@gmail.com
>> <zahidr1000@gmail.com> * wrote ----
>>
>> When I first logged on I asked if there was a suitable IDE for Spark.
>> I did get a couple of responses.
>> *Thanks.*
>>
>> I did actually find one which is suitable IDE for spark.
>> That is  *Apache Zeppelin.*
>>
>> One of many reasons it is suitable for Apache Spark is.
>> The  *up and running Stage* which involves typing *bin/zeppelin-daemon.sh
>> start*
>> Go to browser and type *http://localhost:8080 <http://localhost:8080>*
>> That's it!
>>
>> Then to
>> * Hit the ground running*
>> There are also ready to go Apache Spark examples
>> showing off the type of functionality one will be using in real life
>> production.
>>
>> Zeppelin comes with  embedded Apache Spark  and scala as default
>> interpreter with 20 + interpreters.
>> I have gone on to discover there are a number of other advantages for
>> real time production
>> environment with Zeppelin offered up by other Apache Products.
>>
>> Backbutton.co.uk
>> ¯\_(ツ)_/¯
>> ♡۶Java♡۶RMI ♡۶
>> Make Use Method {MUM}
>> makeuse.org
>> <http://www.backbutton.co.uk>
>>
>>
>>
>
> --
>
>
> Prof. Luiz Camargo
> Educador - Computação
>
>
>

Mime
View raw message