spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Narrell <matt.narr...@gmail.com>
Subject Re: Scala Spark IDE help
Date Tue, 28 Oct 2014 17:01:13 GMT
So, Im using Intellij 13.x, and Scala Spark jobs.

Make sure you have singletons (objects, not classes), then simply debug the main function.
 You’ll need to set your master to some derivation of “local”, but thats it.  Spark
Streaming is kinda wonky when debugging, but data-at-rest behaves like you’d expect.

hth

mn

> On Oct 27, 2014, at 3:03 PM, Eric Tanner <eric.tanner@justenough.com> wrote:
> 
> I am a Scala / Spark newbie (attending Paco Nathan's class).  
> 
> What I need is some advice as to how to set up intellij (or eclipse) to be able to attache
to the process executing to the debugger.  I know that this is not feasible if the code is
executing within the cluster.  However, if spark is running locally (on my laptop) I would
like to attach the debugger process to the spark program that is running locally to be able
to step through the program.
> 
> Any advice will be is helpful.
> 
> Eric
> 
> -- 
> 
> 
> 
> 
> Eric Tanner
> Big Data Developer
> 
> <image005.png>
> 
> 15440 Laguna Canyon, Suite 100
> 
> Irvine, CA 92618
> 
>  
> Cell:
> Tel:
> Skype:
> Web:
> 
>   +1 (951) 313-9274 <>
>   +1 (949) 706-0400 <> 
>   e <http://tonya.nicholls.je/>ric.tanner.je <http://ric.tanner.je/>
>   www.justenough.com <http://www.justenough.com/>
> Confidentiality Note: The information contained in this email and document(s) attached
are for the exclusive use of the addressee and may contain confidential, privileged and non-disclosable
information. If the recipient of this email is not the addressee, such recipient is strictly
prohibited from reading, photocopying, distribution or otherwise using this email or its contents
in any way.
> 


Mime
View raw message