So, Im using Intellij 13.x, and Scala Spark jobs.

Make sure you have singletons (objects, not classes), then simply debug the main function.  You’ll need to set your master to some derivation of “local”, but thats it.  Spark Streaming is kinda wonky when debugging, but data-at-rest behaves like you’d expect.



On Oct 27, 2014, at 3:03 PM, Eric Tanner <> wrote:

I am a Scala / Spark newbie (attending Paco Nathan's class).  

What I need is some advice as to how to set up intellij (or eclipse) to be able to attache to the process executing to the debugger.  I know that this is not feasible if the code is executing within the cluster.  However, if spark is running locally (on my laptop) I would like to attach the debugger process to the spark program that is running locally to be able to step through the program.

Any advice will be is helpful.



Eric Tanner
Big Data Developer


15440 Laguna Canyon, Suite 100

Irvine, CA 92618



  +1 (951) 313-9274
  +1 (949) 706-0400

Confidentiality Note: The information contained in this email and document(s) attached are for the exclusive use of the addressee and may contain confidential, privileged and non-disclosable information. If the recipient of this email is not the addressee, such recipient is strictly prohibited from reading, photocopying, distribution or otherwise using this email or its contents in any way.