spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From andy petrella <andy.petre...@gmail.com>
Subject Re: Scala Spark IDE help
Date Tue, 28 Oct 2014 17:10:16 GMT
Also, I'm following to master students at the University of Liège (one for
computing prob conditional density on massive data and the other
implementing a Markov Chain method on georasters), I proposed them to use
the Spark-Notebook to learn the framework, they're quite happy with it (so
far at least).

I know it's not a dev option, but it can help for proofing some sequence of
steps - it's easier than REPL to run the whole thing when we change
intermediate step for instance, and it's lightweight compared to sbt runs
specially using testing framework).

my0.1c

cheers,


aℕdy ℙetrella
about.me/noootsab
[image: aℕdy ℙetrella on about.me]

<http://about.me/noootsab>

On Tue, Oct 28, 2014 at 6:01 PM, Matt Narrell <matt.narrell@gmail.com>
wrote:

> So, Im using Intellij 13.x, and Scala Spark jobs.
>
> Make sure you have singletons (objects, not classes), then simply debug
> the main function.  You’ll need to set your master to some derivation of
> “local”, but thats it.  Spark Streaming is kinda wonky when debugging, but
> data-at-rest behaves like you’d expect.
>
> hth
>
> mn
>
> On Oct 27, 2014, at 3:03 PM, Eric Tanner <eric.tanner@justenough.com>
> wrote:
>
> I am a Scala / Spark newbie (attending Paco Nathan's class).
>
> What I need is some advice as to how to set up intellij (or eclipse) to be
> able to attache to the process executing to the debugger.  I know that this
> is not feasible if the code is executing within the cluster.  However, if
> spark is running locally (on my laptop) I would like to attach the debugger
> process to the spark program that is running locally to be able to step
> through the program.
>
> Any advice will be is helpful.
>
> Eric
>
> --
>
>
>
>
>
> *Eric Tanner*Big Data Developer
>
> <image005.png>
>
> 15440 Laguna Canyon, Suite 100
>
> Irvine, CA 92618
>
>
> Cell:
> Tel:
> Skype:
> Web:
>
>   +1 (951) 313-9274
>   +1 (949) 706-0400
>   e <http://tonya.nicholls.je/>ric.tanner.je
>   www.justenough.com
>
> Confidentiality Note: The information contained in this email and
> document(s) attached are for the exclusive use of the addressee and may
> contain confidential, privileged and non-disclosable information. If the
> recipient of this email is not the addressee, such recipient is strictly
> prohibited from reading, photocopying, distribution or otherwise using this
> email or its contents in any way.
>
>
>

Mime
View raw message