spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Boesch <java...@gmail.com>
Subject Re: Spark code development practice
Date Fri, 06 Mar 2015 01:26:10 GMT
Hi Xi,
  Yes,
  You can do the following:

val sc = new SparkContext("local[2]", "mptest")

// or .. val sc = new SparkContext("spark://master:7070", "mptest")

val fileDataRdd = sc.textFile("/path/to/dir")

val fileLines = fileDataRdd.take(100)


The key here - i.e. the answer to your specific question - is the creation
of the SparkContext : that is one of the things spark-submit is doing for
you

2015-03-05 17:19 GMT-08:00 Xi Shen <davidshen84@gmail.com>:

> Hi,
>
> I am new to Spark. I see every spark program has a main() function. I
> wonder if I can run the spark program directly, without using spark-submit.
> I think it will be easier for early development and debug.
>
>
> Thanks,
> David
>
>

Mime
View raw message