spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-1565) Spark examples should be changed given spark-submit
Date Tue, 22 Apr 2014 07:59:16 GMT

     [ https://issues.apache.org/jira/browse/SPARK-1565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Wendell updated SPARK-1565:
-----------------------------------

    Description: 
Example: https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed

Now that we have the spark-submit script we should make some changes to the examples:

1. We should always create a SparkConf that sets the app name and use the SparkContext constructor
that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users can do this
on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided" dependency, so that
the examples assembly jar does not include spark.

Then users can launch examples like this:

{code}
./bin/spark-submit examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
  --class org.apache.spark.examples.SparkPi \
  --arg 1000
{code}

  was:
https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed

Now that we have the spark-submit script we should make some changes to the examples:

1. We should always create a SparkConf that sets the app name and use the SparkContext constructor
that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users can do this
on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided" dependency, so that
the examples assembly jar does not include spark.

Then users can launch examples like this:

{code}
./bin/spark-submit examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
  --class org.apache.spark.examples.SparkPi \
  --arg 1000
{code}


> Spark examples should be changed given spark-submit
> ---------------------------------------------------
>
>                 Key: SPARK-1565
>                 URL: https://issues.apache.org/jira/browse/SPARK-1565
>             Project: Spark
>          Issue Type: Improvement
>          Components: Examples
>            Reporter: Patrick Wendell
>            Assignee: Prashant Sharma
>             Fix For: 1.0.0
>
>
> Example: https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed
> Now that we have the spark-submit script we should make some changes to the examples:
> 1. We should always create a SparkConf that sets the app name and use the SparkContext
constructor that accepts a conf.
> 2. We shouldn't set the master based on a command line argument, instead users can do
this on their own with spark-submit.
> 3. The examples projects should mark spark-core/streaming as a "provided" dependency,
so that the examples assembly jar does not include spark.
> Then users can launch examples like this:
> {code}
> ./bin/spark-submit examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar
\
>   --class org.apache.spark.examples.SparkPi \
>   --arg 1000
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message