spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <j...@apache.org>
Subject [jira] [Created] (SPARK-1565) Spark examples should be changed given spark-submit
Date Tue, 22 Apr 2014 07:59:14 GMT
Patrick Wendell created SPARK-1565:
--------------------------------------

             Summary: Spark examples should be changed given spark-submit
                 Key: SPARK-1565
                 URL: https://issues.apache.org/jira/browse/SPARK-1565
             Project: Spark
          Issue Type: Improvement
          Components: Examples
            Reporter: Patrick Wendell
            Assignee: Prashant Sharma
             Fix For: 1.0.0


https://github.com/pwendell/spark/commit/3fdad8ada28c1cef9b3b367993057de22295b8ed

Now that we have the spark-submit script we should make some changes to the examples:

1. We should always create a SparkConf that sets the app name and use the SparkContext constructor
that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users can do this
on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided" dependency, so that
the examples assembly jar does not include spark.

Then users can launch examples like this:

{code}
./bin/spark-submit examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
  --class org.apache.spark.examples.SparkPi \
  --arg 1000
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message