spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (JIRA)" <>
Subject [jira] [Created] (SPARK-1565) Spark examples should be changed given spark-submit
Date Tue, 22 Apr 2014 07:59:14 GMT
Patrick Wendell created SPARK-1565:

             Summary: Spark examples should be changed given spark-submit
                 Key: SPARK-1565
             Project: Spark
          Issue Type: Improvement
          Components: Examples
            Reporter: Patrick Wendell
            Assignee: Prashant Sharma
             Fix For: 1.0.0

Now that we have the spark-submit script we should make some changes to the examples:

1. We should always create a SparkConf that sets the app name and use the SparkContext constructor
that accepts a conf.
2. We shouldn't set the master based on a command line argument, instead users can do this
on their own with spark-submit.
3. The examples projects should mark spark-core/streaming as a "provided" dependency, so that
the examples assembly jar does not include spark.

Then users can launch examples like this:

./bin/spark-submit examples/target/scala-2.10/spark-examples-assembly-1.0.0-SNAPSHOT.jar \
  --class org.apache.spark.examples.SparkPi \
  --arg 1000

This message was sent by Atlassian JIRA

View raw message