spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From benh <...@git.apache.org>
Subject [GitHub] spark pull request: Add role and checkpoint support for Mesos back...
Date Tue, 04 Mar 2014 05:48:49 GMT
Github user benh commented on a diff in the pull request:

    https://github.com/apache/spark/pull/60#discussion_r10243692
  
    --- Diff: docs/running-on-mesos.md ---
    @@ -15,13 +15,15 @@ Spark can run on clusters managed by [Apache Mesos](http://mesos.apache.org/).
F
        * `export MESOS_NATIVE_LIBRARY=<path to libmesos.so>`. This path is usually
`<prefix>/lib/libmesos.so` (where the prefix is `/usr/local` by default, see above).
Also, on Mac OS X, the library is called `libmesos.dylib` instead of `libmesos.so`.
        * `export SPARK_EXECUTOR_URI=<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded
above>`.
        * `export MASTER=mesos://HOST:PORT` where HOST:PORT is the host and port (default:
5050) of your Mesos master (or `zk://...` if using Mesos with ZooKeeper).
    -8. To run a Spark application against the cluster, when you create your `SparkContext`,
pass the string `mesos://HOST:PORT` as the master URL. In addition, you'll need to set the
`spark.executor.uri` property. For example:
    +8. To run a Spark application against the cluster, when you create your `SparkContext`,
pass the string `mesos://HOST:PORT` as the master URL. In addition, you'll need to set the
`spark.executor.uri` property, and other properties for Mesos. For example:
     
     {% highlight scala %}
     val conf = new SparkConf()
       .setMaster("mesos://HOST:5050")
       .setAppName("My app")
       .set("spark.executor.uri", "<path to spark-{{site.SPARK_VERSION}}.tar.gz uploaded
above>")
    +  .set("spark.mesos.role", "spark")
    +  .set("spark.mesos.checkpoint", "true"))
    --- End diff --
    
    Recent versions of Mesos (and all versions going forward) have slaves with checkpointing
turned on by default, so  Spark should get allocated resources either way. That being said,
the only real reason to use this feature in Mesos is if you often have Spark jobs that run
a long time and you'd like to be able to restart a Mesos slave without killing the Spark job
that's running. Assuming Spark jobs finish often enough (or can deal with the occasional failed/restarted
slave) it might not be worth paying the extra overhead on the slave to checkpoint information
for Spark.
    
    Of course, I think wiring this up and letting the end user decide is great!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

Mime
View raw message