spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jimfcarroll <>
Subject Re: Problem with version compatibility
Date Thu, 25 Jun 2015 15:05:09 GMT
Ah. I've avoided using spark-submit primarily because our use of Spark is as
part of an analytics library that's meant to be embedded in other
applications with their own lifecycle management.

One of those application is a REST app running in tomcat which will make the
use of spark-submit difficult (if not impossible).

Also, we're trying to avoid sending jars over the wire per-job and so we
install our library (minus the spark dependencies) on the mesos workers and
refer to it in the spark configuration using spark.executor.extraClassPath
and if I'm reading SparkSubmit.scala correctly, it looks like the user's
assembly ends up sent to the cluster (at least in the case of yarn) though I
could be wrong on this.

Is there a standard way of running an app that's in control of it's own
runtime lifecycle without spark-submit?

Thanks again.

View this message in context:
Sent from the Apache Spark Developers List mailing list archive at

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message