spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Russell Spitzer <>
Subject Re: Official Stance on Not Using Spark Submit
Date Mon, 10 Oct 2016 18:13:15 GMT
Just folks who don't want to use spark-submit, no real use-cases I've seen

I didn't know about SparkLauncher myself and I don't think there are any
official docs on that or launching spark as an embedded library for tests.

On Mon, Oct 10, 2016 at 11:09 AM Matei Zaharia <>

> What are the main use cases you've seen for this? Maybe we can add a page
> to the docs about how to launch Spark as an embedded library.
> Matei
> On Oct 10, 2016, at 10:21 AM, Russell Spitzer <>
> wrote:
> I actually had not seen SparkLauncher before, that looks pretty great :)
> On Mon, Oct 10, 2016 at 10:17 AM Russell Spitzer <
>> wrote:
> I'm definitely only talking about non-embedded uses here as I also use
> embedded Spark (cassandra, and kafka) to run tests. This is almost always
> safe since everything is in the same JVM. It's only once we get to
> launching against a real distributed env do we end up with issues.
> Since Pyspark uses spark submit in the java gateway i'm not sure if that
> matters :)
> The cases I see are usually usually going through main directly, adding
> jars programatically.
> Usually ends up with classpath errors (Spark not on the CP, their jar not
> on the CP, dependencies not on the cp),
> conf errors (executors have the incorrect environment, executor classpath
> broken, not understanding spark-defaults won't do anything),
> Jar version mismatches
> Etc ...
> On Mon, Oct 10, 2016 at 10:05 AM Sean Owen <> wrote:
> I have also 'embedded' a Spark driver without much trouble. It isn't that
> it can't work.
> The Launcher API is ptobably the recommended way to do that though.
> spark-submit is the way to go for non programmatic access.
> If you're not doing one of those things and it is not working, yeah I
> think people would tell you you're on your own. I think that's consistent
> with all the JIRA discussions I have seen over time.
> On Mon, Oct 10, 2016, 17:33 Russell Spitzer <>
> wrote:
> I've seen a variety of users attempting to work around using Spark Submit
> with at best middling levels of success. I think it would be helpful if the
> project had a clear statement that submitting an application without using
> Spark Submit is truly for experts only or is unsupported entirely.
> I know this is a pretty strong stance and other people have had different
> experiences than me so please let me know what you think :)

View raw message