spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: More general submitJob API
Date Tue, 23 Dec 2014 00:04:05 GMT
Hi Alex,

SparkContext.submitJob() is marked as experimental -- most client programs
shouldn't be using it.  What are you looking to do?

For multiplexing jobs, one thing you can do is have multiple threads in
your client JVM each submit jobs on your SparkContext job.  This is
described here in the docs:
http://spark.apache.org/docs/latest/job-scheduling.html#scheduling-within-an-application

Andrew

On Mon, Dec 22, 2014 at 1:32 PM, Alessandro Baretta <alexbaretta@gmail.com>
wrote:

> Fellow Sparkers,
>
> I'm rather puzzled at the submitJob API. I can't quite figure out how it is
> supposed to be used. Is there any more documentation about it?
>
> Also, is there any simpler way to multiplex jobs on the cluster, such as
> starting multiple computations in as many threads in the driver and reaping
> all the results when they are available?
>
> Thanks,
>
> Alex
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message