spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ruebenacker, Oliver A" <Oliver.Ruebenac...@altisource.com>
Subject RE: Programatically running of the Spark Jobs.
Date Thu, 04 Sep 2014 12:06:38 GMT

     Hello,

  Can this be used as a library from within another application?
  Thanks!

     Best, Oliver

From: Matt Chu [mailto:mchu@kabam.com]
Sent: Thursday, September 04, 2014 2:46 AM
To: Vicky Kak
Cc: user
Subject: Re: Programatically running of the Spark Jobs.

https://github.com/spark-jobserver/spark-jobserver

Ooyala's Spark jobserver is the current de facto standard, IIUC. I just added it to our prototype
stack, and will begin trying it out soon. Note that you can only do standalone or Mesos; YARN
isn't quite there yet.

(The repo just moved from https://github.com/ooyala/spark-jobserver, so don't trust Google
on this one (yet); development is happening in the first repo.)


On Wed, Sep 3, 2014 at 11:39 PM, Vicky Kak <vicky.kak@gmail.com<mailto:vicky.kak@gmail.com>>
wrote:
I have been able to submit the spark jobs using the submit script but I would like to do it
via code.
I am unable to search anything matching to my need.
I am thinking of using org.apache.spark.deploy.SparkSubmit to do so, may be have to write
some utility that passes the parameters required for this class.
I would be interested to know how community is doing.
Thanks,
Vicky

***********************************************************************************************************************

This email message and any attachments are intended solely for the use of the addressee. If
you are not the intended recipient, you are prohibited from reading, disclosing, reproducing,
distributing, disseminating or otherwise using this transmission. If you have received this
message in error, please promptly notify the sender by reply email and immediately delete
this message from your system. This message and any attachments may contain information that
is confidential, privileged or exempt from disclosure. Delivery of this message to any person
other than the intended recipient is not intended to waive any right or privilege. Message
transmission is not guaranteed to be secure or free of software viruses.
***********************************************************************************************************************
Mime
View raw message