spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dick Davies <d...@hellooperator.net>
Subject Re: dockerized spark executor on mesos?
Date Thu, 04 Dec 2014 07:48:27 GMT
Oh great, thanks Tim - I'll give that a whirl then.

A Spark rollout isn't in our immediate future, I'm just looking for a good
framework for compute alongside our marathon deployment. So a good
time to experiment!

On 4 December 2014 at 02:36, Tim Chen <tim@mesosphere.io> wrote:
> Hi Dick,
>
> There is a PR on Spark to enable launching Spark executors with Docker
> containers, and pending a committer to review and merge
> (https://github.com/apache/spark/pull/3074)
>
> If you like to try it out please take the patch and try it!
>
> Tim
>
> On Wed, Dec 3, 2014 at 6:31 PM, Kyle Ellrott <kellrott@soe.ucsc.edu> wrote:
>>
>> I'd like to tag a question onto this; has anybody attempted to deploy
>> spark under Kubernetes
>> https://github.com/googlecloudplatform/kubernetes or Kubernetes mesos (
>> https://github.com/mesosphere/kubernetes-mesos ) .
>> On Wednesday, December 3, 2014, Matei Zaharia <matei.zaharia@gmail.com>
>> wrote:
>>>
>>> I'd suggest asking about this on the Mesos list (CCed). As far as I know,
>>> there was actually some ongoing work for this.
>>>
>>> Matei
>>>
>>> > On Dec 3, 2014, at 9:46 AM, Dick Davies <dick@hellooperator.net> wrote:
>>> >
>>> > Just wondered if anyone had managed to start spark
>>> > jobs on mesos wrapped in a docker container?
>>> >
>>> > At present (i.e. very early testing) I'm able to submit executors
>>> > to mesos via spark-submit easily enough, but they fall over
>>> > as we don't have a JVM on our slaves out of the box.
>>> >
>>> > I can push one out via our CM system if push comes to shove,
>>> > but it'd be nice to have that as part of the job (I'm thinking it might
>>> > be a way to get some of the dependencies deployed too).
>>> >
>>> > bear in mind I'm a total clueless newbie at this so please be gentle
>>> > if I'm doing this completely wrong.
>>> >
>>> > Thanks!
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> > For additional commands, e-mail: user-help@spark.apache.org
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message