spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aureliano Buendia <buendia...@gmail.com>
Subject Re: Spark GCE Script
Date Wed, 07 May 2014 21:28:20 GMT
Please send a pull request, this should be maintained by the community,
just in case you do not feel like continuing to maintain it.

Also, nice to see that the gce version is shorter than the aws version.


On Tue, May 6, 2014 at 10:11 AM, Akhil Das <akhil@sigmoidanalytics.com>wrote:

> Hi Matei,
>
> Will clean up the code a little bit and send the pull request :)
>
> Thanks
> Best Regards
>
>
> On Tue, May 6, 2014 at 1:00 AM, François Le lay <fly@spotify.com> wrote:
>
>> Has anyone considered using jclouds tooling to support multiple cloud
>> providers? Maybe using Pallet?
>>
>> François
>>
>> On May 5, 2014, at 3:22 PM, Nicholas Chammas <nicholas.chammas@gmail.com>
>> wrote:
>>
>> I second this motion. :)
>>
>> A unified "cloud deployment" tool would be absolutely great.
>>
>>
>> On Mon, May 5, 2014 at 1:34 PM, Matei Zaharia <matei.zaharia@gmail.com>wrote:
>>
>>> Very cool! Have you thought about sending this as a pull request? We’d
>>> be happy to maintain it inside Spark, though it might be interesting to
>>> find a single Python package that can manage clusters across both EC2 and
>>> GCE.
>>>
>>> Matei
>>>
>>> On May 5, 2014, at 7:18 AM, Akhil Das <akhil@sigmoidanalytics.com>
>>> wrote:
>>>
>>> Hi Sparkers,
>>>
>>> We have created a quick spark_gce script which can launch a spark
>>> cluster in the Google Cloud. I'm sharing it because it might be helpful for
>>> someone using the Google Cloud for deployment rather than AWS.
>>>
>>> Here's the link to the script
>>>
>>> https://github.com/sigmoidanalytics/spark_gce
>>>
>>> Feel free to use it and suggest any feedback around it.
>>>
>>> In short here's what it does:
>>>
>>> Just like the spark_ec2 script, this one also reads certain command-line
>>> arguments (See the github page<https://github.com/sigmoidanalytics/spark_gce>
for
>>> more details) like the cluster name and all, then starts the machines in
>>> the google cloud, sets up the network, adds a 500GB empty disk to all
>>> machines, generate the ssh keys on master and transfer it to all slaves and
>>> install java and downloads and configures Spark/Shark/Hadoop. Also it
>>> starts the shark server automatically. Currently the version is 0.9.1 but
>>> I'm happy to add/support more versions if anyone is interested.
>>>
>>>
>>> Cheers.
>>>
>>>
>>> Thanks
>>> Best Regards
>>>
>>>
>>>
>>
>

Mime
View raw message