spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Davidson <>
Subject Re: sync master with slaves with bittorrent?
Date Mon, 19 May 2014 04:54:48 GMT
Out of curiosity, do you have a library in mind that would make it easy to
setup a bit torrent network and distribute files in an rsync (i.e., apply a
diff to a tree, ideally) fashion? I'm not familiar with this space, but we
do want to minimize the complexity of our standard ec2 launch scripts to
reduce the chance of something breaking.

On Sun, May 18, 2014 at 9:22 PM, Daniel Mahler <> wrote:

> I am launching a rather large cluster on ec2.
> It seems like the launch is taking forever on
> ....
> Setting up spark
> RSYNC'ing /root/spark to slaves...
> ...
> It seems that bittorrent might be a faster way to replicate
> the sizeable spark directory to the slaves
> particularly if there is a lot of not very powerful slaves.
> Just a thought ...
> cheers
> Daniel

View raw message