spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Davies Liu <>
Subject Re: PySpark on Yarn a lot of python scripts project
Date Fri, 05 Sep 2014 20:04:45 GMT
Here is a store about how shared storage simplify all the things:

In Douban, we use Moose FS[1] instead of HDFS as the distributed file system,
it's POSIX compatible and can be mounted just as NFS.

We put all the data and tools and code in it, so we can access them easily on
all the machines, just like local disks. You can modify them in anywhere, and
get the modified one from anywhere.

One example, you will want to know the fields in your compressed log files:

$ bunzip2 -k path | head

Then you will need to modify your code to deal with these fields in logs:

$ vim

you have bunch of libraries or modules in the projects, but you will not need to
worry about them when run distributed jobs, you just need to do:

$ python

If something wrong, you could modify and save some RDDs into
disks, then check the results:

$ head path_to_result_of_rdd

maybe something wrong is your library, then fix them, and run again:

$ python

dump the result as CSV file, then load them into MySQL

mysql xxx < path_of_the_result

In a summary, a shared storage can help a lot in distributed environment,
some simple solution (such as NFS) is natural to solve these problem.
setup once, benefit forever.

PS: I'm also a contributor of Moose FS, has a fork at

PPS: I'm sorry for my pool English, if the above sounds rude to you,



On Fri, Sep 5, 2014 at 11:22 AM, Dimension Data, LLC.
<> wrote:
> I'd have to agree with Marcelo and Andrew here...
> Favoring a simple Build-and-Run/Submit wrapper-script that leverages '--py-files'
> over adding another layer of complexity -- even if seemingly 'trivial' like NFS -- is
> probably a good approach (... b/c more technology is never is 'trivial' over time). =:).
> Less is more.
> On 09/05/2014 01:58 PM, Marcelo Vanzin wrote:
> On Fri, Sep 5, 2014 at 10:50 AM, Davies Liu <> wrote:
> In daily development, it's common to modify your projects and re-run
> the jobs. If using zip or egg to package your code, you need to do
> this every time after modification, I think it will be boring.
> That's why shell scripts were invented. :-)
> Probably a lot easier than setting up and maintaining shared storage
> in a large cluster.
> --
> Sincerely yours,
> Team Dimension Data
> ________________________________
> Dimension Data, LLC. |
> P: 212.882.1276 |
> Data Analytics you can literally count on.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message