spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From JG Perrin <>
Subject RE: HDFS or NFS as a cache?
Date Mon, 02 Oct 2017 18:26:36 GMT

If I refer to the collect() API, it says “Running collect requires moving all the data into
the application's driver process, and doing so on a very large dataset can crash the driver
process with OutOfMemoryError.” So why would you need a distributed FS?


From: Steve Loughran []
Sent: Saturday, September 30, 2017 6:10 AM
To: JG Perrin <>
Cc: Alexander Czech <>;
Subject: Re: HDFS or NFS as a cache?

On 29 Sep 2017, at 20:03, JG Perrin <<>>

You will collect in the driver (often the master) and it will save the data, so for saving,
you will not have to set up HDFS.

no, it doesn't work quite like that.

1. workers generate their data and save somwhere
2. on "task commit" they move their data to some location where it will be visible for "job
commit" (rename, upload, whatever)
3. job commit —which is done in the driver,— takes all the committed task data and makes
it visible in the destination directory.
4. Then they create a _SUCCESS file to say "done!"

This is done with Spark talking between workers and drivers to guarantee that only one task
working on a specific part of the data commits their work, only
committing the job once all tasks have finished

The v1 mapreduce committer implements (2) by moving files under a job attempt dir, and (3)
by moving it from the job attempt dir to the destination. one rename per task commit, another
rename of every file on job commit. In HFDS, Azure wasb and other stores with an O(1) atomic
rename, this isn't *too* expensve, though that final job commit rename still takes time to
list and move lots of files

The v2 committer implements (2) by renaming to the destination directory and (3) as a no-op.
Rename in the tasks then, but not not that second, serialized one at the end

There's no copy of data from workers to driver, instead you need a shared output filesystem
so that the job committer can do its work alongside the tasks.

There are alternatives committer agorithms,

1. look at Ryan Blue's talk:
2. IBM Stocator paper ( and code (
3. Ongoing work in Hadoop itself for better committers. Goal: year end & Hadoop 3.1
. The oode is all there, Parquet is a troublespot, and more testing is welcome from anyone
who wants to help.
4. Databricks have "something"; specifics aren't covered, but I assume its dynamo DB based


From: Alexander Czech []
Sent: Friday, September 29, 2017 8:15 AM
Subject: HDFS or NFS as a cache?

I have a small EC2 cluster with 5 c3.2xlarge nodes and I want to write parquet files to S3.
But the S3 performance for various reasons is bad when I access s3 through the parquet write

Now I want to setup a small cache for the parquet output. One output is about 12-15 GB in
size. Would it be enough to setup a NFS-directory on the master, write the output to it and
then move it to S3? Or should I setup a HDFS on the Master? Or should I even opt for an additional
cluster running a HDFS solution on more than one node?

View raw message