spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shao, Saisai" <saisai.s...@intel.com>
Subject RE: Shuffle to HDFS
Date Mon, 26 Jan 2015 05:44:16 GMT
Hi Larry,

I don’t think current Spark’s shuffle can support HDFS as a shuffle output. Anyway, is
there any specific reason to spill shuffle data to HDFS or NFS, this will severely increase
the shuffle time.

Thanks
Jerry

From: Larry Liu [mailto:larryliu05@gmail.com]
Sent: Sunday, January 25, 2015 4:45 PM
To: user@spark.incubator.apache.org
Subject: Shuffle to HDFS

How to change shuffle output to HDFS or NFS?
Mime
View raw message