spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: RE: Shuffle to HDFS
Date Mon, 26 Jan 2015 10:08:50 GMT
If there is no Reducer, there is no shuffle. The Mapper output goes to
HDFS, yes. But the question here is about shuffle files, right? Those
are written by the Mapper to local disk. Reducers load them from the
Mappers over the network then. Shuffle files do not go to HDFS.

On Mon, Jan 26, 2015 at 10:01 AM, bit1129@163.com <bit1129@163.com> wrote:
> I have also thought that Hadoop mapper output result is saved on HDFS, at
> least if the job only has Mapper but doesn't have Reducer.
> If there is reducer, then the map output will be saved on local disk?

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Mime
View raw message