spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ☼ R Nair (रविशंकर नायर) <>
Subject No Reducer scenarios
Date Mon, 30 Jan 2017 05:31:48 GMT
Dear all,

1) When we don't set the reducer class in driver program, IdentityReducer
is invoked.

2) When we set setNumReduceTasks(0), no reducer, even IdentityReducer is

Now, in the second scenario, we observed that the output is part-m-xx
format(instead of part-r-xx format) , which shows the map output. But we
know that the output of Map is always written to intermediate local file
system. So who/which class is responsible for taking these intermediate Map
outputs from local file system and writes to HDFS ? Does this particular
class performs this write operation only when setNumReduceTasks is set to

Best, Ravion

View raw message