hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudhakara st <sudhakara...@gmail.com>
Subject Re: configuring number of mappers and reducers
Date Sun, 07 Apr 2013 16:29:27 GMT
Hi Samaneh,

            The number of map tasks for a given job is driven by the number
of input splits in the input data. ideally in default configurations  each
input split(for a block) a map task is spawned. So your 2.5G of data
contains 44 blocks, therefore you jobs taking 44 map task. At minimum, with
FileInputFormat derivatives, job will have at least one map per file and
can have multiple maps per file if they extend beyond a single block(file
size is more that block size). The *mapred.map.tasks* parameter is just a
hint to the InputFormat for the number of maps. its does not have any
effect if the number blocks in the input date more then specified value. It
not possible to specify number mapper need run for a job. But it possible
to explicitly specify  number reduce can run for a job by using *
mapred.reduce.tasks* property.

The replication factor in not related in any to number of mapper and
reducer.







On Sun, Apr 7, 2013 at 7:38 PM, Samaneh Shokuhi
<samaneh.shokuhi@gmail.com>wrote:

> Hi All,
> I am doing some experiments by running WordCount example on hadoop.
> I have a cluster with 7 nodes .I want to run WordCount example with
> 3mappers and 3 reducers and compare the response time with another
> experiments when number of mappers and reducers increased to 6 and 12 and
> so on.
> For first experiment i set number of the mappers and reducer to 3 in
> wordCount example source code .and also set the number of replications to 3
> in hadoop configurations.Also  the maximum number of tasks per node is set
> to 1 .
> But when i run the sample with a big data like 2.5 G ,i can see 44 map
> tasks and 3 reduce tasks are running !!
>
> What parameters do i need to set to have like (3Mappers,3 Reducers),
> (6M,6R) and (12M,12R) and as i mentioned i have a cluster with 1 namenode
> and 6 datanodes.
> Is number of replications related to the number of mappers and reducers ?!
> Regards,
> Samaneh
>



-- 

Regards,
.....  Sudhakara.st

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message