hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinayakumar B <vinayakum...@apache.org>
Subject Re: Where is the map input transfered to the 'map worker'/container?
Date Mon, 07 Jul 2014 10:02:52 GMT
Hi Christian,

The input will be read using the InputFormat's record reader.

for example,

In case of  TexInputformat, *LineRecordReader#nextKeyValue(..)* will do the
job of reading one line from input file for every map call and storing the
offset as *key *and *text* as  *value, *and same will be passed to *map(..)*

You can go through *org.apache.hadoop.mapreduce.lib.input.LineRecordReader.java
*for more details.

Regards,
Vinay


On Mon, Jul 7, 2014 at 10:06 AM, Sandy Ryza <sandy.ryza@cloudera.com> wrote:

> Hi Christian,
>
> I'm not sure the exact code path, but HDFS, not MapReduce, is in charge of
> getting the bytes from the remote node.
>
> -Sandy
>
>
> On Mon, Jun 23, 2014 at 1:57 AM, Christian Grote <
> cgrote@mail.uni-paderborn.de> wrote:
>
> > Hey,
> >
> > I'm looking for the place where the actual map input is transfered to the
> > 'map worker'/container (in case it's assigned to a host that doesn't have
> > the data already).
> >
> > Something similar to the copyFromHost(..) method in Fetcher.java
> > (org.apache.hadoop.mapreduce.task.reduce), where the map output is
> > transfered.
> >
> >
> > Best Regards,
> > Christian Grote
> >
> >
> >
> >
> >
> >
> >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message