hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sudhakara st <sudhakara...@gmail.com>
Subject Re: Duplicating map reduce tasks
Date Wed, 02 Jul 2014 14:17:48 GMT
I don't think we can achieve this by changing one class or two class, Look
into from JobSubmitter.java, TaskAttemptID,java ... to speculative
execution may it get helps you.  Running the code on different replicas of
the same block will produce will always produce same results,


On Wed, Jul 2, 2014 at 7:14 PM, Tina Samuel <tinasamuel89@gmail.com> wrote:

> Hi,
>
> Thanks for responding.Is it possible to achieve my requirement by modifying
> some code of map reduce,by which I will be able to replicate every map
> reduce task that is submitted?
>
> Thanks & Regards,
> Tina
>
>
> Regards,
> Tina
>
>
> On Wed, Jul 2, 2014 at 6:47 PM, Wellington Chevreuil <
> wellington.chevreuil@gmail.com> wrote:
>
> > Hi Tina,
> >
> > That's not controllable with hadoop mr. Hadoop sometimes will do this on
> > its own (if you have speculative execution enabled), just for performance
> > reasons case a given task is taking to much to complete under a given
> node,
> > and running the same code under replica of the same file will always
> > produce same results, then it launches the same task under different
> nodes,
> > and once the first is finished, the result is ready and the others are
> > killed.
> >
> > Cheers.
> >
> > On 2 Jul 2014, at 07:00, Tina Samuel <tinasamuel89@gmail.com> wrote:
> >
> > > Hi,
> > > I would like to provide the map reduce jobs in the following format :-
> > >
> > > <Map Reduce task> <number_of_replicas>
> > >
> > > I want to execute the specified task the specified number of
> > > times(number_of_replicas) on different nodes and then I want to compare
> > the
> > > results produced by these task copies. Is it possible to do this in
> > Hadoop
> > > Map reduce? If not possible, is there any means by which I can modify
> the
> > > code of map reduce so that I can do it?
> > >
> > > Thanks & Regards,
> > > Tina
> >
> >
>



-- 

Regards,
...sudhakara

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message