hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: Speculative Execution...
Date Thu, 13 Sep 2012 14:38:50 GMT
Under YARN (branch-2, branch-0.23, and trunk) the speculative execution
decision is pluggable, and can be replaced by a user.  If you could come
up with a better solution to speculative execution that would be great.
We have known for a while that it is not very good (most of the time we
run a speculative task it is just wasted).  In branch-2 we have a new
version that we think is better (more conservative), but I am not sure how
much of a study has been done on exactly how much better it is or what
else can be done to make it even better.  I would look at adding your
ideas into that plugin and not so much of using the config to turn
speculation on or off dynamically, because there are some map/reduce
applications that abuse map/reduce somewhat and will not run correctly if
speculative execution is enabled.

--Bobby Evans 

On 9/13/12 1:08 AM, "Suresh S" <sureshhot@gmail.com> wrote:

>Hello Sir/Madam....
>   I am doing PhD. I am interested to do research in Hadoop for publishing
>I know little bit about speculative execution of slow tasks. I know it is
>possible to enable or disable speculative execution.
>   Is there any idea published already for dynamically enable or disable
>the speculative execution depending on application, cluster load and other
>run time parameters?
>   Is it worth to do research in this direction?
>   Is this contribution is worth to publish a conference or journal
>*Research Scholar,*
>*Department of Computer Applications,*
>*National Institute of Technology,*
>*Tiruchirappalli - 620015.*
>*Mobile: +91-9941506562*

View raw message