tez-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmed Hussein (JIRA)" <j...@apache.org>
Subject [jira] [Created] (TEZ-4067) Tez Speculation decision is calculated on each update by the dispatcher
Date Wed, 08 May 2019 19:25:00 GMT
Ahmed Hussein created TEZ-4067:
----------------------------------

             Summary: Tez Speculation decision is calculated on each update by the dispatcher
                 Key: TEZ-4067
                 URL: https://issues.apache.org/jira/browse/TEZ-4067
             Project: Apache Tez
          Issue Type: Improvement
            Reporter: Ahmed Hussein


LegacySpeculator is an object field in VertexImpl. Therefore, all events are handled synchronously
by the caller (dispatcher). This implies the following:
 # the dispatcher spends long time executing updateStatus as it needs to check the runtime
estimation of the tezAttempts within the vertex.
 # the speculator is per stage: lunching a speculation may not the optimum decision. Ideally,
based on resources, speculated tasks should be the ones with slowest progress.
 # the time between speculation is skewed because there is a big delay for the dispatcher
to complete a full cycle. Also, speculation will be more aggressive compared to MR because
MR waits for "soonest.retry.after.speculate" whenever a task is speculated. On the other hand,
Tez speculates more tasks as it processes stages in parallel.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message