hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mithun Radhakrishnan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-17576) Improve progress-reporting in TezProcessor
Date Fri, 22 Sep 2017 00:09:00 GMT

     [ https://issues.apache.org/jira/browse/HIVE-17576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mithun Radhakrishnan updated HIVE-17576:
----------------------------------------
    Description: 
Another one on behalf of [~selinazh] and [~cdrome]. Following the example in [Apache Tez's
{{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
{{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a Tez task. As per
[~kshukla]'s advice,

{quote}
Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which will give the
correct progress value for a given Input. The TezProcessor(s) in Hive should use this to do
something similar to what MapProcessor in Tez does today, which is use/override ProgressHelper
to get the input progress and then set the progress on the processorContext.
...
The default behavior of the ProgressHelper class sets the processor progress to be the average
of progress values from all inputs.
{quote}

This code is -whacked from- *inspired by* {{MapProcessor}}'s use of {{ProgressHelper}}.

(For my reference, YHIVE-978.)

  was:
Another one on behalf of [~selinazh] and [~cdrome]. Following the example in [Apache Tez's
{{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
{{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a Tez task. As per
[~kshukla]'s advice,

{quote}
Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which will give the
correct progress value for a given Input. The TezProcessor(s) in Hive should use this to do
something similar to what MapProcessor in Tez does today, which is use/override ProgressHelper
to get the input progress and then set the progress on the processorContext.
...
The default behavior of the ProgressHelper class sets the processor progress to be the average
of progress values from all inputs.
{quote}

This code is -whacked from- *inspired by* {{MapProcessor}}'s use of {{ProgressHelper}}.


> Improve progress-reporting in TezProcessor
> ------------------------------------------
>
>                 Key: HIVE-17576
>                 URL: https://issues.apache.org/jira/browse/HIVE-17576
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Mithun Radhakrishnan
>            Assignee: Chris Drome
>
> Another one on behalf of [~selinazh] and [~cdrome]. Following the example in [Apache
Tez's {{MapProcessor}}|https://github.com/apache/tez/blob/247719d7314232f680f028f4e1a19370ffb7b1bb/tez-mapreduce/src/main/java/org/apache/tez/mapreduce/processor/map/MapProcessor.java#L88],
{{TezProcessor}} ought to use {{ProgressHelper}} to report progress for a Tez task. As per
[~kshukla]'s advice,
> {quote}
> Tez... provides {{getProgress()}} API for {{AbstractLogicalInput(s)}} which will give
the correct progress value for a given Input. The TezProcessor(s) in Hive should use this
to do something similar to what MapProcessor in Tez does today, which is use/override ProgressHelper
to get the input progress and then set the progress on the processorContext.
> ...
> The default behavior of the ProgressHelper class sets the processor progress to be the
average of progress values from all inputs.
> {quote}
> This code is -whacked from- *inspired by* {{MapProcessor}}'s use of {{ProgressHelper}}.
> (For my reference, YHIVE-978.)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message