hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dharmesh Kakadia <dhkaka...@gmail.com>
Subject Re: Update interval of default counters
Date Wed, 16 Apr 2014 06:42:58 GMT
Hi Akira,

Thanks for the reply, but as I understand this is the interval of console
counter printing. What I am trying to get

while(!job.isComplete()){
 getcounters() and do some processing on that.
}

Now this is running fine, but the status I get the same counter values
repeatedly and then suddenly a big change in counter values.
For example, getcounters for REDUCE_INPUT_RECORDS returns values like

0
0
..
0
280
280
...
280
516
516
...
516

etc.

I want to get more finer values, instead of directly jumping from 280 to
516.
Did that make sense? mapreduce.client.progressmonitor.pollinterval does not
seem to effect it. Any workaround ?

Thanks,
Dharmesh




On Tue, Apr 15, 2014 at 7:51 PM, Akira AJISAKA
<ajisakaa@oss.nttdata.co.jp>wrote:

> Moved to user@hadoop.apache.org.
>
> You can configure the interval by setting
> "mapreduce.client.progressmonitor.pollinterval" parameter.
> The default value is 1000 ms.
>
> For more details, please see http://hadoop.apache.org/docs/
> stable/hadoop-mapreduce-client/hadoop-mapreduce-
> client-core/mapred-default.xml.
>
> Regards,
> Akira
>
>
> (2014/04/15 15:29), Dharmesh Kakadia wrote:
>
>> Hi,
>>
>> What is the update interval of inbuilt framework counters? Is that
>> configurable?
>> I am trying to collect very fine grained information about the job
>> execution and using counters for that. It would be great if someone can
>> point me to documentation/code for it. Thanks in advance.
>>
>> Thanks,
>> Dharmesh
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message