hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sheetal Dolas (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5393) Physical memory and Virtual Memory counters in Job Counters display incorrect values
Date Tue, 16 Jul 2013 01:50:50 GMT
Sheetal Dolas created MAPREDUCE-5393:
----------------------------------------

             Summary: Physical memory and Virtual Memory counters in Job Counters display
incorrect values
                 Key: MAPREDUCE-5393
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5393
             Project: Hadoop Map/Reduce
          Issue Type: Bug
    Affects Versions: 1.2.0
         Environment: Linux 5.x
            Reporter: Sheetal Dolas


The counters Physical memory (bytes) snapshot - represented by PHYSICAL_MEMORY_BYTES ) and
Virtual memory (bytes) snapshot - represented by VIRTUAL_MEMORY_BYTES have multiple issues

1. It calculates it as MemTotal from /proc/meminfo which is host level metric and not task
level
2. Even if it is calculated as host level metric, the numbers shown in two different tasks
on same host are different and way different than the MemTotal displayed in /proc/meminfo
if you manually check it. E.g. on a host having 48G memory the tasks show the value of physical
memory (byte) snapshot as 214,142,976 and the number varies every time the job is run.

What was the intention of these counters? Why would we need to get the metric for every job
(as they should be more or less static for given host). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message