hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6408) Add a /conf servlet to dump running configuration
Date Mon, 11 Jan 2010 05:37:56 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12798577#action_12798577

Todd Lipcon commented on HADOOP-6408:

Hi Hemanth,

Yes, I understood that was the original intention, but I don't see how the memory usage will
be very large when compared with the rest of the JobInProgress object:

In particular, this map consists mostly of flyweight references. The keys of the map are configuration
keys, which are references to String objects already stored by Configuration. The values are
Strings which are created once per resource that's loaded. So, the hashmap's memory footprint
doesn't double the footprint of the Configuration object by any means - it just has the footprint
of the references themselves.

A moderately full JobConf probably has a couple hundred configuration parameters, and a fairly
full JT has a couple hundred jobs. If each JobConf takes an additional 32 bytes per key (16
bytes for the references, and 16 bytes worth of hashmap overhead) then we should be talking
a few KB per JobConf, and 5-10MB overall on the JobTracker. To me that seems like a pretty
small cost, as any JT that's managing hundreds of jobs probably has many GB of RAM.

If necessary, I'm happy to write a quick test to measure the memory usage of Configuration
with and without the change.

> Add a /conf servlet to dump running configuration
> -------------------------------------------------
>                 Key: HADOOP-6408
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6408
>             Project: Hadoop Common
>          Issue Type: New Feature
>    Affects Versions: 0.22.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>         Attachments: hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt,
hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt, hadoop-6408.txt
> HADOOP-6184 added a command line flag to dump the running configuration. It would be
great for cluster troubleshooting to provide access to this as a servlet, preferably in both
JSON and XML formats. But really, any format would be better than nothing. This should/could
go into all of the daemons.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message