hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-8091) Add a topology mapper that reads hostname to rack mappings from a Java properties file
Date Sat, 18 Feb 2012 13:44:59 GMT
Add a topology mapper that reads hostname to rack mappings from a Java properties file

                 Key: HADOOP-8091
                 URL: https://issues.apache.org/jira/browse/HADOOP-8091
             Project: Hadoop Common
          Issue Type: New Feature
          Components: util
    Affects Versions: 0.24.0
            Reporter: Steve Loughran
            Priority: Minor

Getting mapping scripts right is surprisingly hard -and if its wrong in production bad things
happen. It would be good to have something simpler for beginners -and one that is trivial
to generate by a machine based on infrastructure data.

I propose adding an alternative mapper, one driven by a java property file

 # the specific topology mapper must be identified for loading
 # it uses another key to identify the property file to load. This is checked for on startup
-if missing, fail.
 # one property, perhaps "default-rack" identifies the default rack mapping for any host not
in the list
 # every other entry lists a hostname to rack mapping
 # hostname mapping is done on the first entry in the FQDN, to be less brittle to domain resolution.


 * add a new mapper that builds a concurrent hash map
 * read in every entry in the specified property file, add it to the map
 * when queried, extract the hostname (i.e. everything before any ".")
 * match that in the hash table, return if found
 * if not found: return the default rack

Feature creep would be to poll this file for changes at a (specified) frequency, and pick
up the changes when they occur. This would require removing the caching topology mapper that
wraps all others in the NN and RM.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message