hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-8091) Add a topology mapper that reads hostname to rack mappings from a Java properties file
Date Wed, 22 Feb 2012 15:45:49 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-8091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13213715#comment-13213715

Steve Loughran commented on HADOOP-8091:

HADOOP-7030 is almost superset; so yes. The nice thing about java properties is they have
a standard format for encoding non-ascii stuff and comments, and are fairly well known in
the java land (as well as coming with a built in Properties class), so are very easy to do.
> Add a topology mapper that reads hostname to rack mappings from a Java properties file
> --------------------------------------------------------------------------------------
>                 Key: HADOOP-8091
>                 URL: https://issues.apache.org/jira/browse/HADOOP-8091
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: util
>    Affects Versions: 0.24.0
>            Reporter: Steve Loughran
>            Priority: Minor
> Getting mapping scripts right is surprisingly hard -and if its wrong in production bad
things happen. It would be good to have something simpler for beginners -and one that is trivial
to generate by a machine based on infrastructure data.
> I propose adding an alternative mapper, one driven by a java property file
>  # the specific topology mapper must be identified for loading
>  # it uses another key to identify the property file to load. This is checked for on
startup -if missing, fail.
>  # one property, perhaps "default-rack" identifies the default rack mapping for any host
not in the list
>  # every other entry lists a hostname to rack mapping
>  # hostname mapping is done on the first entry in the FQDN, to be less brittle to domain
> Example
> {code}
> default-rack=/rack1
> host1=/rack1
> host2=/rack1
> host3=/rack2
> host4=/rack2
> {code}
> Implementation
>  * add a new mapper that builds a concurrent hash map
>  * read in every entry in the specified property file, add it to the map
>  * when queried, extract the hostname (i.e. everything before any ".")
>  * match that in the hash table, return if found
>  * if not found: return the default rack
> Feature creep would be to poll this file for changes at a (specified) frequency, and
pick up the changes when they occur. This would require removing the caching topology mapper
that wraps all others in the NN and RM.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message