hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jay vyas (JIRA)" <j...@apache.org>
Subject [jira] [Created] (MAPREDUCE-5894) Make critical YARN properties first class citizens in the build.
Date Sun, 18 May 2014 01:47:15 GMT
jay vyas created MAPREDUCE-5894:

             Summary: Make critical YARN properties first class citizens in the build.
                 Key: MAPREDUCE-5894
                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5894
             Project: Hadoop Map/Reduce
          Issue Type: Improvement
            Reporter: jay vyas

We recently found that when deploy hadoop 2.2 with hadoop 2.0 values 
{noformat} mapreduce_shuffle {noformat} changed to {noformat}  mapreduce.shuffle {noformat}

There are likewise many similar examples of parameters which become deprecated over time.
  See http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

I suggest we:

1)  Use the *set of parameters which are deprecated* over time into java class which ships
directly with the code, maybe even as a static list inside of Configuration() itself, with
*optional extended parameters read from a configurable parameter *, so that ecosystem users
(i.e. like Hbase, or alternative file systems)  can add their own deprecation info.

2) have this list *checked on yarn daemon startup*.  so that unused parameters which are *obviously
artifacts are flagged immediately* by the daemon failing immediately.

3)Have a list of all mandatory *current* parameters stored in the code, and also, a list of
deprecated ones. Then, have the build * automatically fail * a parameter in the "madatory"
list is NOT accessed.  this would (a) make it so that unit testing of parameters does not
regress and (b) force all updates to the code which change a parameter name, to also include
update to deprecated parameter list, before build passes.

This message was sent by Atlassian JIRA

View raw message