giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Avery Ching <>
Subject Re: hadoop version profiles
Date Wed, 21 Mar 2012 06:53:40 GMT
I agree with this approach, although munge is kinda hacky.  It is easy 
though. =)


On 3/20/12 5:52 PM, Eugene Koontz wrote:
> Hi Giraphers,
>     I think it might be good to look at how we can add support for new 
> hadoop versions. Currently we have hadoop_facebook 
> ( I am considering 
> adding new ones such as hadoop_0.24. Looking at the code, it seems 
> that the main hadoop variation between the stock hadoop used (0.203.0) 
> versus facebook has to do with the new security-related APIs in the 
> latter that is, fortunately, also available in hadoop 0.23 and 0.24. 
> So, hopefully we can make use of the existing work that Avery has done 
> for hadoop_facebook and apply it to other hadoop versions. Therefore I 
> would propose that:
> 1. a new munge flag HADOOP_SECURE to be used in 
> and a few other places, where we are currently checking for 
> 2. we make a new profile called hadoop_secure, which, as with 
> hadoop_facebook, will use the above munge flag.
> 3. we make a new profile "hadoop_0.20.203" for the existing default 
> hadoop and make it the default profile (activeByDefault=true). This 
> will makes it easier to handle the differences in the hadoop library 
> dependency set that have happened between 0.20.203 and hadoop trunk.
> Please see for 
> my branch that implements the above.
> Thanks,
> -Eugene

View raw message