whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Savu <savu.and...@gmail.com>
Subject Re: CDH behaves differently in the cloud?
Date Tue, 01 Nov 2011 12:31:37 GMT
Marco -

Can you check the remote config files for CDH Hadoop & versions?

Right now Whirr is using the latest release from Cloudera in this case
cdh3u2 release recently. It's possible that our setup scripts are affected
by this upgrade. We are tracking progress on this in:
https://issues.apache.org/jira/browse/WHIRR-415

Thanks,

-- Andrei Savu

On Sun, Oct 30, 2011 at 10:28 AM, Marco Didonna <m.didonna86@gmail.com>wrote:

> Hello everyone,
> I am pretty new to whirr and I would like to share with you my doubts.
> I am using on my laptop latest version of cdh available from
> repositories and I'd expect it to behave the same way on the cloud
> since I used the "cloudera recipe" to launch a cdh cluster in EC2,
> m1.large instance type. Here's the whirr config file
> http://pastebin.com/JXHYvMNb
> First of all seems like whirr installs a different version of cdh in
> the cloud according to what hadoop version says: I don't think this is
> a big deal but still...
> Secondly some options have absolutely no effect:
>
> - hadoop-mapreduce.mapred.child.java.opts=-Xmx1600m is basically
> ignored, even though it is present in mapred-site (on my laptop it
> works). According to ps aux the Xmx is 200 as default. I've also tried
> to manually edit mapred-site and add
> mapred.map.child.java.opts=-Xmx1000m and I got error saying there was
> no enough space (4GB were free).
>
> - hadoop-hdfs.dfs.replication=1 is ignored as well since I see
> replication factor 3 when I move my data from s3 to hdfs.
>
> - hadoop-hdfs.dfs.block.size=134217728 I actually don't know how to
> see if this has affected hdfs config.
>
> I hope someone can shed some light and maybe some configuration tweaks
> for m1.large he/she is using.
>
> Thank you
>
> Marco Didonna
>

Mime
View raw message