whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Didonna <m.didonn...@gmail.com>
Subject Re: Setting Hadoop head size
Date Fri, 16 Dec 2011 15:58:36 GMT
On 16 December 2011 16:03, Hans Drexler <Hans.Drexler@humaninference.com> wrote:
> Thanks for your reply. We installed munin on the nodes and noticed that not all memory
was being used by hadoop. So we think we can make it faster by allocating more memory. But
maybe we have been changing the wrong paremeters.
> By the way, technically we did succeed in increasing the HADOOP_HEAPSIZE setting by changing
the whirr-cdh-0.6.0-incubating.jar file inside the jar. But I still have the feeling we do
it the "wrong" way.

Changing HADOOP_HEAPSIZE will not change how much memory the map and
reduce task will use (200mb by default). Again do use the suggested
parameters to modify the default memory allocation.

> PPS. We use this home-grown shell script to deploy munin on all hadoop nodes. It reads
a file cluster-nodes.txt that must contain ip-addres (or full host name) and password of each
node. Munin helps us keeping tabs on what goes on at the cluster nodes. Maybe somebody can
use it. Any remarks welcome (yes, I know sshpass is dirty!)
> #!/bin/sh
> while read line; do
>  set -- $line
>  host=$1
>  pass=$2
>  hostname=`echo $host | tr '.' "-"`
>  echo "Installing munin node on $host"
>  sshpass -p "$pass" ssh "root@$host" -o StrictHostKeyChecking=no /usr/bin/aptitude -y
install munin-node < /dev/null
>  sshpass -p "$pass" ssh "root@$host" -o StrictHostKeyChecking=no '/bin/echo "allow ^50\.57\.191\.88$"
>> /etc/munin/munin-node.conf' < /dev/null
>  sshpass -p "$pass" ssh "root@$host" -o StrictHostKeyChecking=no '/usr/sbin/service
munin-node restart' < /dev/null
>  echo "Adding $hostname to local config"
>  rm "/etc/munin/munin-conf.d/conf-$hostname.conf"
>  echo "[$hostname.localdomain]" > "/etc/munin/munin-conf.d/conf-$hostname.conf"
>  echo "\taddress $host" >> "/etc/munin/munin-conf.d/conf-$hostname.conf"
>  echo "\tuse_node_name yes" >> "/etc/munin/munin-conf.d/conf-$hostname.conf"
>  echo "Done installing on $host"
> done < cluster-nodes.txt
> Kind regards,
> Hans

View raw message