whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Schilling <ch...@thecleversense.com>
Subject Re: authentication trouble
Date Tue, 11 Oct 2011 22:52:41 GMT
Customized the java install script?  I downloaded the latest and just
unpacked and tried to run:
curl -O
http://www.apache.org/dist/incubator/whirr/whirr-0.6.0-incubating/whirr-0.6.0-incubating.tar.gz

I can wait until morning.  Thanks for taking a look...
Chris


On Tue, Oct 11, 2011 at 3:47 PM, Andrei Savu <savu.andrei@gmail.com> wrote:

> I see that you've customised the java install script. Are you building on
> top of the 0.6.0 release or on top of trunk?
>
> I see nothing strange in the log file. I will be able to say more tomorrow
> as I try to replicate the same behaviour.
>
> @Adrian I know that you've tried to launch a cluster from within the Amazon
> cloud. Any feedback on this? Thanks!
>
>
> On Tue, Oct 11, 2011 at 11:36 PM, Chris Schilling <
> chris@thecleversense.com> wrote:
>
>> Okay, no the ssh keypair does not need a password.  I installed whirr on a
>> separate ec2 instance.  So, this may be internal communication issues
>> between ec2.  Here is the full whirr.log
>>
>> 2011-10-11 22:08:47,807 DEBUG [org.apache.whirr.service.ComputeCache]
>> (main) creating new ComputeServiceContext
>> org.apache.whirr.service.ComputeCache$Key@1a689880
>> 2011-10-11 22:09:50,094 DEBUG [org.apache.whirr.service.ComputeCache]
>> (main) creating new ComputeServiceContext
>> org.apache.whirr.service.ComputeCache$Key@1a689880
>> 2011-10-11 22:09:56,433 DEBUG [org.apache.whirr.service.ComputeCache]
>> (main) created new ComputeServiceContext  [id=aws-ec2, endpoint=
>> https://ec2.us-east-1.amazonaws.com, apiVersion=2010-06-15,
>> identity=1FTR7NCN01CEAR6FK2G2, iso3166Codes=[US-VA, US-CA, IE, SG, JP-13]]
>> 2011-10-11 22:09:56,454 INFO
>> [org.apache.whirr.actions.BootstrapClusterAction] (main) Bootstrapping
>> cluster
>> 2011-10-11 22:09:56,455 INFO
>> [org.apache.whirr.actions.BootstrapClusterAction] (main) Configuring
>> template
>> 2011-10-11 22:09:56,473 DEBUG
>> [org.apache.whirr.actions.BootstrapClusterAction] (main) Running script:
>> #!/bin/bash
>> set +u
>> shopt -s xpg_echo
>> shopt -s expand_aliases
>> unset PATH JAVA_HOME LD_LIBRARY_PATH
>> function abort {
>>    echo "aborting: $@" 1>&2
>>    exit 1
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function configure_hostnames() {
>>   local OPTIND
>>   local OPTARG
>>
>>   CLOUD_PROVIDER=
>>   while getopts "c:" OPTION; do
>>     case $OPTION in
>>     c)
>>       CLOUD_PROVIDER="$OPTARG"
>>       shift $((OPTIND-1)); OPTIND=1
>>       ;;
>>     esac
>>   done
>>
>>   case $CLOUD_PROVIDER in
>>     cloudservers | cloudservers-uk | cloudservers-us )
>>       if which dpkg &> /dev/null; then
>>         PRIVATE_IP=`/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2
>> | awk '{ print $1}'`
>>         HOSTNAME=`echo $PRIVATE_IP | tr . -`.static.cloud-ips.com
>>         echo $HOSTNAME > /etc/hostname
>>         sed -i -e "s/$PRIVATE_IP.*/$PRIVATE_IP $HOSTNAME/" /etc/hosts
>>         set +e
>>         /etc/init.d/hostname restart
>>         set -e
>>         sleep 2
>>         hostname
>>       fi
>>       ;;
>>   esac
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function install_java_deb() {
>>   # Enable multiverse
>>   # TODO: check that it is not already enabled
>>   sed -i -e 's/universe$/universe multiverse/' /etc/apt/sources.list
>>
>>   DISTRO=`lsb_release -s -c`
>>   cat > /etc/apt/sources.list.d/canonical.com.list <<EOF
>> deb http://archive.canonical.com/ubuntu $DISTRO partner
>> deb-src http://archive.canonical.com/ubuntu $DISTRO partner
>> EOF
>>
>>   apt-get update
>>
>>   echo 'sun-java6-bin   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jdk   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jre   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jre   sun-java6-jre/stopthread        boolean true
>> sun-java6-jre   sun-java6-jre/jcepolicy note
>> sun-java6-bin   shared/present-sun-dlj-v1-1     note
>> sun-java6-jdk   shared/present-sun-dlj-v1-1     note
>> sun-java6-jre   shared/present-sun-dlj-v1-1     note
>> ' | debconf-set-selections
>>
>>   apt-get -y install sun-java6-jdk
>>
>>   echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile
>>   export JAVA_HOME=/usr/lib/jvm/java-6-sun
>>   java -version
>>
>> }
>>
>> function install_java_rpm() {
>>   MACHINE_TYPE=`uname -m`
>>   if [ ${MACHINE_TYPE} == 'x86_64' ]; then
>>     JDK_PACKAGE=jdk-6u21-linux-x64-rpm.bin
>>   else
>>     JDK_PACKAGE=jdk-6u21-linux-i586-rpm.bin
>>   fi
>>   JDK_INSTALL_PATH=/usr/java
>>   mkdir -p $JDK_INSTALL_PATH
>>   cd $JDK_INSTALL_PATH
>>   wget http://whirr-third-party.s3.amazonaws.com/$JDK_PACKAGE
>>   chmod +x $JDK_PACKAGE
>>   mv /bin/more /bin/more.no
>>   yes | ./$JDK_PACKAGE -noregister
>>   mv /bin/more.no /bin/more
>>   rm -f *.rpm $JDK_PACKAGE
>>
>>   export JAVA_HOME=$(ls -d $JDK_INSTALL_PATH/jdk*)
>>   echo "export JAVA_HOME=$JAVA_HOME" >> /etc/profile
>>   alternatives --install /usr/bin/java java $JAVA_HOME/bin/java 17000
>>   alternatives --set java $JAVA_HOME/bin/java
>>   java -version
>> }
>>
>> function install_java() {
>>   if which dpkg &> /dev/null; then
>>     install_java_deb
>>   elif which rpm &> /dev/null; then
>>     install_java_rpm
>>   fi
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function install_tarball() {
>>   if [[ "$1" != "" ]]; then
>>     # Download a .tar.gz file and extract to target dir
>>
>>     local tar_url=$1
>>     local tar_file=`basename $tar_url`
>>     local tar_file_md5=`basename $tar_url.md5`
>>
>>     local target=${2:-/usr/local/}
>>     mkdir -p $target
>>
>>     local curl="curl -L --silent --show-error --fail --connect-timeout 10
>> --max-time 600 --retry 5"
>>     # any download should take less than 10 minutes
>>
>>     for retry_count in `seq 1 3`;
>>     do
>>       $curl -O $tar_url || true
>>       $curl -O $tar_url.md5 || true
>>
>>       if [ ! -e $tar_file_md5 ]; then
>>         echo "Could not download  $tar_url.md5. Continuing."
>>         break;
>>       elif md5sum -c $tar_file_md5; then
>>         break;
>>       else
>>         # workaround for cassandra broken .md5 files
>>         if [ `md5sum $tar_file | awk '{print $1}'` = `cat $tar_file_md5`
>> ]; then
>>           break;
>>         fi
>>
>>         rm -f $tar_file $tar_file_md5
>>       fi
>>
>>       if [ ! $retry_count -eq "3" ]; then
>>         sleep 10
>>       fi
>>     done
>>
>>     if [ ! -e $tar_file ]; then
>>       echo "Failed to download $tar_file. Aborting."
>>       exit 1
>>     fi
>>
>>     tar xzf $tar_file -C $target
>>     rm -f $tar_file $tar_file_md5
>>   fi
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function update_repo() {
>>   if which dpkg &> /dev/null; then
>>     sudo apt-get update
>>   elif which rpm &> /dev/null; then
>>     yum update -y yum
>>   fi
>> }
>>
>> function install_hadoop() {
>>   local OPTIND
>>   local OPTARG
>>
>>   CLOUD_PROVIDER=
>>   HADOOP_TAR_URL=
>>   while getopts "c:u:" OPTION; do
>>     case $OPTION in
>>     c)
>>       CLOUD_PROVIDER="$OPTARG"
>>       ;;
>>     u)
>>       HADOOP_TAR_URL="$OPTARG"
>>       ;;
>>     esac
>>   done
>>
>>   HADOOP_HOME=/usr/local/$(basename $HADOOP_TAR_URL .tar.gz)
>>
>>   update_repo
>>
>>   if ! id hadoop &> /dev/null; then
>>     useradd hadoop
>>   fi
>>
>>   install_tarball $HADOOP_TAR_URL
>>   ln -s $HADOOP_HOME /usr/local/hadoop
>>
>>   echo "export HADOOP_HOME=$HADOOP_HOME" >> ~root/.bashrc
>>   echo 'export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH' >>
>> ~root/.bashrc
>> }
>>
>> export PATH=/usr/ucb/bin:/bin:/sbin:/usr/bin:/usr/sbin
>> configure_hostnames -c aws-ec2 || exit 1
>> install_java || exit 1
>> install_tarball || exit 1
>> install_hadoop -c aws-ec2 -u
>> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz|| exit 1
>> exit 0
>>
>> 2011-10-11 22:09:56,490 DEBUG [jclouds.compute] (main) >> searching
>> params([biggest=false, fastest=false, imageName=null, imageDescription=null,
>> imageId=us-east-1/ami-da0cf8b3, imagePredicate=null, imageVersion=null,
>> location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2,
>> iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=null,
>> osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null,
>> hardwareId=m1.large])
>> 2011-10-11 22:09:56,491 DEBUG [jclouds.compute] (user thread 0) >>
>> providing images
>> 2011-10-11 22:09:56,497 DEBUG [jclouds.compute] (user thread 1) >>
>> providing images
>> 2011-10-11 22:09:58,046 DEBUG [jclouds.compute] (user thread 1) <<
>> images(32)
>> 2011-10-11 22:10:01,457 DEBUG [jclouds.compute] (user thread 0) <<
>> images(3123)
>> 2011-10-11 22:10:02,183 DEBUG [jclouds.compute] (main) <<   matched
>> hardware(m1.large)
>> 2011-10-11 22:10:02,184 DEBUG [jclouds.compute] (main) <<   matched
>> image(us-east-1/ami-da0cf8b3)
>> 2011-10-11 22:10:02,194 INFO
>> [org.apache.whirr.actions.BootstrapClusterAction] (main) Configuring
>> template
>> 2011-10-11 22:10:02,196 INFO  [org.apache.whirr.actions.NodeStarter]
>> (pool-3-thread-2) Starting 1 node(s) with roles [hadoop-datanode,
>> hadoop-tasktracker]
>> 2011-10-11 22:10:02,196 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> running 1 node group(hadoop) location(us-east-1)
>> image(us-east-1/ami-da0cf8b3) hardwareProfile(m1.large)
>> options([groupIds=[], keyPair=null, noKeyPair=false,
>> monitoringEnabled=false, placementGroup=null, noPlacementGroup=false,
>> subnetId=null, userData=null, blockDeviceMappings=[], spotPrice=null,
>> spotOptions=[formParameters={}]])
>> 2011-10-11 22:10:02,199 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> searching params([biggest=false, fastest=false, imageName=null,
>> imageDescription=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml,
>> imageId=null, imagePredicate=null, imageVersion=20101020,
>> location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2,
>> iso3166Codes=[US-VA], metadata={}], minCores=2.0, minRam=7680,
>> osFamily=ubuntu, osName=null,
>> osDescription=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml,
>> osVersion=10.04, osArch=paravirtual, os64Bit=true, hardwareId=null])
>> 2011-10-11 22:10:02,199 DEBUG
>> [org.apache.whirr.actions.BootstrapClusterAction] (main) Running script:
>> #!/bin/bash
>> set +u
>> shopt -s xpg_echo
>> shopt -s expand_aliases
>> unset PATH JAVA_HOME LD_LIBRARY_PATH
>> function abort {
>>    echo "aborting: $@" 1>&2
>>    exit 1
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function configure_hostnames() {
>>   local OPTIND
>>   local OPTARG
>>
>>   CLOUD_PROVIDER=
>>   while getopts "c:" OPTION; do
>>     case $OPTION in
>>     c)
>>       CLOUD_PROVIDER="$OPTARG"
>>       shift $((OPTIND-1)); OPTIND=1
>>       ;;
>>     esac
>>   done
>>
>>   case $CLOUD_PROVIDER in
>>     cloudservers | cloudservers-uk | cloudservers-us )
>>       if which dpkg &> /dev/null; then
>>         PRIVATE_IP=`/sbin/ifconfig eth0 | grep 'inet addr:' | cut -d: -f2
>> | awk '{ print $1}'`
>>         HOSTNAME=`echo $PRIVATE_IP | tr . -`.static.cloud-ips.com
>>         echo $HOSTNAME > /etc/hostname
>>         sed -i -e "s/$PRIVATE_IP.*/$PRIVATE_IP $HOSTNAME/" /etc/hosts
>>         set +e
>>         /etc/init.d/hostname restart
>>         set -e
>>         sleep 2
>>         hostname
>>       fi
>>       ;;
>>   esac
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function install_java_deb() {
>>   # Enable multiverse
>>   # TODO: check that it is not already enabled
>>   sed -i -e 's/universe$/universe multiverse/' /etc/apt/sources.list
>>
>>   DISTRO=`lsb_release -s -c`
>>   cat > /etc/apt/sources.list.d/canonical.com.list <<EOF
>> deb http://archive.canonical.com/ubuntu $DISTRO partner
>> deb-src http://archive.canonical.com/ubuntu $DISTRO partner
>> EOF
>>
>>   apt-get update
>>
>>   echo 'sun-java6-bin   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jdk   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jre   shared/accepted-sun-dlj-v1-1    boolean true
>> sun-java6-jre   sun-java6-jre/stopthread        boolean true
>> sun-java6-jre   sun-java6-jre/jcepolicy note
>> sun-java6-bin   shared/present-sun-dlj-v1-1     note
>> sun-java6-jdk   shared/present-sun-dlj-v1-1     note
>> sun-java6-jre   shared/present-sun-dlj-v1-1     note
>> ' | debconf-set-selections
>>
>>   apt-get -y install sun-java6-jdk
>>
>>   echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun" >> /etc/profile
>>   export JAVA_HOME=/usr/lib/jvm/java-6-sun
>>   java -version
>>
>> }
>>
>> function install_java_rpm() {
>>   MACHINE_TYPE=`uname -m`
>>   if [ ${MACHINE_TYPE} == 'x86_64' ]; then
>>     JDK_PACKAGE=jdk-6u21-linux-x64-rpm.bin
>>   else
>>     JDK_PACKAGE=jdk-6u21-linux-i586-rpm.bin
>>   fi
>>   JDK_INSTALL_PATH=/usr/java
>>   mkdir -p $JDK_INSTALL_PATH
>>   cd $JDK_INSTALL_PATH
>>   wget http://whirr-third-party.s3.amazonaws.com/$JDK_PACKAGE
>>   chmod +x $JDK_PACKAGE
>>   mv /bin/more /bin/more.no
>>   yes | ./$JDK_PACKAGE -noregister
>>   mv /bin/more.no /bin/more
>>   rm -f *.rpm $JDK_PACKAGE
>>
>>   export JAVA_HOME=$(ls -d $JDK_INSTALL_PATH/jdk*)
>>   echo "export JAVA_HOME=$JAVA_HOME" >> /etc/profile
>>   alternatives --install /usr/bin/java java $JAVA_HOME/bin/java 17000
>>   alternatives --set java $JAVA_HOME/bin/java
>>   java -version
>> }
>>
>> function install_java() {
>>   if which dpkg &> /dev/null; then
>>     install_java_deb
>>   elif which rpm &> /dev/null; then
>>     install_java_rpm
>>   fi
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function install_tarball() {
>>   if [[ "$1" != "" ]]; then
>>     # Download a .tar.gz file and extract to target dir
>>
>>     local tar_url=$1
>>     local tar_file=`basename $tar_url`
>>     local tar_file_md5=`basename $tar_url.md5`
>>
>>     local target=${2:-/usr/local/}
>>     mkdir -p $target
>>
>>     local curl="curl -L --silent --show-error --fail --connect-timeout 10
>> --max-time 600 --retry 5"
>>     # any download should take less than 10 minutes
>>
>>     for retry_count in `seq 1 3`;
>>     do
>>       $curl -O $tar_url || true
>>       $curl -O $tar_url.md5 || true
>>
>>       if [ ! -e $tar_file_md5 ]; then
>>         echo "Could not download  $tar_url.md5. Continuing."
>>         break;
>>       elif md5sum -c $tar_file_md5; then
>>         break;
>>       else
>>         # workaround for cassandra broken .md5 files
>>         if [ `md5sum $tar_file | awk '{print $1}'` = `cat $tar_file_md5`
>> ]; then
>>           break;
>>         fi
>>
>>         rm -f $tar_file $tar_file_md5
>>       fi
>>
>>       if [ ! $retry_count -eq "3" ]; then
>>         sleep 10
>>       fi
>>     done
>>
>>     if [ ! -e $tar_file ]; then
>>       echo "Failed to download $tar_file. Aborting."
>>       exit 1
>>     fi
>>
>>     tar xzf $tar_file -C $target
>>     rm -f $tar_file $tar_file_md5
>>   fi
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function update_repo() {
>>   if which dpkg &> /dev/null; then
>>     sudo apt-get update
>>   elif which rpm &> /dev/null; then
>>     yum update -y yum
>>   fi
>> }
>>
>> function install_hadoop() {
>>   local OPTIND
>>   local OPTARG
>>
>>   CLOUD_PROVIDER=
>>   HADOOP_TAR_URL=
>>   while getopts "c:u:" OPTION; do
>>     case $OPTION in
>>     c)
>>       CLOUD_PROVIDER="$OPTARG"
>>       ;;
>>     u)
>>       HADOOP_TAR_URL="$OPTARG"
>>       ;;
>>     esac
>>   done
>>
>>   HADOOP_HOME=/usr/local/$(basename $HADOOP_TAR_URL .tar.gz)
>>
>>   update_repo
>>
>>   if ! id hadoop &> /dev/null; then
>>     useradd hadoop
>>   fi
>>
>>   install_tarball $HADOOP_TAR_URL
>>   ln -s $HADOOP_HOME /usr/local/hadoop
>>
>>   echo "export HADOOP_HOME=$HADOOP_HOME" >> ~root/.bashrc
>>   echo 'export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH' >>
>> ~root/.bashrc
>> }
>>
>> export PATH=/usr/ucb/bin:/bin:/sbin:/usr/bin:/usr/sbin
>> configure_hostnames -c aws-ec2 || exit 1
>> install_java || exit 1
>> install_tarball || exit 1
>> install_hadoop -c aws-ec2 -u
>> http://archive.apache.org/dist/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz|| exit 1
>> exit 0
>>
>> 2011-10-11 22:10:02,200 DEBUG [jclouds.compute] (main) >> searching
>> params([biggest=false, fastest=false, imageName=null, imageDescription=null,
>> imageId=us-east-1/ami-da0cf8b3, imagePredicate=null, imageVersion=null,
>> location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2,
>> iso3166Codes=[US-VA], metadata={}], minCores=0.0, minRam=0, osFamily=null,
>> osName=null, osDescription=null, osVersion=null, osArch=null, os64Bit=null,
>> hardwareId=m1.large])
>> 2011-10-11 22:10:02,203 DEBUG [jclouds.compute] (main) <<   matched
>> hardware(m1.large)
>> 2011-10-11 22:10:02,203 DEBUG [jclouds.compute] (main) <<   matched
>> image(us-east-1/ami-da0cf8b3)
>> 2011-10-11 22:10:02,204 INFO  [org.apache.whirr.actions.NodeStarter]
>> (pool-3-thread-4) Starting 1 node(s) with roles [hadoop-namenode,
>> hadoop-jobtracker]
>> 2011-10-11 22:10:02,205 DEBUG [jclouds.compute] (pool-3-thread-4) >>
>> running 1 node group(hadoop) location(us-east-1)
>> image(us-east-1/ami-da0cf8b3) hardwareProfile(m1.large)
>> options([groupIds=[], keyPair=null, noKeyPair=false,
>> monitoringEnabled=false, placementGroup=null, noPlacementGroup=false,
>> subnetId=null, userData=null, blockDeviceMappings=[], spotPrice=null,
>> spotOptions=[formParameters={}]])
>> 2011-10-11 22:10:02,205 DEBUG [jclouds.compute] (pool-3-thread-4) >>
>> searching params([biggest=false, fastest=false, imageName=null,
>> imageDescription=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml,
>> imageId=null, imagePredicate=null, imageVersion=20101020,
>> location=[id=us-east-1, scope=REGION, description=us-east-1, parent=aws-ec2,
>> iso3166Codes=[US-VA], metadata={}], minCores=2.0, minRam=7680,
>> osFamily=ubuntu, osName=null,
>> osDescription=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml,
>> osVersion=10.04, osArch=paravirtual, os64Bit=true, hardwareId=null])
>> 2011-10-11 22:10:02,321 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> matched hardware(m1.large)
>> 2011-10-11 22:10:02,325 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> matched image(us-east-1/ami-da0cf8b3)
>> 2011-10-11 22:10:02,327 DEBUG [jclouds.compute] (pool-3-thread-4) >>
>> creating keyPair region(us-east-1) group(hadoop)
>> 2011-10-11 22:10:02,342 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> matched hardware(m1.large)
>> 2011-10-11 22:10:02,366 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> matched image(us-east-1/ami-da0cf8b3)
>> 2011-10-11 22:10:02,367 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> creating keyPair region(us-east-1) group(hadoop)
>> 2011-10-11 22:10:03,645 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> created keyPair(jclouds#hadoop#us-east-1#2)
>> 2011-10-11 22:10:03,646 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> creating securityGroup region(us-east-1) name(jclouds#hadoop#us-east-1)
>> 2011-10-11 22:10:03,776 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> reused securityGroup(jclouds#hadoop#us-east-1)
>> 2011-10-11 22:10:03,776 DEBUG [jclouds.compute] (pool-3-thread-2) >>
>> running 1 instance region(us-east-1) zone(null) ami(ami-da0cf8b3)
>> params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#hadoop#us-east-1],
>> KeyName=[jclouds#hadoop#us-east-1#2]})
>> 2011-10-11 22:10:04,765 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> created keyPair(jclouds#hadoop#us-east-1#0)
>> 2011-10-11 22:10:04,765 DEBUG [jclouds.compute] (pool-3-thread-4) >>
>> running 1 instance region(us-east-1) zone(null) ami(ami-da0cf8b3)
>> params({InstanceType=[m1.large], SecurityGroup.1=[jclouds#hadoop#us-east-1],
>> KeyName=[jclouds#hadoop#us-east-1#0]})
>> 2011-10-11 22:10:05,067 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> started instances([region=us-east-1, name=i-08153d68])
>> 2011-10-11 22:10:05,128 DEBUG [jclouds.compute] (pool-3-thread-2) <<
>> present instances([region=us-east-1, name=i-08153d68])
>> 2011-10-11 22:10:05,186 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> started instances([region=us-east-1, name=i-12153d72])
>> 2011-10-11 22:10:05,249 DEBUG [jclouds.compute] (pool-3-thread-4) <<
>> present instances([region=us-east-1, name=i-12153d72])
>> 2011-10-11 22:10:38,407 DEBUG [jclouds.compute] (user thread 0) >>
>> blocking on socket [address=184.72.177.130, port=22] for 600000 seconds
>> 2011-10-11 22:10:43,449 DEBUG [jclouds.compute] (user thread 0) << socket
>> [address=184.72.177.130, port=22] opened
>> 2011-10-11 22:10:44,681 DEBUG [jclouds.compute] (user thread 7) >>
>> blocking on socket [address=50.19.59.109, port=22] for 600000 seconds
>> 2011-10-11 22:10:46,462 DEBUG [jclouds.compute] (user thread 0) >> running
>> [sudo ./setup-ubuntu init] as ubuntu@184.72.177.130
>> 2011-10-11 22:10:46,534 DEBUG [jclouds.compute] (user thread 0) << init(0)
>> 2011-10-11 22:10:46,535 DEBUG [jclouds.compute] (user thread 0) >> running
>> [sudo ./setup-ubuntu start] as ubuntu@184.72.177.130
>> 2011-10-11 22:10:47,653 DEBUG [jclouds.compute] (user thread 0) <<
>> start(0)
>> 2011-10-11 22:10:56,729 DEBUG [jclouds.compute] (user thread 7) << socket
>> [address=50.19.59.109, port=22] opened
>> 2011-10-11 22:11:00,695 DEBUG [jclouds.compute] (user thread 7) >> running
>> [sudo ./setup-ubuntu init] as ubuntu@50.19.59.109
>> 2011-10-11 22:11:00,954 DEBUG [jclouds.compute] (user thread 7) << init(0)
>> 2011-10-11 22:11:00,954 DEBUG [jclouds.compute] (user thread 7) >> running
>> [sudo ./setup-ubuntu start] as ubuntu@50.19.59.109
>> 2011-10-11 22:11:02,078 DEBUG [jclouds.compute] (user thread 7) <<
>> start(0)
>> 2011-10-11 22:11:36,157 DEBUG [jclouds.compute] (user thread 0) <<
>> complete(true)
>> 2011-10-11 22:11:36,235 DEBUG [jclouds.compute] (user thread 0) << stdout
>> from setup-ubuntu as ubuntu@184.72.177.130
>> Hit http://security.ubuntu.com lucid-security/main Packages
>> Hit http://archive.canonical.com lucid/partner Packages
>> Hit http://security.ubuntu.com lucid-security/universe Packages
>> Hit http://security.ubuntu.com lucid-security/multiverse Packages
>> Hit http://security.ubuntu.com lucid-security/main Sources
>> Hit http://security.ubuntu.com lucid-security/universe Sources
>> Hit http://security.ubuntu.com lucid-security/multiverse Sources
>> Hit http://archive.canonical.com lucid/partner Sources
>> Reading package lists...
>> hadoop-0.20.2.tar.gz: OK
>>
>> 2011-10-11 22:11:36,291 DEBUG [jclouds.compute] (user thread 0) << stderr
>> from setup-ubuntu as ubuntu@184.72.177.130
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/native2ascii to
>> provide /usr/bin/native2ascii (native2ascii) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/rmic to provide
>> /usr/bin/rmic (rmic) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/schemagen to
>> provide /usr/bin/schemagen (schemagen) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/serialver to
>> provide /usr/bin/serialver (serialver) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsgen to provide
>> /usr/bin/wsgen (wsgen) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsimport to provide
>> /usr/bin/wsimport (wsimport) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/xjc to provide
>> /usr/bin/xjc (xjc) in auto mode.
>> java version "1.6.0_26"
>> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>>
>> 2011-10-11 22:11:36,293 DEBUG [jclouds.compute] (user thread 0) << options
>> applied node(us-east-1/i-08153d68)
>> 2011-10-11 22:11:36,296 INFO  [org.apache.whirr.actions.NodeStarter]
>> (pool-3-thread-2) Nodes started: [[id=us-east-1/i-08153d68,
>> providerId=i-08153d68, group=hadoop, name=null, location=[id=us-east-1b,
>> scope=ZONE, description=us-east-1b, parent=us-east-1, iso3166Codes=[US-VA],
>> metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
>> family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-16-E5-E8,
>> privateAddresses=[10.96.230.22], publicAddresses=[184.72.177.130],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]]]
>> 2011-10-11 22:11:47,206 DEBUG [jclouds.compute] (user thread 7) <<
>> complete(true)
>> 2011-10-11 22:11:47,282 DEBUG [jclouds.compute] (user thread 7) << stdout
>> from setup-ubuntu as ubuntu@50.19.59.109
>> Hit http://security.ubuntu.com lucid-security/main Packages
>> Hit http://archive.canonical.com lucid/partner Packages
>> Hit http://security.ubuntu.com lucid-security/universe Packages
>> Hit http://security.ubuntu.com lucid-security/multiverse Packages
>> Hit http://security.ubuntu.com lucid-security/main Sources
>> Hit http://security.ubuntu.com lucid-security/universe Sources
>> Hit http://security.ubuntu.com lucid-security/multiverse Sources
>> Hit http://archive.canonical.com lucid/partner Sources
>> Reading package lists...
>> hadoop-0.20.2.tar.gz: OK
>>
>> 2011-10-11 22:11:47,338 DEBUG [jclouds.compute] (user thread 7) << stderr
>> from setup-ubuntu as ubuntu@50.19.59.109
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/native2ascii to
>> provide /usr/bin/native2ascii (native2ascii) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/rmic to provide
>> /usr/bin/rmic (rmic) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/schemagen to
>> provide /usr/bin/schemagen (schemagen) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/serialver to
>> provide /usr/bin/serialver (serialver) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsgen to provide
>> /usr/bin/wsgen (wsgen) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/wsimport to provide
>> /usr/bin/wsimport (wsimport) in auto mode.
>> update-alternatives: using /usr/lib/jvm/java-6-sun/bin/xjc to provide
>> /usr/bin/xjc (xjc) in auto mode.
>> java version "1.6.0_26"
>> Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
>>
>> 2011-10-11 22:11:47,339 DEBUG [jclouds.compute] (user thread 7) << options
>> applied node(us-east-1/i-12153d72)
>> 2011-10-11 22:11:47,340 INFO  [org.apache.whirr.actions.NodeStarter]
>> (pool-3-thread-4) Nodes started: [[id=us-east-1/i-12153d72,
>> providerId=i-12153d72, group=hadoop, name=null, location=[id=us-east-1b,
>> scope=ZONE, description=us-east-1b, parent=us-east-1, iso3166Codes=[US-VA],
>> metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
>> family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-17-30-02,
>> privateAddresses=[10.97.51.240], publicAddresses=[50.19.59.109],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]]]
>> 2011-10-11 22:11:47,464 INFO  [org.apache.whirr.service.FirewallManager]
>> (main) Authorizing firewall ingress to [Instance{roles=[hadoop-namenode,
>> hadoop-jobtracker], publicIp=50.19.59.109, privateIp=10.97.51.240,
>> id=us-east-1/i-12153d72, nodeMetadata=[id=us-east-1/i-12153d72,
>> providerId=i-12153d72, group=hadoop, name=null, location=[id=us-east-1b,
>> scope=ZONE, description=us-east-1b, parent=us-east-1, iso3166Codes=[US-VA],
>> metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
>> family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-17-30-02,
>> privateAddresses=[10.97.51.240], publicAddresses=[50.19.59.109],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]]}] on ports [50070,
>> 50030] for [75.101.253.125/32]
>> 2011-10-11 22:11:47,529 WARN
>> [org.apache.whirr.service.jclouds.FirewallSettings] (main) The permission '
>> 75.101.253.125/32-1-50070-50070' has already been authorized on the
>> specified group
>> 2011-10-11 22:11:47,574 WARN
>> [org.apache.whirr.service.jclouds.FirewallSettings] (main) The permission '
>> 75.101.253.125/32-1-50030-50030' has already been authorized on the
>> specified group
>> 2011-10-11 22:11:47,575 INFO  [org.apache.whirr.service.FirewallManager]
>> (main) Authorizing firewall ingress to [Instance{roles=[hadoop-namenode,
>> hadoop-jobtracker], publicIp=50.19.59.109, privateIp=10.97.51.240,
>> id=us-east-1/i-12153d72, nodeMetadata=[id=us-east-1/i-12153d72,
>> providerId=i-12153d72, group=hadoop, name=null, location=[id=us-east-1b,
>> scope=ZONE, description=us-east-1b, parent=us-east-1, iso3166Codes=[US-VA],
>> metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
>> family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-17-30-02,
>> privateAddresses=[10.97.51.240], publicAddresses=[50.19.59.109],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]]}] on ports [8020,
>> 8021] for [50.19.59.109/32]
>> 2011-10-11 22:11:47,806 DEBUG [jclouds.compute] (main) >> listing node
>> details matching(withIds([us-east-1/i-08153d68, us-east-1/i-12153d72]))
>> 2011-10-11 22:11:50,315 DEBUG [jclouds.compute] (main) << list(2)
>> 2011-10-11 22:11:50,315 DEBUG
>> [org.apache.whirr.actions.ConfigureClusterAction] (main) Nodes in cluster:
>> [[id=us-east-1/i-08153d68, providerId=i-08153d68, group=hadoop, name=null,
>> location=[id=us-east-1b, scope=ZONE, description=us-east-1b,
>> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
>> imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04,
>> arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-16-E5-E8,
>> privateAddresses=[10.96.230.22], publicAddresses=[184.72.177.130],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]],
>> [id=us-east-1/i-12153d72, providerId=i-12153d72, group=hadoop, name=null,
>> location=[id=us-east-1b, scope=ZONE, description=us-east-1b,
>> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
>> imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04,
>> arch=paravirtual, is64Bit=true,
>> description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
>> state=RUNNING, loginPort=22, hostname=domU-12-31-39-17-30-02,
>> privateAddresses=[10.97.51.240], publicAddresses=[50.19.59.109],
>> hardware=[id=m1.large, providerId=m1.large, name=null,
>> processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null,
>> type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
>> [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false,
>> isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc,
>> durable=false, isBootDevice=false]],
>> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,is64Bit()),
>> tags=[]], loginUser=ubuntu, userMetadata={}, tags=[]]]
>> 2011-10-11 22:11:50,316 INFO
>> [org.apache.whirr.actions.ConfigureClusterAction] (main) Running
>> configuration script on nodes: [us-east-1/i-08153d68]
>> 2011-10-11 22:11:50,318 DEBUG
>> [org.apache.whirr.actions.ConfigureClusterAction] (main) script:
>> #!/bin/bash
>> set +u
>> shopt -s xpg_echo
>> shopt -s expand_aliases
>> unset PATH JAVA_HOME LD_LIBRARY_PATH
>> function abort {
>>    echo "aborting: $@" 1>&2
>>    exit 1
>> }
>> #
>> # Licensed to the Apache Software Foundation (ASF) under one or more
>> # contributor license agreements.  See the NOTICE file distributed with
>> # this work for additional information regarding copyright ownership.
>> # The ASF licenses this file to You under the Apache License, Version 2.0
>> # (the "License"); you may not use this file except in compliance with
>> # the License.  You may obtain a copy of the License at
>> #
>> #     http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>> #
>> function configure_hadoop() {
>>   local OPTIND
>>   local OPTARG
>>
>>   ROLES=$1
>>   shift
>>
>>   CLOUD_PROVIDER=
>>   while getopts "c:" OPTION; do
>>     case $OPTION in
>>     c)
>>       CLOUD_PROVIDER="$OPTARG"
>>       ;;
>>     esac
>>   done
>>
>>   case $CLOUD_PROVIDER in
>>     ec2 | aws-ec2 )
>>       # Alias /mnt as /data
>>       ln -s /mnt /data
>>       ;;
>>     *)
>>       ;;
>>   esac
>>
>>   HADOOP_HOME=/usr/local/hadoop
>>   HADOOP_CONF_DIR=$HADOOP_HOME/conf
>>
>>   mkdir -p /data/hadoop
>>   chown hadoop:hadoop /data/hadoop
>>   if [ ! -e /data/tmp ]; then
>>     mkdir /data/tmp
>>     chmod a+rwxt /data/tmp
>>   fi
>>   mkdir /etc/hadoop
>>   ln -s $HADOOP_CONF_DIR /etc/hadoop/conf
>>
>>   # Copy generated configuration files in place
>>   cp /tmp/{core,hdfs,mapred}-site.xml $HADOOP_CONF_DIR
>>
>>   # Keep PID files in a non-temporary directory
>>   sed -i -e "s|# export HADOOP_PID_DIR=.*|export
>> HADOOP_PID_DIR=/var/run/hadoop|" \
>>     $HADOOP_CONF_DIR/hadoop-env.sh
>>   mkdir -p /var/run/hadoop
>>   chown -R hadoop:hadoop /var/run/hadoop
>>
>>   # Set SSH options within the cluster
>>   sed -i -e 's|# export HADOOP_SSH_OPTS=.*|export HADOOP_SSH_OPTS="-o
>> StrictHostKeyChecking=no"|' \
>>     $HADOOP_CONF_DIR/hadoop-env.sh
>>
>>   # Disable IPv6
>>   sed -i -e 's|# export HADOOP_OPTS=.*|export
>> HADOOP_OPTS="-Djava.net.preferIPv4Stack=true"|' \
>>     $HADOOP_CONF_DIR/hadoop-env.sh
>>
>>   # Hadoop logs should be on the /data partition
>>   sed -i -e 's|# export HADOOP_LOG_DIR=.*|export
>> HADOOP_LOG_DIR=/var/log/hadoop/logs|' \
>>     $HADOOP_CONF_DIR/hadoop-env.sh
>>   rm -rf /var/log/hadoop
>>   mkdir /data/hadoop/logs
>>   chown hadoop:hadoop /data/hadoop/logs
>>   ln -s /data/hadoop/logs /var/log/hadoop
>>   chown -R hadoop:hadoop /var/log/hadoop
>>
>>   for role in $(echo "$ROLES" | tr "," "\n"); do
>>     case $role in
>>     hadoop-namenode)
>>       start_namenode
>>       ;;
>>     hadoop-secondarynamenode)
>>       start_hadoop_daemon secondarynamenode
>>       ;;
>>     hadoop-jobtracker)
>>       start_hadoop_daemon jobtracker
>>       ;;
>>     hadoop-datanode)
>>       start_hadoop_daemon datanode
>>       ;;
>>     hadoop-tasktracker)
>>       start_hadoop_daemon tasktracker
>>       ;;
>>     esac
>>   done
>>
>> }
>>
>> function start_namenode() {
>>   if which dpkg &> /dev/null; then
>>     AS_HADOOP="su -s /bin/bash - hadoop -c"
>>   elif which rpm &> /dev/null; then
>>     AS_HADOOP="/sbin/runuser -s /bin/bash - hadoop -c"
>>   fi
>>
>>   # Format HDFS
>>   [ ! -e /data/hadoop/hdfs ] && $AS_HADOOP "$HADOOP_HOME/bin/hadoop
>> namenode -format"
>>
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop-daemon.sh start namenode"
>>
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop dfsadmin -safemode wait"
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -mkdir /user"
>>   # The following is questionable, as it allows a user to delete another
>> user
>>   # It's needed to allow users to create their own user directories
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -chmod +w /user"
>>
>>   # Create temporary directory for Pig and Hive in HDFS
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -mkdir /tmp"
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -chmod +w /tmp"
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -mkdir /user/hive/warehouse"
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop fs -chmod +w /user/hive/warehouse"
>>
>> }
>>
>> function start_hadoop_daemon() {
>>   if which dpkg &> /dev/null; then
>>     AS_HADOOP="su -s /bin/bash - hadoop -c"
>>   elif which rpm &> /dev/null; then
>>     AS_HADOOP="/sbin/runuser -s /bin/bash - hadoop -c"
>>   fi
>>   $AS_HADOOP "$HADOOP_HOME/bin/hadoop-daemon.sh start $1"
>> }
>>
>> export PATH=/usr/ucb/bin:/bin:/sbin:/usr/bin:/usr/sbin
>> cat >> /tmp/core-site.xml <<'END_OF_FILE'
>> <configuration>
>>   <property>
>>     <name>hadoop.tmp.dir</name>
>>     <value>/data/tmp/hadoop-${user.name}</value>
>>   </property>
>>   <property>
>>     <name>io.file.buffer.size</name>
>>     <value>65536</value>
>>   </property>
>>   <property>
>>     <name>hadoop.rpc.socket.factory.class.default</name>
>>     <value>org.apache.hadoop.net.StandardSocketFactory</value>
>>     <final>true</final>
>>   </property>
>>   <property>
>>     <name>hadoop.rpc.socket.factory.class.ClientProtocol</name>
>>     <value></value>
>>   </property>
>>   <property>
>>     <name>hadoop.rpc.socket.factory.class.JobSubmissionProtocol</name>
>>     <value></value>
>>   </property>
>>   <property>
>>     <name>fs.trash.interval</name>
>>     <value>1440</value>
>>   </property>
>>   <property>
>>     <name>fs.default.name</name>
>>     <value>hdfs://ec2-50-19-59-109.compute-1.amazonaws.com:8020/</value>
>>   </property>
>> </configuration>
>> END_OF_FILE
>> cat >> /tmp/hdfs-site.xml <<'END_OF_FILE'
>> <configuration>
>>   <property>
>>     <name>dfs.block.size</name>
>>     <value>134217728</value>
>>   </property>
>>   <property>
>>     <name>dfs.data.dir</name>
>>     <value>/data/hadoop/hdfs/data</value>
>>   </property>
>>   <property>
>>     <name>dfs.datanode.du.reserved</name>
>>     <value>1073741824</value>
>>   </property>
>>   <property>
>>     <name>dfs.name.dir</name>
>>     <value>/data/hadoop/hdfs/name</value>
>>   </property>
>>   <property>
>>     <name>fs.checkpoint.dir</name>
>>     <value>/data/hadoop/hdfs/secondary</value>
>>   </property>
>> </configuration>
>> END_OF_FILE
>> cat >> /tmp/mapred-site.xml <<'END_OF_FILE'
>> <configuration>
>>   <property>
>>     <name>mapred.local.dir</name>
>>     <value>/data/hadoop/mapred/local</value>
>>   </property>
>>   <property>
>>     <name>mapred.map.tasks.speculative.execution</name>
>>     <value>true</value>
>>   </property>
>>   <property>
>>     <name>mapred.reduce.tasks.speculative.execution</name>
>>     <value>false</value>
>>   </property>
>>   <property>
>>     <name>mapred.system.dir</name>
>>     <value>/hadoop/system/mapred</value>
>>   </property>
>>   <property>
>>     <name>mapreduce.jobtracker.staging.root.dir</name>
>>     <value>/user</value>
>>   </property>
>>   <property>
>>     <name>mapred.compress.map.output</name>
>>     <value>true</value>
>>   </property>
>>   <property>
>>     <name>mapred.output.compression.type</name>
>>     <value>BLOCK</value>
>>   </property>
>>   <property>
>>     <name>mapred.child.java.opts</name>
>>     <value>-Xmx550m</value>
>>   </property>
>>   <property>
>>     <name>mapred.child.ulimit</name>
>>     <value>1126400</value>
>>   </property>
>>   <property>
>>     <name>mapred.tasktracker.map.tasks.maximum</name>
>>     <value>2</value>
>>   </property>
>>   <property>
>>     <name>mapred.tasktracker.reduce.tasks.maximum</name>
>>     <value>2</value>
>>   </property>
>>   <property>
>>     <name>mapred.reduce.tasks</name>
>>     <value>2</value>
>>   </property>
>>   <property>
>>     <name>mapred.job.tracker</name>
>>     <value>ec2-50-19-59-109.compute-1.amazonaws.com:8021</value>
>>   </property>
>> </configuration>
>> END_OF_FILE
>> configure_hadoop hadoop-datanode,hadoop-tasktracker -c aws-ec2 || exit 1
>> exit 0
>>
>> 2011-10-11 22:11:50,970 DEBUG [jclouds.compute] (user thread 7) >>
>> blocking on socket [address=184.72.177.130, port=22] for 600000 seconds
>> 2011-10-11 22:11:53,992 DEBUG [jclouds.compute] (user thread 7) << socket
>> [address=184.72.177.130, port=22] opened
>> 2011-10-11 22:12:57,316 DEBUG [org.apache.whirr.service.ComputeCache]
>> (Thread-1) closing ComputeServiceContext  [id=aws-ec2, endpoint=
>> https://ec2.us-east-1.amazonaws.com, apiVersion=2010-06-15,
>> identity=1FTR7NCN01CEAR6FK2G2, iso3166Codes=[US-VA, US-CA, IE, SG, JP-13]]
>>
>>
>> On Tue, Oct 11, 2011 at 3:31 PM, Andrei Savu <savu.andrei@gmail.com>wrote:
>>
>>> Chris -
>>>
>>> We've seen this issue in the past. I will take a closer look in the
>>> morning (in ~10 hours). Can you upload the full log somewhere? Also make the
>>> sure that the SSH keypair does not need a password.
>>>
>>> Cheers,
>>>
>>> -- Andrei Savu / andreisavu.ro
>>>
>>>
>>> On Tue, Oct 11, 2011 at 11:13 PM, Chris Schilling <
>>> chris@thecleversense.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> New to whirr, having trouble *running whirr from an ec2 instance*
>>>> (authentication when setting up other machines)
>>>>
>>>> First, here is my configuration:
>>>> whirr.cluster-name=hadoop
>>>> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1
>>>> hadoop-datanode+hadoop-tasktracker
>>>>
>>>> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment
>>>> variables.
>>>> whirr.provider=aws-ec2
>>>> whirr.identity=${env:AWS_ACCESS_KEY_ID}
>>>> whirr.credential=${env:AWS_SECRET_ACCESS_KEY}
>>>>
>>>> # The size of the instance to use. See
>>>> http://aws.amazon.com/ec2/instance-types/
>>>> whirr.hardware-id=m1.large
>>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>>> whirr.location-id=us-east-1
>>>> # By default use the user system SSH keys. Override them here.
>>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_whirr
>>>> whirr.public-key-file=${whirr.private-key-file}.pub
>>>>
>>>>
>>>>
>>>> I export the credentials, then create the key:
>>>>  ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr
>>>>
>>>> Then I launch the cluster:
>>>> whirr launch-cluster --config hadoop-ec2.properties --private-key-file
>>>> ~/.ssh/id_rsa_whirr
>>>>
>>>> The nodes start (costs me $!), but then authentication errors all over
>>>> the place, along with Preconditions failures.  Here are some samples of the
>>>>
>>>> java.lang.NullPointerException: architecture
>>>>         at
>>>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
>>>>         at org.jclouds.ec2.domain.Image.<init>(Image.java:81)
>>>>         at
>>>> org.jclouds.ec2.xml.DescribeImagesResponseHandler.endElement(DescribeImagesResponseHandler.java:169)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.endElement(AbstractSAXParser.java:604)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEndElement(XMLDocumentFragmentScannerImpl.java:1759)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2915)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:625)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:488)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:812)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:741)
>>>>         at
>>>> com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:123)
>>>> ......
>>>>
>>>> Then the authentication errors begin:
>>>> <<authenticated>> woke to: net.schmizz.sshj.userauth.UserAuthException:
>>>> publickey auth failed
>>>> << (ubuntu@184.72.177.130:22) error acquiring SSHClient(ubuntu@
>>>> 184.72.177.130:22): Exhausted available authentication methods
>>>> net.schmizz.sshj.userauth.UserAuthException: Exhausted available
>>>> authentication methods
>>>>         at
>>>> net.schmizz.sshj.userauth.UserAuthImpl.authenticate(UserAuthImpl.java:114)
>>>>         at net.schmizz.sshj.SSHClient.auth(SSHClient.java:204)
>>>>         at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:304)
>>>>         at net.schmizz.sshj.SSHClient.authPublickey(SSHClient.java:323)
>>>>         at
>>>> org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:183)
>>>>         at
>>>> org.jclouds.sshj.SshjSshClient$1.create(SshjSshClient.java:155)
>>>>         at
>>>> org.jclouds.sshj.SshjSshClient.acquire(SshjSshClient.java:204)
>>>>         at
>>>> org.jclouds.sshj.SshjSshClient.connect(SshjSshClient.java:229)
>>>>         at
>>>> org.jclouds.compute.callables.RunScriptOnNodeAsInitScriptUsingSsh.call(RunScriptOnNodeAsInitScriptUsingSsh.java:107)
>>>>         at
>>>> org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:69)
>>>>         at
>>>> org.jclouds.compute.strategy.RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(RunScriptOnNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:44)
>>>> ......
>>>>
>>>> Please advise!
>>>>
>>>>
>>>> Chris Schilling
>>>> Sr. Data Mining Engineer
>>>> Clever Sense, Inc.
>>>> "Curating the World Around You"
>>>> --------------------------------------------------------------
>>>> Winner of the 2011 Fortune Brainstorm Start-up Idol<http://tech.fortune.cnn.com/2011/07/20/startup-idol-brainstorm-clever-sense/>
>>>>
>>>> Wanna join the Clever Team? We're hiring!<http://www.thecleversense.com/jobs.html>
>>>> --------------------------------------------------------------
>>>>
>>>>
>>>
>>
>>
>> --
>> Chris Schilling
>> Sr. Data Fiend
>> Clever Sense, Inc.
>> "Curating the World Around You!"
>> --------------------------------------------------------------
>> Winner of the 2011 Fortune Brainstorm Start-up Idol<http://tech.fortune.cnn.com/2011/07/20/startup-idol-brainstorm-clever-sense/>
>>
>> Wanna join the Clever Team? We're hiring!<http://www.thecleversense.com/jobs.html>
>> --------------------------------------------------------------
>>
>>
>


-- 
Chris Schilling
Sr. Data Fiend
Clever Sense, Inc.
"Curating the World Around You!"
--------------------------------------------------------------
Winner of the 2011 Fortune Brainstorm Start-up
Idol<http://tech.fortune.cnn.com/2011/07/20/startup-idol-brainstorm-clever-sense/>

Wanna join the Clever Team? We're
hiring!<http://www.thecleversense.com/jobs.html>
--------------------------------------------------------------

Mime
View raw message