whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrei Savu <savu.and...@gmail.com>
Subject Re: Whirr: Cannot connect to Hadoop cluster on EC2 after lauch-cluster
Date Mon, 16 Apr 2012 17:05:02 GMT
Check /tmp/logs on the remote machine.

Also make sure you invalidate your keys because they are part of the log
output.
On Apr 16, 2012 7:42 PM, "Đỗ Hoàng Khiêm" <dohoangkhiem@gmail.com> wrote:

> Thanks Andrei, I've tried with Whirr 0.7.1, It now works properly when I
> browse the HDFS but the MapReduce still throws error
>
> khiem@master /etc/hadoop-0.20/conf.whirr $ hadoop jar
> $HADOOP_INSTALL/hadoop-examples-*.jar wordcount input output
> 12/04/16 23:33:30 WARN conf.Configuration: DEPRECATED: hadoop-site.xml
> found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use
> core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of
> core-default.xml, mapred-default.xml and hdfs-default.xml respectively
> 12/04/16 23:33:30 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 23:33:34 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 0 time(s).
> 12/04/16 23:33:35 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 1 time(s).
> 12/04/16 23:33:37 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 2 time(s).
> 12/04/16 23:33:38 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 3 time(s).
> 12/04/16 23:33:39 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 4 time(s).
> 12/04/16 23:33:41 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 5 time(s).
> 12/04/16 23:33:42 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 6 time(s).
> 12/04/16 23:33:43 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 7 time(s).
> 12/04/16 23:33:44 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 8 time(s).
> 12/04/16 23:33:46 INFO ipc.Client: Retrying connect to server:
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021. Already
> tried 9 time(s).
> 12/04/16 23:33:46 ERROR security.UserGroupInformation:
> PriviledgedActionException as:khiem (auth:SIMPLE)
> cause:java.io.IOException: Call to
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021 failed on
> local exception: java.net.SocketException: Malformed reply from SOCKS server
> java.io.IOException: Call to
> ec2-204-236-252-96.compute-1.amazonaws.com/204.236.252.96:8021 failed on
> local exception: java.net.SocketException: Malformed reply from SOCKS server
>     at org.apache.hadoop.ipc.Client.wrapException(Client.java:1142)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1110)
>     at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226)
>     at org.apache.hadoop.mapred.$Proxy1.getProtocolVersion(Unknown Source)
>     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:398)
>     at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:384)
>     at
> org.apache.hadoop.mapred.JobClient.createRPCProxy(JobClient.java:501)
>     at org.apache.hadoop.mapred.JobClient.init(JobClient.java:486)
>     at org.apache.hadoop.mapred.JobClient.<init>(JobClient.java:469)
>     at org.apache.hadoop.mapreduce.Job$1.run(Job.java:489)
>     at java.security.AccessController.doPrivileged(Native Method)
>     at javax.security.auth.Subject.doAs(Subject.java:396)
>     at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
>     at org.apache.hadoop.mapreduce.Job.connect(Job.java:487)
>     at org.apache.hadoop.mapreduce.Job.submit(Job.java:475)
>     at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
>     at org.apache.hadoop.examples.WordCount.main(WordCount.java:67)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>     at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>     at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>     at java.lang.reflect.Method.invoke(Method.java:597)
>     at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
> Caused by: java.net.SocketException: Malformed reply from SOCKS server
>     at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:90)
>     at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:472)
>     at java.net.Socket.connect(Socket.java:529)
>     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:406)
>     at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:425)
>     at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:532)
>     at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:210)
>     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1247)
>     at org.apache.hadoop.ipc.Client.call(Client.java:1078)
>     ... 27 more
>
> I also confirmed that the jobtracker webUI at
> ec2-204-236-252-96.compute-1.amazonaws.com:50030 doesn't work (the.
> namenode UI is OK)
>
> Here is the log at cluster lauching
>
> *khiem@master ~ $ whirr launch-cluster --config hadoop.properties
> Bootstrapping cluster
> Configuring template
> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
> Configuring template
> Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]
> Nodes started: [[id=us-east-1/i-3666f951, providerId=i-3666f951,
> group=myhadoopcluster, name=myhadoopcluster-3666f951,
> location=[id=us-east-1d, scope=ZONE, description=us-east-1d,
> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
> imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu,
> version=10.04, arch=paravirtual, is64Bit=false,
> description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930],
> state=RUNNING, loginPort=22, hostname=domU-12-31-39-15-2C-5F,
> privateAddresses=[10.207.47.173], publicAddresses=[204.236.252.96],
> hardware=[id=m1.small, providerId=m1.small, name=null,
> processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null,
> type=LOCAL, size=150.0, device=/dev/sda2, durable=false,
> isBootDevice=false], [id=vol-48903027, type=SAN, size=null,
> device=/dev/sda1, durable=true, isBootDevice=true]],
> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())),
> tags=[]], loginUser=ubuntu, userMetadata={Name=myhadoopcluster-3666f951},
> tags=[]]]
> Nodes started: [[id=us-east-1/i-f666f991, providerId=i-f666f991,
> group=myhadoopcluster, name=myhadoopcluster-f666f991,
> location=[id=us-east-1d, scope=ZONE, description=us-east-1d,
> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
> imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu,
> version=10.04, arch=paravirtual, is64Bit=false,
> description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930],
> state=RUNNING, loginPort=22, hostname=domU-12-31-39-03-40-E6,
> privateAddresses=[10.249.67.20], publicAddresses=[75.101.215.191],
> hardware=[id=m1.small, providerId=m1.small, name=null,
> processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null,
> type=LOCAL, size=150.0, device=/dev/sda2, durable=false,
> isBootDevice=false], [id=vol-2490304b, type=SAN, size=null,
> device=/dev/sda1, durable=true, isBootDevice=true]],
> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())),
> tags=[]], loginUser=ubuntu, userMetadata={Name=myhadoopcluster-f666f991},
> tags=[]]]
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [50030]
> for [123.24.155.127/32]
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [8021] for
> [204.236.252.96/32]
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [50070]
> for [123.24.155.127/32]
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [8020,
> 8021] for [204.236.252.96/32]
> The permission '204.236.252.96/32-1-8021-8021' has already been
> authorized on the specified group
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [50030]
> for [123.24.155.127/32]
> The permission '123.24.155.127/32-1-50030-50030' has already been
> authorized on the specified group
> Authorizing firewall ingress to [us-east-1/i-3666f951] on ports [8021] for
> [204.236.252.96/32]
> The permission '204.236.252.96/32-1-8021-8021' has already been
> authorized on the specified group
> Starting to run scripts on cluster for phase configureinstances:
> us-east-1/i-f666f991
> Starting to run scripts on cluster for phase configureinstances:
> us-east-1/i-3666f951
> Running configure phase script on: us-east-1/i-f666f991
> Running configure phase script on: us-east-1/i-3666f951
> configure phase script run completed on: us-east-1/i-f666f991
> configure phase script run completed on: us-east-1/i-3666f951
> Successfully executed configure script: [output=Fetched 248kB in 0s
> (7,956kB/s)
> Selecting previously deselected package hadoop-0.20-namenode.
> (Reading database ... 28596 files and directories currently installed.)
> Unpacking hadoop-0.20-namenode (from
> .../hadoop-0.20-namenode_0.20.2+923.197-1~lucid-cdh3_all.deb) ...
> Processing triggers for ureadahead ...
> Setting up hadoop-0.20-namenode (0.20.2+923.197-1~lucid-cdh3) ...
>
> Starting Hadoop namenode daemon: starting namenode, logging to
> /var/log/hadoop/logs/hadoop-hadoop-namenode-domU-12-31-39-15-2C-5F.out
> hadoop-0.20-namenode.
> Safe mode is OFF
> , error=12/04/16 16:27:54 INFO security.UserGroupInformation: JAAS
> Configuration already set up for Hadoop, not re-installing.
> 12/04/16 16:27:56 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:27:58 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:27:59 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:01 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:03 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:05 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:07 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:08 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> 12/04/16 16:28:10 INFO security.UserGroupInformation: JAAS Configuration
> already set up for Hadoop, not re-installing.
> , exitCode=0]
> Successfully executed configure script: [output=Get:1
> http://archive.cloudera.com/debian/ lucid-cdh3/contrib
> hadoop-0.20-tasktracker 0.20.2+923.197-1~lucid-cdh3 [248kB]
> Fetched 248kB in 0s (6,047kB/s)
> Selecting previously deselected package hadoop-0.20-tasktracker.
> (Reading database ... 28596 files and directories currently installed.)
> Unpacking hadoop-0.20-tasktracker (from
> .../hadoop-0.20-tasktracker_0.20.2+923.197-1~lucid-cdh3_all.deb) ...
> Processing triggers for ureadahead ...
> Setting up hadoop-0.20-tasktracker (0.20.2+923.197-1~lucid-cdh3) ...
>
> Starting Hadoop tasktracker daemon: starting tasktracker, logging to
> /var/log/hadoop/logs/hadoop-hadoop-tasktracker-domU-12-31-39-03-40-E6.out
> hadoop-0.20-tasktracker.
> , error=dpkg-preconfigure: unable to re-open stdin:
> update-rc.d: warning: hadoop-0.20-datanode start runlevel arguments (2 3 4
> 5) do not match LSB Default-Start values (3 5)
> update-rc.d: warning: hadoop-0.20-datanode stop runlevel arguments (0 1 6)
> do not match LSB Default-Stop values (0 1 2 4 6)
> dpkg-preconfigure: unable to re-open stdin:
> update-rc.d: warning: hadoop-0.20-tasktracker start runlevel arguments (2
> 3 4 5) do not match LSB Default-Start values (3 5)
> update-rc.d: warning: hadoop-0.20-tasktracker stop runlevel arguments (0 1
> 6) do not match LSB Default-Stop values (0 1 2 4 6)
> , exitCode=0]
> Finished running configure phase scripts on all cluster instances
> Completed configuration of myhadoopcluster role hadoop-jobtracker
> Jobtracker web UI available at
> http://ec2-204-236-252-96.compute-1.amazonaws.com:50030
> Completed configuration of myhadoopcluster role hadoop-namenode
> Namenode web UI available at
> http://ec2-204-236-252-96.compute-1.amazonaws.com:50070
> Wrote Hadoop site file /home/khiem/.whirr/myhadoopcluster/hadoop-site.xml
> Wrote Hadoop proxy script
> /home/khiem/.whirr/myhadoopcluster/hadoop-proxy.sh
> Completed configuration of myhadoopcluster role hadoop-datanode
> Completed configuration of myhadoopcluster role hadoop-tasktracker
> Wrote instances file /home/khiem/.whirr/myhadoopcluster/instances
> Started cluster of 2 instances
> Cluster{instances=[Instance{roles=[hadoop-datanode, hadoop-tasktracker],
> publicIp=75.101.215.191, privateIp=10.249.67.20, id=us-east-1/i-f666f991,
> nodeMetadata=[id=us-east-1/i-f666f991, providerId=i-f666f991,
> group=myhadoopcluster, name=myhadoopcluster-f666f991,
> location=[id=us-east-1d, scope=ZONE, description=us-east-1d,
> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
> imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu,
> version=10.04, arch=paravirtual, is64Bit=false,
> description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930],
> state=RUNNING, loginPort=22, hostname=domU-12-31-39-03-40-E6,
> privateAddresses=[10.249.67.20], publicAddresses=[75.101.215.191],
> hardware=[id=m1.small, providerId=m1.small, name=null,
> processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null,
> type=LOCAL, size=150.0, device=/dev/sda2, durable=false,
> isBootDevice=false], [id=vol-2490304b, type=SAN, size=null,
> device=/dev/sda1, durable=true, isBootDevice=true]],
> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())),
> tags=[]], loginUser=ubuntu, userMetadata={Name=myhadoopcluster-f666f991},
> tags=[]]}, Instance{roles=[hadoop-jobtracker, hadoop-namenode],
> publicIp=204.236.252.96, privateIp=10.207.47.173, id=us-east-1/i-3666f951,
> nodeMetadata=[id=us-east-1/i-3666f951, providerId=i-3666f951,
> group=myhadoopcluster, name=myhadoopcluster-3666f951,
> location=[id=us-east-1d, scope=ZONE, description=us-east-1d,
> parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null,
> imageId=us-east-1/ami-ab36fbc2, os=[name=null, family=ubuntu,
> version=10.04, arch=paravirtual, is64Bit=false,
> description=099720109477/ebs/ubuntu-images/ubuntu-lucid-10.04-i386-server-20110930],
> state=RUNNING, loginPort=22, hostname=domU-12-31-39-15-2C-5F,
> privateAddresses=[10.207.47.173], publicAddresses=[204.236.252.96],
> hardware=[id=m1.small, providerId=m1.small, name=null,
> processors=[[cores=1.0, speed=1.0]], ram=1740, volumes=[[id=null,
> type=LOCAL, size=150.0, device=/dev/sda2, durable=false,
> isBootDevice=false], [id=vol-48903027, type=SAN, size=null,
> device=/dev/sda1, durable=true, isBootDevice=true]],
> supportsImage=And(ALWAYS_TRUE,Or(isWindows(),requiresVirtualizationType(paravirtual)),ALWAYS_TRUE,Not(is64Bit())),
> tags=[]], loginUser=ubuntu, userMetadata={Name=myhadoopcluster-3666f951},
> tags=[]]}], configuration={hadoop.job.ugi=root,root, mapred.job.tracker=
> ec2-204-236-252-96.compute-1.amazonaws.com:8021,
> hadoop.socks.server=localhost:6666,
> fs.s3n.awsAccessKeyId=AKIAIGXAURLAB7CQE77A,
> fs.s3.awsSecretAccessKey=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,
> fs.s3.awsAccessKeyId=AKIAIGXAURLAB7CQE77A,
> hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory,
> fs.default.name=hdfs://ec2-204-236-252-96.compute-1.amazonaws.com:8020/,
> fs.s3n.awsSecretAccessKey=dWDRq2z0EQhpdPrbbL8Djs3eCu98O32r3gOrIbOK}}
> You can log into instances using the following ssh commands:
> 'ssh -i /home/khiem/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o
> StrictHostKeyChecking=no khiem@75.101.215.191'
> 'ssh -i /home/khiem/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o
> StrictHostKeyChecking=no khiem@204.236.252.96'
> **
> *
> On Mon, Apr 16, 2012 at 4:57 PM, Andrei Savu <savu.andrei@gmail.com>wrote:
>
>> Try the latest release - 0.7.1. All previous releases are unable to
>> install a JDK.
>> On Apr 16, 2012 10:12 AM, "Đỗ Hoàng Khiêm" <dohoangkhiem@gmail.com>
>> wrote:
>>
>>> Hi, I am new to Whirr and I'm trying to setup a Hadoop cluster on EC2
>>> with Whirr,I have followed the tutorial on Cloudera
>>> https://ccp.cloudera.com/display/CDHDOC/Whirr+Installation
>>>
>>> Before install Whirr, I install Hadoop (0.20.2-cdh3u3), then install
>>> Whirr (0.5.0-cdh3u3) on my local machine (running Linux Mint 11).
>>>
>>> Here's my cluster config file
>>>
>>> whirr.cluster-name=large-cluster
>>> whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
>>> whirr.provider=aws-ec2
>>> whirr.identity=XXXXXXXXXXXXXXX
>>> whirr.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
>>> whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
>>> whirr.hadoop-install-function=install_cdh_hadoop
>>> whirr.hadoop-configure-function=configure_cdh_hadoop
>>> whirr.hardware-id=m1.large
>>> whirr.image-id=us-east-1/ami-da0cf8b3
>>> whirr.location-id=us-east-1
>>>
>>> The cluster lauching looks normally
>>>
>>> khiem@master ~ $ whirr launch-cluster --config large-hadoop.properties
>>> Bootstrapping cluster
>>> Configuring template
>>> Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
>>> Configuring template
>>> Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]
>>> Nodes started: [[id=us-east-1/i-9aa01dfd, providerId=i-9aa01dfd, group=large-cluster,
name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1,
iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.196.142.64], publicAddresses=[107.20.64.97],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]]
>>> Nodes started: [[id=us-east-1/i-0aa31e6d, providerId=i-0aa31e6d, group=large-cluster,
name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1,
iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null,
family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]]
>>> Authorizing firewall ingress to [Instance{roles=[hadoop-jobtracker, hadoop-namenode],
publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d,
providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE,
description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3,
os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]}] on ports [50070, 50030] for [116.96.138.41/32]
>>> Authorizing firewall ingress to [Instance{roles=[hadoop-jobtracker, hadoop-namenode],
publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d,
providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE,
description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3,
os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]}] on ports [8020, 8021] for [50.17.128.123/32]
>>> Running configuration script
>>> Configuration script run completed
>>> Running configuration script
>>> Configuration script run completed
>>> Completed configuration of large-cluster
>>> Namenode web UI available at http://ec2-50-17-128-123.compute-1.amazonaws.com:50070
>>> Jobtracker web UI available at http://ec2-50-17-128-123.compute-1.amazonaws.com:50030
>>> Wrote Hadoop site file /home/khiem/.whirr/large-cluster/hadoop-site.xml
>>> Wrote Hadoop proxy script /home/khiem/.whirr/large-cluster/hadoop-proxy.sh
>>> Wrote instances file /home/khiem/.whirr/large-cluster/instances
>>> Started cluster of 2 instances
>>> Cluster{instances=[Instance{roles=[hadoop-datanode, hadoop-tasktracker], publicIp=107.20.64.97,
privateIp=10.196.142.64, id=us-east-1/i-9aa01dfd, nodeMetadata=[id=us-east-1/i-9aa01dfd, providerId=i-9aa01dfd,
group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a,
parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3,
os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.196.142.64], publicAddresses=[107.20.64.97],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]}, Instance{roles=[hadoop-jobtracker, hadoop-namenode],
publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d,
providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE,
description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3,
os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml],
state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123],
hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]],
ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true],
[id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null,
type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()],
loginUser=ubuntu, userMetadata={}]}], configuration={hadoop.job.ugi=root,root, mapred.job.tracker=ec2-50-17-128-123.compute-1.amazonaws.com:8021,
hadoop.socks.server=localhost:6666, fs.s3n.awsAccessKeyId=AKIAIGXAURLAB7CQE77A, fs.s3.awsSecretAccessKey=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,
fs.s3.awsAccessKeyId=AZIAIGXIOPLAB7CQE77A, hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory,
fs.default.name=hdfs://ec2-50-17-128-123.compute-1.amazonaws.com:8020/, fs.s3n.awsSecretAccessKey=dWDRq2z0EQegdPrbbL8Dab3eCu98O32r3gOrIbOK}}
>>>
>>>
>>>
>>>
>>>
>>> I've also started the proxy and update the local Hadoop configuration
>>> follow Cloudera tutorial, but when I tried to test the HDFS with hadoop
>>> fs -ls /
>>>
>>> the terminal prints connection error:
>>>
>>> 12/04/12 11:54:43 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found
in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml
and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml
respectively
>>> 12/04/12 11:54:43 INFO security.UserGroupInformation: JAAS Configuration already
set up for Hadoop, not re-installing.
>>> 12/04/12 11:54:45 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020.
Already tried 0 time(s).
>>> 12/04/12 11:54:46 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020.
Already tried 1 time(s).
>>> 12/04/12 11:54:48 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020.
Already tried 2 time(s).
>>> 12/04/12 11:54:49 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020.
Already tried 3 time(s)
>>>
>>>
>>>
>>>
>>>
>>> In the proxy terminal
>>>
>>> Running proxy to Hadoop cluster atec2-50-17-128-123.compute-1.amazonaws.com.
Use Ctrl-c to quit.
>>> Warning: Permanently added 'ec2-50-17-128-123.compute-1.amazonaws.com,50.17.128.123'
(RSA) to the list of known hosts.
>>> channel 2: open failed: connect failed: Connection refused channel 2: open failed:
connect failed: Connection refused
>>> channel 2: open failed: connect failed: Connection refused
>>> channel 2: open failed: connect failed: Connection refused
>>> channel 2: open failed: connect failed: Connection refused
>>>
>>> The namenode webUI (50070 port) also not available, I can ssh to the
>>> namenode but inside the namenode, it looks like there's none of Hadoop or
>>> Java installation, is this strange thing?
>>>
>>> Any comment is appreciated.
>>>
>>> .
>>>
>>>
>>>
>>>
>

Mime
View raw message