whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From a b <autohan...@yahoo.com>
Subject Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
Date Tue, 06 Aug 2013 18:04:35 GMT
i can't find java or hadoop - i'll look at the whirr.log for more info:

You can log into instances using the following ssh commands:
[hadoop-datanode+hadoop-tasktracker]: ssh -i /home/ab/.ssh/id_rsa -o "UserKnownHostsFile /dev/null"
-o StrictHostKeyChecking=no ab@54.224.175.65
[hadoop-namenode+hadoop-jobtracker]: ssh -i /home/ab/.ssh/id_rsa -o "UserKnownHostsFile /dev/null"
-o StrictHostKeyChecking=no ab@54.227.189.132
To destroy cluster, run 'whirr destroy-cluster' with the same options used to launch it.
ab@ubuntu12-64:~$ ssh -i /home/ab/.ssh/id_rsa -o "UserKnownHostsFile /dev/null" -o StrictHostKeyChecking=no
ab@54.224.175.65
Warning: Permanently added '54.224.175.65' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-48-virtual x86_64)

 * Documentation:  https://help.ubuntu.com/

  System information as of Tue Aug  6 17:59:19 UTC 2013

  System load:  0.08              Processes:           57
  Usage of /:   10.2% of 7.87GB   Users logged in:     0
  Memory usage: 26%               IP address for eth0: 10.179.37.185
  Swap usage:   0%

  Graph this data and manage this system at https://landscape.canonical.com/

  Get cloud support with Ubuntu Advantage Cloud Guest:
    http://www.ubuntu.com/business/services/cloud

  Use Juju to deploy your cloud instances and workloads:
    https://juju.ubuntu.com/#cloud-precise

24 packages can be updated.
24 updates are security updates.

Last login: Tue Aug  6 17:53:09 2013 from 50.0.160.200
ab@ip-10-179-37-185:~$ which java
ab@ip-10-179-37-185:~$ ps -ef 
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:49 ?        00:00:00 /sbin/init
root         2     0  0 17:49 ?        00:00:00 [kthreadd]
root         3     2  0 17:49 ?        00:00:00 [ksoftirqd/0]
root         4     2  0 17:49 ?        00:00:00 [kworker/0:0]
root         5     2  0 17:49 ?        00:00:00 [kworker/u:0]
root         6     2  0 17:49 ?        00:00:00 [migration/0]
root         7     2  0 17:49 ?        00:00:00 [watchdog/0]
root         8     2  0 17:49 ?        00:00:00 [cpuset]
root         9     2  0 17:49 ?        00:00:00 [khelper]
root        10     2  0 17:49 ?        00:00:00 [kdevtmpfs]
root        11     2  0 17:49 ?        00:00:00 [netns]
root        12     2  0 17:49 ?        00:00:00 [xenwatch]
root        13     2  0 17:49 ?        00:00:01 [xenbus]
root        14     2  0 17:49 ?        00:00:00 [sync_supers]
root        15     2  0 17:49 ?        00:00:00 [bdi-default]
root        16     2  0 17:49 ?        00:00:00 [kintegrityd]
root        17     2  0 17:49 ?        00:00:00 [kblockd]
root        18     2  0 17:49 ?        00:00:00 [ata_sff]
root        19     2  0 17:49 ?        00:00:00 [khubd]
root        20     2  0 17:49 ?        00:00:00 [md]
root        21     2  0 17:49 ?        00:00:00 [kworker/0:1]
root        23     2  0 17:49 ?        00:00:00 [kworker/u:1]
root        24     2  0 17:49 ?        00:00:00 [khungtaskd]
root        25     2  0 17:49 ?        00:00:00 [kswapd0]
root        26     2  0 17:49 ?        00:00:00 [ksmd]
root        27     2  0 17:49 ?        00:00:00 [fsnotify_mark]
root        28     2  0 17:49 ?        00:00:00 [ecryptfs-kthrea]
root        29     2  0 17:49 ?        00:00:00 [crypto]
root        37     2  0 17:49 ?        00:00:00 [kthrotld]
root        38     2  0 17:49 ?        00:00:00 [khvcd]
root        57     2  0 17:49 ?        00:00:00 [devfreq_wq]
root       158     2  0 17:49 ?        00:00:00 [jbd2/xvda1-8]
root       159     2  0 17:49 ?        00:00:00 [ext4-dio-unwrit]
root       270     1  0 17:49 ?        00:00:00 upstart-udev-bridge --daemon
root       272     1  0 17:49 ?        00:00:00 /sbin/udevd --daemon
root       319   272  0 17:49 ?        00:00:00 /sbin/udevd --daemon
root       320   272  0 17:49 ?        00:00:00 /sbin/udevd --daemon
root       413     1  0 17:49 ?        00:00:00 upstart-socket-bridge --daemon
root       468     1  0 17:49 ?        00:00:00 dhclient3 -e IF_METRIC=100
-pf /var/run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases -1
root       615     1  0 17:49 ?        00:00:00 /usr/sbin/sshd -D
syslog     631     1  0 17:49 ?        00:00:00 rsyslogd -c5
102        651     1  0 17:49 ?        00:00:00 dbus-daemon --system --fork
--activation=upstart
root       700     1  0 17:49 tty4     00:00:00 /sbin/getty -8 38400 tty4
root       707     1  0 17:49 tty5     00:00:00 /sbin/getty -8 38400 tty5
root       713     1  0 17:49 tty2     00:00:00 /sbin/getty -8 38400 tty2
root       714     1  0 17:49 tty3     00:00:00 /sbin/getty -8 38400 tty3
root       717     1  0 17:49 tty6     00:00:00 /sbin/getty -8 38400 tty6
daemon     730     1  0 17:49 ?        00:00:00 atd
root       731     1  0 17:49 ?        00:00:00 cron
root       732     1  0 17:49 ?        00:00:00 acpid -c /etc/acpi/events
-s /var/run/acpid.socket
root       764     1  0 17:49 tty1     00:00:00 /sbin/getty -8 38400 tty1
whoopsie   768     1  0 17:49 ?        00:00:00 whoopsie
root      1523   615  0 17:59 ?        00:00:00 sshd: ab [priv]  
ab        1620  1523  0 17:59 ?        00:00:00 sshd: ab@pts/0   
ab        1621  1620  1 17:59 pts/0    00:00:00 -bash
root      1670     2  0 17:59 ?        00:00:00 [flush-202:1]
ab        1672  1621  0 17:59 pts/0    00:00:00 ps -ef
ab@ip-10-179-37-185:~$ which hadoop
ab@ip-10-179-37-185:~$ 





________________________________
 From: Andrew Bayer <andrew.bayer@gmail.com>
To: user@whirr.apache.org; a b <autohandle@yahoo.com> 
Sent: Tuesday, August 6, 2013 10:35 AM
Subject: Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
 


Do you have the whirr.log from that attempt?

A.


On Mon, Aug 5, 2013 at 8:34 PM, a b <autohandle@yahoo.com> wrote:

i built whirr in my own directory - i didn't change it (yet) - i just check it out and tried
to compile - you can see i had some memory issues that i didn't notice right away:
>
>
>
> 1530  git clone git://git.apache.org/whirr.git
> 1531  mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs
> 1532  cd whirr/
> 1533  mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs
> 1534  mvn install
> 1535  cd
> 1536  rm whirr.log
> 1537  ~/git/whirr/bin/whirr launch-cluster
 --config ~/whirr/recipes/hadoop.properties 
> 1538  export MAVEN_OPTS=-Xmx200m
> 1539  cd ~/git/whirr/
> 1540  mvn install
> 1541  export MAVEN_OPTS=-Xmx1G
> 1542  mvn install
> 1543  cd
> 1544  ~/git/whirr/bin/whirr launch-cluster --config ~/whirr/recipes/hadoop.properties

> 1545  history
>
>
>
>this is the console at the end of "mvn install"
>
>
>[INFO] ------------------------------------------------------------------------
>[INFO] Reactor Summary:
>[INFO] 
>[INFO] Apache Whirr Build Tools .......................... SUCCESS [2.519s]
>[INFO] Whirr ............................................. SUCCESS [4.662s]
>[INFO] Apache Whirr Core ................................. SUCCESS [1:43.913s]
>[INFO] Apache Whirr Cassandra ............................ SUCCESS [10.705s]
>[INFO] Apache Whirr Ganglia .............................. SUCCESS [3.683s]
>[INFO] Apache Whirr Hadoop ............................... SUCCESS [11.294s]
>[INFO] Apache Whirr ZooKeeper ............................ SUCCESS [4.477s]
>[INFO] Apache Whirr HBase
 ................................ SUCCESS [5.196s]
>[INFO] Apache Whirr YARN ................................. SUCCESS [4.867s]
>[INFO] Apache Whirr CDH .................................. SUCCESS [5.552s]
>[INFO] Apache Whirr Mahout ............................... SUCCESS [2.452s]
>[INFO] Apache Whirr Pig .................................. SUCCESS [2.393s]
>[INFO] Apache Whirr ElasticSearch ........................ SUCCESS [4.145s]
>[INFO] Apache Whirr Hama ................................. SUCCESS [2.841s]
>[INFO] Apache Whirr Puppet ............................... SUCCESS [3.242s]
>[INFO] Apache Whirr Chef ................................. SUCCESS [12.153s]
>[INFO] Apache Whirr Solr ................................. SUCCESS [4.997s]
>[INFO] Apache Whirr Kerberos ............................. SUCCESS [12.207s]
>[INFO] Apache Whirr CLI .................................. SUCCESS [14.940s]
>[INFO] Apache Whirr Examples
 ............................. SUCCESS [4.494s]
>[INFO] ------------------------------------------------------------------------
>[INFO] BUILD SUCCESS
>[INFO] ------------------------------------------------------------------------
>[INFO] Total time: 3:43.355s
>[INFO] Finished at: Mon Aug 05 20:11:58 PDT 2013
>[INFO] Final Memory: 109M/262M
>[INFO] ------------------------------------------------------------------------
>ab@ubuntu12-64:~/git/whirr$ cd
>ab@ubuntu12-64:~$ ~/git/whirr/bin/whirr launch-cluster --config ~/whirr/recipes/hadoop.properties

>Running on provider aws-ec2 using identity XXXXXXXXXXXXXXXXXXX
>Exception in thread "main" java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException:
java.io.IOException: Too many instance failed while bootstrapping! 0 successfully started
instances while 0 instances failed
>    at
 org.apache.whirr.ClusterController.launchCluster(ClusterController.java:128)
>    at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
>    at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
>    at org.apache.whirr.cli.Main.run(Main.java:69)
>    at org.apache.whirr.cli.Main.main(Main.java:102)
>Caused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException:
Too many instance failed while bootstrapping! 0 successfully started instances while 0 instances
failed
>    at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125)
>    at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:131)
>    at
 org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
>    at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
>    ... 4 more
>Caused by: java.util.concurrent.ExecutionException: java.io.IOException: Too many instance
failed while bootstrapping! 0 successfully started instances while 0 instances failed
>    at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>    at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>    at org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120)
>    ... 7 more
>Caused by: java.io.IOException: Too many instance failed while bootstrapping! 0 successfully
started instances while 0 instances failed
>    at
 org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:93)
>    at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
>    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>    at java.lang.Thread.run(Thread.java:724)
>
>
>
>2 instances were started - like i asked for - the 1st instance was terminated almost immediately,
the 2nd was left running for a while - eventually, i hit ^c on the launch and terminated the
2nd instance from the aws console.
>
>
>i need some more coaching.
>
>
>
>
>________________________________
> From: Andrew Bayer <andrew.bayer@gmail.com>
>To: user@whirr.apache.org; a b <autohandle@yahoo.com> 
>Sent: Monday, August 5, 2013 5:54 PM
>
>Subject: Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
> 
>
>
>Try not building as root - that can throw things off.
>
>
>A.
>
>
>On Mon, Aug 5, 2013 at 5:44 PM, a b <autohandle@yahoo.com> wrote:
>
>as for the java 7 problem - i found this suggestion:
>>
>>http://stackoverflow.com/questions/13225149/how-to-install-jdk-7-on-ec2-cluster-via-whirr
>>
>>
>>i tried to download whirr - as suggested here:
>>
>>
>>https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
>>
>>
>>o i did: git clone ...
>>
>>o i modified: core/src/main/resources/functions/...
>>o i did: mvn eclipse:eclipse ...
>>o i skipped: eclipse import
>>o i ran: mvn install
>>
>>
>>it fails in the "mvn install" during test
>>
>>
>>Tests in error: 
>> 
 testActionIsExecutedOnAllRelevantNodes(org.apache.whirr.actions.StopServicesActionTest):
cluster-user != root or do not run as root
>>  testFilterScriptExecutionByRole(org.apache.whirr.actions.StopServicesActionTest):
cluster-user != root or do not run as root
>>  testFilterScriptExecutionByInstanceId(org.apache.whirr.actions.StopServicesActionTest):
cluster-user != root or do not run as root
>>  testFilterScriptExecutionByRoleAndInstanceId(org.apache.whirr.actions.StopServicesActionTest):
cluster-user != root or do not run as root
>>  testNoScriptExecutionsForNoop(org.apache.whirr.actions.StopServicesActionTest):
cluster-user != root or do not run as root
>>[..]
>>  testExecuteOnlyBootstrapForNoop(org.apache.whirr.service.DryRunModuleTest): cluster-user
!= root or do not run as root
>>  testExecuteOnlyBootstrapForNoopWithListener(org.apache.whirr.service.DryRunModuleTest):
cluster-user != root or do not run as root
>>  testNoInitScriptsAfterConfigurationStartedAndNoConfigScriptsAfterDestroy(org.apache.whirr.service.DryRunModuleTest):
cluster-user != root or do not run as root
>>  testOverrides(org.apache.whirr.command.AbstractClusterCommandTest): cluster-user
!= root or do not run as root
>>
>>Tests run: 99, Failures: 0, Errors: 49, Skipped: 0
>>
>>[INFO] ------------------------------------------------------------------------
>>[INFO] Reactor Summary:
>>[INFO] 
>>[INFO] Apache Whirr Build Tools .......................... SUCCESS [2.612s]
>>[INFO] Whirr ............................................. SUCCESS [4.488s]
>>[INFO] Apache Whirr Core ................................. FAILURE [17.113s]
>>[INFO] Apache Whirr Cassandra ............................ SKIPPED
>>[INFO]
 Apache Whirr Ganglia .............................. SKIPPED
>>[INFO] Apache Whirr Hadoop ............................... SKIPPED
>>
>>
>>
>>is there a better way to try this suggestion?
>>
>>
>>
>>
>>________________________________
>> From: Andrew Bayer <andrew.bayer@gmail.com>
>>
>>To: a b <autohandle@yahoo.com> 
>>Cc: "user@whirr.apache.org" <user@whirr.apache.org> 
>>Sent: Monday, August 5, 2013 4:48 PM
>>
>>Subject: Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>> 
>>
>>
>>Looks like it's the Oracle JDK7 download that's failing - http://download.oracle.com/otn-pub/java/jdk/7/jdk-7-linux-x64.tar.gz
isn't actually there. I don't know if they actually have a consistent link to get the JDK7
tarball regardless of version. I'd just use OpenJDK for now, if you can.
>>
>>
>>A.
>>
>>
>>On Mon, Aug 5, 2013 at 4:33 PM, a b <autohandle@yahoo.com> wrote:
>>
>>Get:19 http://us-east-1.ec2.archive.ubuntu.com precise/main amd64 Packages [1,273
kB]^M
>>>Get:20 http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64 Packages
[4,786 kB]^M
>>>Get:21 http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages [1,274
kB]^M
>>>Get:22 http://us-east-1.ec2.archive.ubuntu.com precise/universe i386 Packages
[4,796 kB]^M
>>>Get:23 http://us-east-1.ec2.archive.ubuntu.com precise/main TranslationIndex [3,706
B]^M
>>>Get:24 http://us-east-1.ec2.archive.ubuntu.com precise/universe TranslationIndex
[2,922 B]^M
>>>Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Sources [412
kB]^M
>>>Get:26 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Sources
[93.1 kB]^M
>>>Get:27 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64 Packages
[672 kB]^M
>>>Get:28 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe amd64
Packages [210 kB]^M
>>>Get:29 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386 Packages
[692 kB]^M
>>>Get:30 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe i386 Packages
[214 kB]^M
>>>Get:31 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main TranslationIndex
[3,564 B]^M
>>>Get:32 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe TranslationIndex
[2,850 B]^M
>>>Get:33 http://us-east-1.ec2.archive.ubuntu.com precise/main Translation-en [726
kB]^M
>>>Get:34 http://us-east-1.ec2.archive.ubuntu.com precise/universe Translation-en
[3,341 kB]^M
>>>Get:35 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main Translation-en
[298 kB]^M
>>>Get:36 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe Translation-en
[123 kB]^M
>>>Fetched 26.1 MB in 21s (1,241 kB/s)^M
>>>Reading package lists...^M
>>>Could not download  http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz.md5.
Continuing.^M
>>>, error=^M
>>>gzip: stdin: not in gzip format^M
>>>tar: Child returned status 1^M
>>>tar: Error is not recoverable: exiting now^M
>>>mv: cannot stat `jdk1*': No such file or directory^M
>>>update-alternatives: error: alternative path /usr/lib/jvm/java-7-oracle/bin/java
doesn't exist.^M
>>>
>>>
>>>
>>>
>>>________________________________
>>> From: Andrew Bayer <andrew.bayer@gmail.com>
>>>To: user@whirr.apache.org; a b <autohandle@yahoo.com> 
>>>Sent: Monday, August 5, 2013 4:22 PM
>>>
>>>Subject: Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>>> 
>>>
>>>
>>>Ok, Whirr does try to download the md5 file, but if it fails to find it, that's
not a blocking error - it'll keep going anyway. What's after that in the logs?
>>>
>>>
>>>A.
>>>
>>>
>>>On Mon, Aug 5, 2013 at 4:19 PM, a b <autohandle@yahoo.com> wrote:
>>>
>>>ok - i'm not sure what you are asking.
>>>>
>>>>whirr launch-cluster --config
~/whirr/recipes/hadoop.properties 
>>>>
>>>>
>>>>
>>>>where these are the properties i think i changed or added from the original
recipe:
>>>>
>>>>
>>>>whirr.cluster-name=hadoop-ec2
>>>>whirr.instance-templates=1
hadoop-namenode+hadoop-jobtracker,1 hadoop-datanode+hadoop-tasktracker
>>>>whirr.hardware-id=t1.micro
>>>>whirr.image-id=us-east-1/ami-25d9a94c
>>>>whirr.hadoop.version=1.2.1
>>>>whirr.provider=aws-ec2
>>>>whirr.identity=${env:AWS_ACCESS_KEY} 
>>>>whirr.credential=${env:AWS_SECRET_KEY}
>>>>whirr.location-id=us-east-1
>>>>whirr.java.install-function=install_oracle_jdk7
>>>>
>>>>
>>>>
>>>>________________________________
>>>> From: Andrew Bayer <andrew.bayer@gmail.com>
>>>>To: user@whirr.apache.org; a b <autohandle@yahoo.com> 
>>>>Sent: Monday, August 5, 2013 4:11 PM
>>>>Subject: Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>>>> 
>>>>
>>>>
>>>>Ok, it looks like they've actually been doing .mds for a while. Where are
you seeing this error?
>>>>
>>>>
>>>>A.
>>>>
>>>>
>>>>On Mon, Aug 5, 2013 at 4:09 PM, Andrew Bayer <andrew.bayer@gmail.com>
wrote:
>>>>
>>>>That looks like they broke the hadoop-1.2.1 release - the file should be .md5.
I'd bug the Hadoop project about that.
>>>>>
>>>>>A.
>>>>>
>>>>>
>>>>>
>>>>>On Mon, Aug 5, 2013 at 3:42 PM, a b <autohandle@yahoo.com> wrote:
>>>>>
>>>>>i get a whirr error:
>>>>>>
>>>>>>
>>>>>>Could not download  http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz.md5
>>>>>>
>>>>>>
>>>>>>when i browse over to: http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/
>>>>>>i can see the file is named: hadoop-1.2.1.tar.gz.mds
>>>>>>
>>>>>>
>>>>>>how do i tell whirr to use a different suffix?
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>>
>
>
>
Mime
View raw message