whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Bayer <andrew.ba...@gmail.com>
Subject Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
Date Tue, 06 Aug 2013 17:51:50 GMT
You can send it to me directly if you want.

On the openjdk thing - I just don't get why openjdk-6-jdk isn't going to
install its dependencies. That's normal apt-get behavior.

A.

On Tue, Aug 6, 2013 at 10:42 AM, a b <autohandle@yahoo.com> wrote:

> no - but i can recreate it - it is probably too large to post - will it be
> obvious what to extract?
>
>
>   ------------------------------
>  *From:* Andrew Bayer <andrew.bayer@gmail.com>
> *To:* user@whirr.apache.org; a b <autohandle@yahoo.com>
> *Sent:* Tuesday, August 6, 2013 10:35 AM
>
> *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>
> Do you have the whirr.log from that attempt?
>
> A.
>
> On Mon, Aug 5, 2013 at 8:34 PM, a b <autohandle@yahoo.com> wrote:
>
> i built whirr in my own directory - i didn't change it (yet) - i just
> check it out and tried to compile - you can see i had some memory issues
> that i didn't notice right away:
>
>  1530  git clone git://git.apache.org/whirr.git
>  1531  mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs
>  1532  cd whirr/
>  1533  mvn eclipse:eclipse -DdownloadSources=true -DdownloadJavadocs
>  1534  mvn install
>  1535  cd
>  1536  rm whirr.log
>  1537  ~/git/whirr/bin/whirr launch-cluster --config
> ~/whirr/recipes/hadoop.properties
>  1538  export MAVEN_OPTS=-Xmx200m
>  1539  cd ~/git/whirr/
>  1540  mvn install
>  1541  export MAVEN_OPTS=-Xmx1G
>  1542  mvn install
>  1543  cd
>  1544  ~/git/whirr/bin/whirr launch-cluster --config
> ~/whirr/recipes/hadoop.properties
>  1545  history
>
> this is the console at the end of "mvn install"
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Whirr Build Tools .......................... SUCCESS [2.519s]
> [INFO] Whirr ............................................. SUCCESS [4.662s]
> [INFO] Apache Whirr Core ................................. SUCCESS
> [1:43.913s]
> [INFO] Apache Whirr Cassandra ............................ SUCCESS
> [10.705s]
> [INFO] Apache Whirr Ganglia .............................. SUCCESS [3.683s]
> [INFO] Apache Whirr Hadoop ............................... SUCCESS
> [11.294s]
> [INFO] Apache Whirr ZooKeeper ............................ SUCCESS [4.477s]
> [INFO] Apache Whirr HBase ................................ SUCCESS [5.196s]
> [INFO] Apache Whirr YARN ................................. SUCCESS [4.867s]
> [INFO] Apache Whirr CDH .................................. SUCCESS [5.552s]
> [INFO] Apache Whirr Mahout ............................... SUCCESS [2.452s]
> [INFO] Apache Whirr Pig .................................. SUCCESS [2.393s]
> [INFO] Apache Whirr ElasticSearch ........................ SUCCESS [4.145s]
> [INFO] Apache Whirr Hama ................................. SUCCESS [2.841s]
> [INFO] Apache Whirr Puppet ............................... SUCCESS [3.242s]
> [INFO] Apache Whirr Chef ................................. SUCCESS
> [12.153s]
> [INFO] Apache Whirr Solr ................................. SUCCESS [4.997s]
> [INFO] Apache Whirr Kerberos ............................. SUCCESS
> [12.207s]
> [INFO] Apache Whirr CLI .................................. SUCCESS
> [14.940s]
> [INFO] Apache Whirr Examples ............................. SUCCESS [4.494s]
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 3:43.355s
> [INFO] Finished at: Mon Aug 05 20:11:58 PDT 2013
> [INFO] Final Memory: 109M/262M
> [INFO]
> ------------------------------------------------------------------------
> ab@ubuntu12-64:~/git/whirr$ cd
> ab@ubuntu12-64:~$ ~/git/whirr/bin/whirr launch-cluster --config
> ~/whirr/recipes/hadoop.properties
> Running on provider aws-ec2 using identity XXXXXXXXXXXXXXXXXXX
> Exception in thread "main" java.lang.RuntimeException:
> java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: Too many instance failed while bootstrapping! 0
> successfully started instances while 0 instances failed
>     at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:128)
>     at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
>     at
> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
>     at org.apache.whirr.cli.Main.run(Main.java:69)
>     at org.apache.whirr.cli.Main.main(Main.java:102)
> Caused by: java.io.IOException: java.util.concurrent.ExecutionException:
> java.io.IOException: Too many instance failed while bootstrapping! 0
> successfully started instances while 0 instances failed
>     at
> org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:125)
>     at
> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:131)
>     at
> org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
>     at
> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
>     ... 4 more
> Caused by: java.util.concurrent.ExecutionException: java.io.IOException:
> Too many instance failed while bootstrapping! 0 successfully started
> instances while 0 instances failed
>     at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>     at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>     at
> org.apache.whirr.actions.BootstrapClusterAction.doAction(BootstrapClusterAction.java:120)
>     ... 7 more
> Caused by: java.io.IOException: Too many instance failed while
> bootstrapping! 0 successfully started instances while 0 instances failed
>     at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:93)
>     at org.apache.whirr.compute.StartupProcess.call(StartupProcess.java:41)
>     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
>     at java.util.concurrent.FutureTask.run(FutureTask.java:166)
>     at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>     at java.lang.Thread.run(Thread.java:724)
>
> 2 instances were started - like i asked for - the 1st instance was
> terminated almost immediately, the 2nd was left running for a while -
> eventually, i hit ^c on the launch and terminated the 2nd instance from the
> aws console.
>
> i need some more coaching.
>
>   ------------------------------
>  *From:* Andrew Bayer <andrew.bayer@gmail.com>
> *To:* user@whirr.apache.org; a b <autohandle@yahoo.com>
> *Sent:* Monday, August 5, 2013 5:54 PM
>
> *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>
> Try not building as root - that can throw things off.
>
> A.
>
> On Mon, Aug 5, 2013 at 5:44 PM, a b <autohandle@yahoo.com> wrote:
>
> as for the java 7 problem - i found this suggestion:
>
>
> http://stackoverflow.com/questions/13225149/how-to-install-jdk-7-on-ec2-cluster-via-whirr
>
> i tried to download whirr - as suggested here:
>
> https://cwiki.apache.org/confluence/display/WHIRR/How+To+Contribute
>
> o i did: git clone ...
> o i modified: core/src/main/resources/functions/...
> o i did: mvn eclipse:eclipse ...
> o i skipped: eclipse import
> o i ran: mvn install
>
> it fails in the "mvn install" during test
>
> Tests in error:
>
> testActionIsExecutedOnAllRelevantNodes(org.apache.whirr.actions.StopServicesActionTest):
> cluster-user != root or do not run as root
>
> testFilterScriptExecutionByRole(org.apache.whirr.actions.StopServicesActionTest):
> cluster-user != root or do not run as root
>
> testFilterScriptExecutionByInstanceId(org.apache.whirr.actions.StopServicesActionTest):
> cluster-user != root or do not run as root
>
> testFilterScriptExecutionByRoleAndInstanceId(org.apache.whirr.actions.StopServicesActionTest):
> cluster-user != root or do not run as root
>
> testNoScriptExecutionsForNoop(org.apache.whirr.actions.StopServicesActionTest):
> cluster-user != root or do not run as root
> [..]
>
> testExecuteOnlyBootstrapForNoop(org.apache.whirr.service.DryRunModuleTest):
> cluster-user != root or do not run as root
>
> testExecuteOnlyBootstrapForNoopWithListener(org.apache.whirr.service.DryRunModuleTest):
> cluster-user != root or do not run as root
>
> testNoInitScriptsAfterConfigurationStartedAndNoConfigScriptsAfterDestroy(org.apache.whirr.service.DryRunModuleTest):
> cluster-user != root or do not run as root
>   testOverrides(org.apache.whirr.command.AbstractClusterCommandTest):
> cluster-user != root or do not run as root
>
> Tests run: 99, Failures: 0, Errors: 49, Skipped: 0
>
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Apache Whirr Build Tools .......................... SUCCESS [2.612s]
> [INFO] Whirr ............................................. SUCCESS [4.488s]
> [INFO] Apache Whirr Core ................................. FAILURE
> [17.113s]
> [INFO] Apache Whirr Cassandra ............................ SKIPPED
> [INFO] Apache Whirr Ganglia .............................. SKIPPED
> [INFO] Apache Whirr Hadoop ............................... SKIPPED
>
> is there a better way to try this suggestion?
>
>   ------------------------------
>  *From:* Andrew Bayer <andrew.bayer@gmail.com>
>
> *To:* a b <autohandle@yahoo.com>
> *Cc:* "user@whirr.apache.org" <user@whirr.apache.org>
> *Sent:* Monday, August 5, 2013 4:48 PM
>
> *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>
> Looks like it's the Oracle JDK7 download that's failing -
> http://download.oracle.com/otn-pub/java/jdk/7/jdk-7-linux-x64.tar.gzisn't actually there.
I don't know if they actually have a consistent link
> to get the JDK7 tarball regardless of version. I'd just use OpenJDK for
> now, if you can.
>
> A.
>
> On Mon, Aug 5, 2013 at 4:33 PM, a b <autohandle@yahoo.com> wrote:
>
> Get:19 http://us-east-1.ec2.archive.ubuntu.com precise/main amd64
> Packages [1,273 kB]^M
> Get:20 http://us-east-1.ec2.archive.ubuntu.com precise/universe amd64
> Packages [4,786 kB]^M
> Get:21 http://us-east-1.ec2.archive.ubuntu.com precise/main i386 Packages
> [1,274 kB]^M
> Get:22 http://us-east-1.ec2.archive.ubuntu.com precise/universe i386
> Packages [4,796 kB]^M
> Get:23 http://us-east-1.ec2.archive.ubuntu.com precise/main
> TranslationIndex [3,706 B]^M
> Get:24 http://us-east-1.ec2.archive.ubuntu.com precise/universe
> TranslationIndex [2,922 B]^M
> Get:25 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main
> Sources [412 kB]^M
> Get:26 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe
> Sources [93.1 kB]^M
> Get:27 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main amd64
> Packages [672 kB]^M
> Get:28 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe
> amd64 Packages [210 kB]^M
> Get:29 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main i386
> Packages [692 kB]^M
> Get:30 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe
> i386 Packages [214 kB]^M
> Get:31 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main
> TranslationIndex [3,564 B]^M
> Get:32 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe
> TranslationIndex [2,850 B]^M
> Get:33 http://us-east-1.ec2.archive.ubuntu.com precise/main
> Translation-en [726 kB]^M
> Get:34 http://us-east-1.ec2.archive.ubuntu.com precise/universe
> Translation-en [3,341 kB]^M
> Get:35 http://us-east-1.ec2.archive.ubuntu.com precise-updates/main
> Translation-en [298 kB]^M
> Get:36 http://us-east-1.ec2.archive.ubuntu.com precise-updates/universe
> Translation-en [123 kB]^M
> Fetched 26.1 MB in 21s (1,241 kB/s)^M
> Reading package lists...^M
> Could not download
> http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz.md5.
> Continuing.^M
> , error=^M
> gzip: stdin: not in gzip format^M
> tar: Child returned status 1^M
> tar: Error is not recoverable: exiting now^M
> mv: cannot stat `jdk1*': No such file or directory^M
> update-alternatives: error: alternative path
> /usr/lib/jvm/java-7-oracle/bin/java doesn't exist.^M
>
>   ------------------------------
>  *From:* Andrew Bayer <andrew.bayer@gmail.com>
> *To:* user@whirr.apache.org; a b <autohandle@yahoo.com>
> *Sent:* Monday, August 5, 2013 4:22 PM
>
> *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>
> Ok, Whirr does try to download the md5 file, but if it fails to find it,
> that's not a blocking error - it'll keep going anyway. What's after that in
> the logs?
>
> A.
>
> On Mon, Aug 5, 2013 at 4:19 PM, a b <autohandle@yahoo.com> wrote:
>
> ok - i'm not sure what you are asking.
>
> *whirr launch-cluster --config ~/whirr/recipes/hadoop.properties*
>
> where these are the properties i think i changed or added from the
> original recipe:
>
> *whirr.cluster-name=hadoop-ec2*
> *whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1
> hadoop-datanode+hadoop-tasktracker*
> *whirr.hardware-id=t1.micro*
> ***whirr.image-id=us-east-1/ami-25d9a94c*
>  ***whirr.hadoop.version=1.2.1*
>  ***whirr.provider=aws-ec2***
> *whirr.identity=${env:AWS_ACCESS_KEY}*
> *whirr.credential=${env:AWS_SECRET_KEY}*
>  *whirr.location-id=us-east-1*
> *whirr.java.install-function=install_oracle_jdk7*
>
>   ------------------------------
>  *From:* Andrew Bayer <andrew.bayer@gmail.com>
> *To:* user@whirr.apache.org; a b <autohandle@yahoo.com>
> *Sent:* Monday, August 5, 2013 4:11 PM
> *Subject:* Re: cannot find hadoop-1.2.1.tar.gz.md5 - file suffix is mds
>
> Ok, it looks like they've actually been doing .mds for a while. Where are
> you seeing this error?
>
> A.
>
> On Mon, Aug 5, 2013 at 4:09 PM, Andrew Bayer <andrew.bayer@gmail.com>wrote:
>
> That looks like they broke the hadoop-1.2.1 release - the file should be
> .md5. I'd bug the Hadoop project about that.
>
> A.
>
>
> On Mon, Aug 5, 2013 at 3:42 PM, a b <autohandle@yahoo.com> wrote:
>
> i get a whirr error:
>
> Could not download
> http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/hadoop-1.2.1.tar.gz.
> md5
>
> when i browse over to:
> http://apache.osuosl.org/hadoop/common/hadoop-1.2.1/
> i can see the file is named: hadoop-1.2.1.tar.gz.mds
>
> how do i tell whirr to use a different suffix?
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Mime
View raw message