whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Max Zhao" <mzhao...@qq.com>
Subject Anyone has ready-to-use hadoop properties file for Amazon EC2?
Date Sat, 10 May 2014 23:22:48 GMT
Dear All,
  
 I just started to learn Hadoop and am trying to launch a hadoop cluster in Amazon EC2.  I
follow the steps listed at the following link:   https://github.com/ptgoetz/whirr-storm
  
 Step 1.  Install Apache Whirr.
 https://github.com/ptgoetz/whirr-storm
  
  
wget http://www.apache.org/dist/whirr/whirr-0.8.2/whirr-0.8.2.tar.gz
 
tar -zxf whirr-0.8.2.tar.gz
 
cd whirr-0.8.2

  
 Step 2.  Generate an SSH key for Whirr:
  
ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa_whirr

  
 Step 3.  Set up your Amazon EC2 credentials
  
mkdir ~/.whirr
 
echo "PROVIDER=aws-ec2" > ~/.whirr/credentials
 
echo "IDENTITY=[your EC2 Access Key ID]" >> ~/.whirr/credentials
 
echo "CREDENTIAL=[your EC2 Secret Access Key]" >> ~/.whirr/credentials
 
 
 Step 4. Set up the properties file for hadoop, specifying the properties files.
  
  
 Here are the error messages I have got. Does anyone has any idea what's going wrong here?
 
 
 
Running on provider aws-ec2 using identity ${env:AWS_ACCESS_KEY_ID}
Unable to start the cluster. Terminating all nodes.
com.google.common.util.concurrent.UncheckedExecutionException: com.google.inject.CreationException:
Guice creation errors:
 
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used
as a key; It is not fully specified.
 
1 error
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
        at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
        at org.apache.whirr.ClusterController.bootstrapCluster(ClusterController.java:137)
        at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:113)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: com.google.inject.CreationException: Guice creation errors:
 
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used
as a key; It is not fully specified.
 
1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
        at com.google.inject.Guice.createInjector(Guice.java:95)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
        at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
        at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
        ... 13 more
Unable to load cluster state, assuming it has no running nodes.
java.io.FileNotFoundException: /home/ec2-user/.whirr/hadoop-yarn/instances (No such file or
directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at com.google.common.io.Files$1.getInput(Files.java:109)
        at com.google.common.io.Files$1.getInput(Files.java:106)
        at com.google.common.io.CharStreams$2.getInput(CharStreams.java:93)
        at com.google.common.io.CharStreams$2.getInput(CharStreams.java:90)
        at com.google.common.io.CharStreams.readLines(CharStreams.java:307)
        at com.google.common.io.Files.readLines(Files.java:543)
        at org.apache.whirr.state.FileClusterStateStore.load(FileClusterStateStore.java:55)
        at org.apache.whirr.state.ClusterStateStore.tryLoadOrEmpty(ClusterStateStore.java:59)
        at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
        at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Exception in thread "main" com.google.common.util.concurrent.UncheckedExecutionException:
com.google.inject.CreationException: Guice creation errors:
 
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used
as a key; It is not fully specified.
 
1 error
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2258)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3990)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3994)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4878)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4884)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:88)
        at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:80)
        at org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:110)
        at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:233)
        at org.apache.whirr.ClusterController.destroyCluster(ClusterController.java:226)
        at org.apache.whirr.ClusterController.launchCluster(ClusterController.java:121)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:69)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:59)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)
Caused by: com.google.inject.CreationException: Guice creation errors:
 
1) org.jclouds.rest.RestContext<org.jclouds.aws.ec2.AWSEC2Client, A> cannot be used
as a key; It is not fully specified.
 
1 error
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:435)
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:154)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:106)
        at com.google.inject.Guice.createInjector(Guice.java:95)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:401)
        at org.jclouds.ContextBuilder.buildInjector(ContextBuilder.java:325)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:600)
        at org.jclouds.ContextBuilder.buildView(ContextBuilder.java:580)
        at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:119)
        at org.apache.whirr.service.ComputeCache$1.load(ComputeCache.java:98)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3589)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2374)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2337)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2252)
        ... 14 more
[ec2-user@ip-172-31-28-246 whirr-0.8.2]$

 
 

 After I setup the above mentioned items, here are the status of the macine:
 [ec2-user@ip-172-31-28-246 whirr-0.8.2]$ bin/whirr version
Apache Whirr 0.8.2
jclouds 1.5.8
  
  
 I am trying to setup a cluster with only node. I would really appreciate if anybody can give
me a sample recipe files that works with current EC2 machine.
  
 Thanks in advance!
 Max
Mime
View raw message