whirr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun Ramakrishnan <sinchronized.a...@gmail.com>
Subject Re: issues upgrading to 0.6 from 0.5
Date Mon, 10 Oct 2011 22:24:24 GMT
Thanks Andrei. That fixed all the issues and I am able to launch a hadoop
core cluster.
I do wish maven would do more than nearest first resolution ( at least some
warnings ) and not relegate ensuing issues to the user to figure out.

Arun

On Sat, Oct 8, 2011 at 2:16 AM, Andrei Savu <savu.andrei@gmail.com> wrote:

> I think that if you are upgrading jclouds you also need to upgrade the
> guava version in Whirr (root pom.xml) to match the one used by jclouds.
>
> -- Andrei Savu
>
> On Sat, Oct 8, 2011 at 1:25 AM, Arun Ramakrishnan <
> sinchronized.arun@gmail.com> wrote:
>
>> yup,  I did a "mvn clean install assembly:assembly" on a  fresh checkout
>> with it pointing to a current build of jclouds 1.2.0-SNAPSHOT and
>> jsr305-1.3.9 added as a dependency to whirr-core.
>> And I get the same errors while using the command line.
>>
>> thanks
>> Arun
>>
>>
>> On Fri, Oct 7, 2011 at 5:29 AM, Andrei Savu <savu.andrei@gmail.com>wrote:
>>
>>> Looks like a guava classpath ordering issue. Are you seeing this when
>>> building a clean checkout?
>>> On Oct 7, 2011 2:55 AM, "Arun Ramakrishnan" <sinchronized.arun@gmail.com>
>>> wrote:
>>>
>>>> Alas! While trying to run from the command line , I run into the
>>>> following issue. Any ideas ?
>>>>
>>>> ########################
>>>> whirr-0.7.0-SNAPSHOT>bin/whirr launch-cluster --config
>>>> ../whirr_cdh.properties
>>>> Exception in thread "main"
>>>> com.google.common.collect.ComputationException:
>>>> java.lang.NoClassDefFoundError: com/google/common/cache/CacheLoader
>>>>     at
>>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:325)
>>>>     at
>>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingSegment.compute(ComputingConcurrentHashMap.java:140)
>>>>     at
>>>> com.google.common.collect.ComputingConcurrentHashMap.apply(ComputingConcurrentHashMap.java:71)
>>>>     at
>>>> com.google.common.collect.MapMaker$ComputingMapAdapter.get(MapMaker.java:848)
>>>>     at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:66)
>>>>     at org.apache.whirr.service.ComputeCache.apply(ComputeCache.java:57)
>>>>     at
>>>> org.apache.whirr.actions.ScriptBasedClusterAction.execute(ScriptBasedClusterAction.java:62)
>>>>     at
>>>> org.apache.whirr.ClusterController.launchCluster(ClusterController.java:106)
>>>>     at
>>>> org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:62)
>>>>     at org.apache.whirr.cli.Main.run(Main.java:64)
>>>>     at org.apache.whirr.cli.Main.main(Main.java:97)
>>>> Caused by: java.lang.NoClassDefFoundError:
>>>> com/google/common/cache/CacheLoader
>>>>     at
>>>> org.jclouds.aws.ec2.AWSEC2ContextBuilder.addClientModule(AWSEC2ContextBuilder.java:42)
>>>>     at
>>>> org.jclouds.rest.RestContextBuilder.addClientModuleIfNotPresent(RestContextBuilder.java:267)
>>>>     at
>>>> org.jclouds.rest.RestContextBuilder.buildInjector(RestContextBuilder.java:179)
>>>>     at
>>>> org.jclouds.compute.ComputeServiceContextBuilder.buildInjector(ComputeServiceContextBuilder.java:52)
>>>>     at
>>>> org.jclouds.compute.ComputeServiceContextBuilder.buildComputeServiceContext(ComputeServiceContextBuilder.java:65)
>>>>     at
>>>> org.jclouds.compute.ComputeServiceContextFactory.buildContextUnwrappingExceptions(ComputeServiceContextFactory.java:70)
>>>>     at
>>>> org.jclouds.compute.ComputeServiceContextFactory.createContext(ComputeServiceContextFactory.java:121)
>>>>     at
>>>> org.apache.whirr.service.ComputeCache$1.apply(ComputeCache.java:82)
>>>>     at
>>>> org.apache.whirr.service.ComputeCache$1.apply(ComputeCache.java:72)
>>>>     at
>>>> com.google.common.collect.ComputingConcurrentHashMap$ComputingValueReference.compute(ComputingConcurrentHashMap.java:316)
>>>>     ... 10 more
>>>> Caused by: java.lang.ClassNotFoundException:
>>>> com.google.common.cache.CacheLoader
>>>>     at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
>>>>     at java.security.AccessController.doPrivileged(Native Method)
>>>>     at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
>>>>     at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
>>>>     at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
>>>>     ... 20 more
>>>> ###########################################################
>>>>
>>>> On Thu, Oct 6, 2011 at 3:51 PM, Arun Ramakrishnan <
>>>> sinchronized.arun@gmail.com> wrote:
>>>>
>>>>> Hurray! Then I included jsr305 in whirr-core and everything builds
>>>>> fine.
>>>>>
>>>>> While following your suggestion, I deleted my ~/.m2 directory and did
a
>>>>> mvn clean install. Is that what you meant by clean the cache ? . I still
had
>>>>> the same problem.
>>>>>
>>>>> here is some info about my environment.
>>>>> #######################################
>>>>> /Users/arun/workspace/whirr-trunk>mvn --version
>>>>> Apache Maven 3.0.3 (r1075438; 2011-02-28 09:31:09-0800)
>>>>> Maven home: /usr/share/maven
>>>>> Java version: 1.6.0_26, vendor: Apple Inc.
>>>>> Java home:
>>>>> /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
>>>>> Default locale: en_US, platform encoding: MacRoman
>>>>> OS name: "mac os x", version: "10.7.1", arch: "x86_64", family: "mac"
>>>>> #######################################
>>>>>
>>>>>
>>>>> thanks
>>>>> Arun
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Oct 6, 2011 at 1:41 PM, Andrei Savu <savu.andrei@gmail.com>wrote:
>>>>>
>>>>>> It seems like this is somehow related to Guava.
>>>>>>
>>>>>> http://stackoverflow.com/questions/3800033/guava-r07-gwt-and-javax-annotation-nullable
>>>>>>
>>>>>> Can you retry to build Whirr after removing the maven cache? I'm
not
>>>>>> sure this is the problem you are seeing.
>>>>>>
>>>>>> -- Andrei Savu
>>>>>>
>>>>>> On Thu, Oct 6, 2011 at 11:40 PM, Andrei Savu <savu.andrei@gmail.com>wrote:
>>>>>>
>>>>>>> I think I've seen this in the past on a different project. Can
you
>>>>>>> tell me more about os, IDE, jdk version and maven version?
>>>>>>>
>>>>>>> -- Andrei Savu
>>>>>>>
>>>>>>> On Thu, Oct 6, 2011 at 10:21 PM, Arun Ramakrishnan <
>>>>>>> sinchronized.arun@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> I applied the patch to jclouds trunk and I ran into this
problem. I
>>>>>>>> didn't spend too much time investigating it. But, let me
know if you an idea
>>>>>>>> whats wrong.
>>>>>>>>
>>>>>>>> #############################################################
>>>>>>>> [INFO] Apache Whirr Core .................................
FAILURE
>>>>>>>> [7.074s]
>>>>>>>>
>>>>>>>> ...
>>>>>>>>
>>>>>>>> [ERROR] Failed to execute goal
>>>>>>>> org.apache.maven.plugins:maven-compiler-plugin:2.0.2:compile
>>>>>>>> (default-compile) on project whirr-core: Compilation failure:
Compilation
>>>>>>>> failure:
>>>>>>>> [ERROR]
>>>>>>>> /Users/arun/workspace/whirr-trunk/core/src/main/java/org/apache/whirr/RolePredicates.java:[29,23]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol  : class Nullable
>>>>>>>> [ERROR] location: package javax.annotation
>>>>>>>> [ERROR]
>>>>>>>> [ERROR]
>>>>>>>> /Users/arun/workspace/whirr-trunk/core/src/main/java/org/apache/whirr/RolePredicates.java:[108,28]
>>>>>>>> cannot find symbol
>>>>>>>> [ERROR] symbol: class Nullable
>>>>>>>> [ERROR] public boolean apply(@Nullable Instance input) {
>>>>>>>> [ERROR] -> [Help 1]
>>>>>>>> ###########################################################
>>>>>>>>
>>>>>>>> thanks
>>>>>>>> Arun
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 4, 2011 at 10:45 PM, Andrei Savu <savu.andrei@gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> mvn package assembly:assembly will generate the release
tarball
>>>>>>>>> from the source code.
>>>>>>>>>
>>>>>>>>> -- Andrei
>>>>>>>>>
>>>>>>>>> On Wed, Oct 5, 2011 at 4:07 AM, Arun Ramakrishnan <
>>>>>>>>> sinchronized.arun@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Its always been less than 10 security groups in total.
I will give
>>>>>>>>>> this a try however.
>>>>>>>>>> I am able to "mvn install" successfully in whirr-trunk.
Is there a
>>>>>>>>>> maven goal or a script that does the final packaging
for command line use ?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Oct 4, 2011 at 2:08 PM, Tom White <tom@cloudera.com>wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Arun,
>>>>>>>>>>>
>>>>>>>>>>> I had this issue too - I think it is a consequence
of using an
>>>>>>>>>>> account
>>>>>>>>>>> with a large number of existing groups. There's
a fix in
>>>>>>>>>>> https://issues.apache.org/jira/browse/WHIRR-381
that you might
>>>>>>>>>>> like to
>>>>>>>>>>> try - this will be fixed in the next release.
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Tom
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Oct 4, 2011 at 1:30 PM, Arun Ramakrishnan
>>>>>>>>>>> <sinchronized.arun@gmail.com> wrote:
>>>>>>>>>>> > I am having this issue upgrading to 0.6.
>>>>>>>>>>> >
>>>>>>>>>>> > During the security group creation, the
internal access rules
>>>>>>>>>>> within the
>>>>>>>>>>> > cluster don't get added, consequentially
blocking the datanodes
>>>>>>>>>>> from being
>>>>>>>>>>> > able to connect to the namenode. I added
these rules manually
>>>>>>>>>>> and it seems
>>>>>>>>>>> > fine.
>>>>>>>>>>> >
>>>>>>>>>>> > This bug recurs for me consistently in my
environment.
>>>>>>>>>>> >
>>>>>>>>>>> > log
>>>>>>>>>>> > ########################################################
>>>>>>>>>>> >
>>>>>>>>>>> > 2011-10-04 12:23:20,949 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) >>
>>>>>>>>>>> > creating keyPair region(us-east-1) group(ccore27)
>>>>>>>>>>> > 2011-10-04 12:23:20,949 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-2) >>
>>>>>>>>>>> > creating keyPair region(us-east-1) group(ccore27)
>>>>>>>>>>> > 2011-10-04 12:23:21,414 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) << created
>>>>>>>>>>> > keyPair(jclouds#ccore27#us-east-1#72)
>>>>>>>>>>> > 2011-10-04 12:23:21,414 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) >>
>>>>>>>>>>> > creating securityGroup region(us-east-1)
>>>>>>>>>>> name(jclouds#ccore27#us-east-1)
>>>>>>>>>>> > 2011-10-04 12:23:21,692 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) << created
>>>>>>>>>>> > securityGroup(jclouds#ccore27#us-east-1)
>>>>>>>>>>> > 2011-10-04 12:23:21,692 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) >>
>>>>>>>>>>> > authorizing securityGroup region(us-east-1)
>>>>>>>>>>> name(jclouds#ccore27#us-east-1)
>>>>>>>>>>> > port(22)
>>>>>>>>>>> > 2011-10-04 12:23:21,926 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) <<
>>>>>>>>>>> > authorized securityGroup(jclouds#ccore27#us-east-1)
>>>>>>>>>>> > 2011-10-04 12:23:21,926 DEBUG [jclouds.compute]
>>>>>>>>>>> (pool-3-thread-4) >>
>>>>>>>>>>> > authorizing securityGroup region(us-east-1)
>>>>>>>>>>> name(jclouds#ccore27#us-east-1)
>>>>>>>>>>> > permission to itself
>>>>>>>>>>> > 2011-10-04 12:23:22,306 ERROR
>>>>>>>>>>> > [org.apache.whirr.actions.BootstrapClusterAction]
>>>>>>>>>>> (pool-3-thread-3)
>>>>>>>>>>> > Unexpected error while starting 4 nodes,
minimum 4 nodes for
>>>>>>>>>>> > [hadoop-datanode, hadoop-tasktracker] of
cluster ccore27
>>>>>>>>>>> > java.util.concurrent.ExecutionException:
>>>>>>>>>>> java.lang.RuntimeException:
>>>>>>>>>>> > request: POST https://ec2.us-east-1.amazonaws.com/
HTTP/1.1;
>>>>>>>>>>> cause:
>>>>>>>>>>> > java.lang.NullPointerException
>>>>>>>>>>> >     at
>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>>>>>>>>>> >     at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.waitForOutcomes(BootstrapClusterAction.java:320)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:273)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> org.apache.whirr.actions.BootstrapClusterAction$StartupProcess.call(BootstrapClusterAction.java:234)
>>>>>>>>>>> >     at
>>>>>>>>>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>>>>>>>>>>> >     at java.util.concurrent.FutureTask.run(FutureTask.java:138)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> >     at java.lang.Thread.run(Thread.java:680)
>>>>>>>>>>> > Caused by: java.lang.RuntimeException: request:
POST
>>>>>>>>>>> > https://ec2.us-east-1.amazonaws.com/ HTTP/1.1;
cause:
>>>>>>>>>>> > java.lang.NullPointerException
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:152)
>>>>>>>>>>> >     at
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.parse(ParseSax.java:116)
>>>>>>>>>>> >     at
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:78)
>>>>>>>>>>> >     at
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:51)
>>>>>>>>>>> >     at
>>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:439)
>>>>>>>>>>> >     at
>>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:437)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:713)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>>>>>> >     at java.lang.Thread.run(Thread.java:680)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.addDetailsAndPropagate(ParseSax.java:152)
>>>>>>>>>>> >     at
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:80)
>>>>>>>>>>> >     at
>>>>>>>>>>> org.jclouds.http.functions.ParseSax.apply(ParseSax.java:51)
>>>>>>>>>>> >     at
>>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:439)
>>>>>>>>>>> >     at
>>>>>>>>>>> com.google.common.util.concurrent.Futures$4.apply(Futures.java:437)
>>>>>>>>>>> >     at
>>>>>>>>>>> >
>>>>>>>>>>> com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:713)
>>>>>>>>>>> >     ... 3 more
>>>>>>>>>>> >
>>>>>>>>>>> > ###########################################################
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > This are the rules that existed for the
cluster created with
>>>>>>>>>>> whirr 0.6
>>>>>>>>>>> > GROUP    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> > jclouds#ccore27#us-east-1
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    tcp
>>>>>>>>>>> > 22    22    FROM    CIDR    0.0.0.0/0
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    tcp
>>>>>>>>>>> > 8020    8020    FROM    CIDR    184.72.183.32/32
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    tcp
>>>>>>>>>>> > 8021    8021    FROM    CIDR    184.72.183.32/32
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    tcp
>>>>>>>>>>> > 50030    50030    FROM    CIDR    24.43.39.218/32
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    tcp
>>>>>>>>>>> > 50070    50070    FROM    CIDR    24.43.39.218/32
>>>>>>>>>>> >
>>>>>>>>>>> > I had to manually add
>>>>>>>>>>> > PERMISSION    673040621396    jclouds#ccore27#us-east-1
>>>>>>>>>>> ALLOWS    all
>>>>>>>>>>> >         FROM    USER    673040621396   
GRPNAME
>>>>>>>>>>> jclouds#ccore27#us-east-1
>>>>>>>>>>> > to allow the datanodes to talk to the namenode.
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > Here is my config file with updated property
names to match 0.6
>>>>>>>>>>> > ##########################################
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > whirr.cluster-name=ccore27
>>>>>>>>>>> > whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,4
>>>>>>>>>>> > hadoop-datanode+hadoop-tasktracker
>>>>>>>>>>> >
>>>>>>>>>>> > whirr.provider=aws-ec2
>>>>>>>>>>> > whirr.identity=*************************
>>>>>>>>>>> > whirr.credential=**********************
>>>>>>>>>>> > whirr.private-key-file=/Users/arun/.ec2/hadoopkey
>>>>>>>>>>> > whirr.public-key-file=/Users/arun/.ec2/hadoopkey.pub
>>>>>>>>>>> > whirr.client-cidrs=24.43.39.218/32
>>>>>>>>>>> >
>>>>>>>>>>> > whirr.location-id=us-east-1
>>>>>>>>>>> > whirr.hardware-id=c1.xlarge
>>>>>>>>>>> > #c1.xlarge
>>>>>>>>>>> >
>>>>>>>>>>> > # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
or
>>>>>>>>>>> > http://aws.amazon.com/amis/4348
>>>>>>>>>>> > # ebs root only
>>>>>>>>>>> > whirr.image-id=us-east-1/ami-4a0df923
>>>>>>>>>>> >
>>>>>>>>>>> > whirr.hadoop.install-function=install_cdh_hadoop
>>>>>>>>>>> > whirr.hadoop.configure-function=configure_cdh_hadoop
>>>>>>>>>>> >
>>>>>>>>>>> > #######################################
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>

Mime
View raw message