jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yaron Rosenbaum (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (JCLOUDS-767) Failed spot request but still created, node is null - unable to cancel
Date Thu, 20 Nov 2014 07:14:33 GMT

    [ https://issues.apache.org/jira/browse/JCLOUDS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14219079#comment-14219079
] 

Yaron Rosenbaum commented on JCLOUDS-767:
-----------------------------------------

Hi

After further research with Amazon, it looks like jClouds is doing some sort of POLLING on
spot requests, to check if they have been fulfilled and/or the instance has been created.
It looks like, this polling happens too frequently for Amazon, and they perceive this as some
sort of attack, and therefore send the error I have reported.

I would like to suggest:
1) Increasing the polling interval,
2) Sending a new polling request only when the previous one has returned (or a long timeout
has ben breached)
3) Possibly allowing this to be a configurable parameter

At the moment, the result of this behaviour is that no more than one concurrent spot request
may be handled by jClouds.
Note, that I am requesting a number of spots on different threads, and not issuing one request
for multiple spots.


Here's what Amazon support say:


"This error comes from querying bid status (e.g., "ec2-describe-spot-instance-requests") too
frequently. In this case, you are calling this API before a previous call returned (perhaps
calling from multiple clients in a tight loop). While that's supported by many AWS APIs, we've
had problems with brown-outs because of it and need to protect the service. We believe that
most cases where a customer is throttled will be avoided by simply waiting for a response
before calling again (sleeping for 1 second between response received and next request would
avoid throttling for most Spot customers). The underlying bid data changes slowly, so there
little to be gained by calling more than twice a minute in any case.
"



> Failed spot request but still created, node is null - unable to cancel
> ----------------------------------------------------------------------
>
>                 Key: JCLOUDS-767
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-767
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-compute
>    Affects Versions: 1.8.1
>         Environment: AWS, Spot instances, CoreOS
>            Reporter: Yaron Rosenbaum
>            Priority: Critical
>
> My program creates spot instances (see 'addNode' below).
> Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).
> The problem is that JClouds throws an exception, but the spot request IS created and
executed by AWS. The node is never properly configured. Thus, I have no way to cancel the
spot request (there's no node ID).
> Correct behaviour should be: Failure should leave no active spot requests.
> At the very least, there should be a way to cancel the spot request if it fails.
> Also, it does seem like it shouldn't have failed - since the spot request is successful...
> 	public ComputeMetadata addNode(Template template) throws RunNodesException {
> 		ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(),
1, template));
> 		LOG.info("Node \"{}\" created", result.getName());
> 		if (isLoadBalancerLogicEnabled()) {
> 			LOG.info("Adding node \"{}\" to load balancer", result.getName());
> 			addNodeToLoadBalancer(result);
> 		}
> 10:05:43.966 [main] INFO  io.multicloud.controller.AWSProvider - Successfully created
compute context
> 10:06:02.795 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:06:02.795 [main] INFO  io.multicloud.controller.AWSProvider - quorum is missing 3
nodes
> 10:06:03.077 [main] INFO  i.m.controller.CloudProviderImpl - Adding 3 nodes
> 10:06:04.113 [pool-3-thread-1] WARN  jclouds.compute - << not api visible instances([{region=us-east-1,
name=sir-02gebgm2}])
> 10:06:04.342 [pool-3-thread-3] WARN  jclouds.compute - << not api visible instances([{region=us-east-1,
name=sir-02gez2h5}])
> 10:06:04.460 [pool-3-thread-2] WARN  jclouds.compute - << not api visible instances([{region=us-east-1,
name=sir-02ghn9t2}])
> 10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5):

> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/
HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you
requested and frequency of requests together exceed the levels allowed by the Spot service',
context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create
node: 
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1)
image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gez2h5:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/
HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef',
requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you
requested and frequency of requests together exceed the levels allowed by the Spot service',
context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> 	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
> 	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> 	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2):

> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/
HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you
requested and frequency of requests together exceed the levels allowed by the Spot service',
context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
~[sts-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
~[jclouds-core-1.8.1.jar:1.8.1]
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
~[guava-17.0.jar:na]
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
~[aws-ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
~[ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
[jclouds-compute-1.8.1.jar:1.8.1]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create
node: 
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1)
image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b,
subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gebgm2:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/
HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645',
requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you
requested and frequency of requests together exceed the levels allowed by the Spot service',
context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> 	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222)
~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148)
~[ec2-1.8.1.jar:1.8.1]
> 	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> 	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227)
~[classes/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66"
created
> 10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66"
to load balancer
> 10:09:25.110 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Creating node
using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0,
location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0,
is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root,
userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}},
hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750,
volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL,
size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc,
bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())},
location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]},
options={userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616,
spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
> 10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Name: AWS-East-1-8b5f5f66,
User metadata: {TYPE=SPOT, Name=AWS-East-1-8b5f5f66}
> 10:09:39.106 [main] INFO  io.multicloud.controller.AWSProvider - Optimizing availability:
Adding 1 nodes
> 10:09:39.106 [main] INFO  i.m.controller.CloudProviderImpl - Adding 1 nodes



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message