jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yaron Rosenbaum (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-767) Creating multiple spot instances fails. Status polling too frequent?
Date Thu, 20 Nov 2014 08:36:34 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yaron Rosenbaum updated JCLOUDS-767:
------------------------------------
    Description: 
My program creates spot instances (see 'addNode' below).
Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).

I am creating multiple nodes on multiple threads (3).
It appears that jClouds is polling node status too frequently, and AWS perceives this as a DOS attack. AWS polling queries fail on "The amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service".

jClouds interprets this as a failure to create, whereas this is just a failure to poll.

The spot requests, however, are valid - and eventually fulfilled, but because of this error - they are never properly configured by jClouds, and in fact - are just left as orphans.

This scenario, in turn, ends up creating more and more nodes - endlessly, all left as orphans (but costing dear $).

So,
1) This specific error should not be interpreted as failure
2) The polling interval should be increased, or at least configurable
3) Maybe some sort of exponential backoff should be used in case this error happens
4) Failed spot requests should verify that the spot request is cancelled

(the code below happens on 3 threads, concurrently)
	public ComputeMetadata addNode(Template template) throws RunNodesException {
		ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(), 1, template));
		LOG.info("Node \"{}\" created", result.getName());

10:06:02.795 [main] INFO  io.multicloud.controller.AWSProvider - quorum is missing 3 nodes
10:06:03.077 [main] INFO  i.m.controller.CloudProviderImpl - Adding 3 nodes
10:06:04.113 [pool-3-thread-1] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gebgm2}])
10:06:04.342 [pool-3-thread-3] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gez2h5}])
10:06:04.460 [pool-3-thread-2] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02ghn9t2}])
10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5): 
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:

0 error[s]
Node failures:

1) AWSResponseException on node us-east-1/sir-02gez2h5:
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


1 error[s]
	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2): 
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:

0 error[s]
Node failures:

1) AWSResponseException on node us-east-1/sir-02gebgm2:
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


1 error[s]
	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66" created
10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66" to load balancer
10:09:25.110 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Creating node using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750, volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}



  was:
My program creates spot instances (see 'addNode' below).
Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).

The problem is that JClouds throws an exception, but the spot request IS created and executed by AWS. The node is never properly configured. Thus, I have no way to cancel the spot request (there's no node ID).

Correct behaviour should be: Failure should leave no active spot requests.
At the very least, there should be a way to cancel the spot request if it fails.

Also, it does seem like it shouldn't have failed - since the spot request is successful...




	public ComputeMetadata addNode(Template template) throws RunNodesException {
		ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(), 1, template));
		LOG.info("Node \"{}\" created", result.getName());
		if (isLoadBalancerLogicEnabled()) {
			LOG.info("Adding node \"{}\" to load balancer", result.getName());
			addNodeToLoadBalancer(result);
		}


10:05:43.966 [main] INFO  io.multicloud.controller.AWSProvider - Successfully created compute context
10:06:02.795 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
10:06:02.795 [main] INFO  io.multicloud.controller.AWSProvider - quorum is missing 3 nodes
10:06:03.077 [main] INFO  i.m.controller.CloudProviderImpl - Adding 3 nodes
10:06:04.113 [pool-3-thread-1] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gebgm2}])
10:06:04.342 [pool-3-thread-3] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gez2h5}])
10:06:04.460 [pool-3-thread-2] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02ghn9t2}])
10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5): 
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:

0 error[s]
Node failures:

1) AWSResponseException on node us-east-1/sir-02gez2h5:
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


1 error[s]
	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2): 
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
Execution failures:

0 error[s]
Node failures:

1) AWSResponseException on node us-east-1/sir-02gebgm2:
org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)


1 error[s]
	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66" created
10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66" to load balancer
10:09:25.110 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Creating node using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750, volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}
10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Current nodes:
10:09:39.105 [main] DEBUG i.m.controller.CloudProviderImpl - Name: AWS-East-1-8b5f5f66, User metadata: {TYPE=SPOT, Name=AWS-East-1-8b5f5f66}
10:09:39.106 [main] INFO  io.multicloud.controller.AWSProvider - Optimizing availability: Adding 1 nodes
10:09:39.106 [main] INFO  i.m.controller.CloudProviderImpl - Adding 1 nodes




> Creating multiple spot instances fails. Status polling too frequent?
> --------------------------------------------------------------------
>
>                 Key: JCLOUDS-767
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-767
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-compute
>    Affects Versions: 1.8.1
>         Environment: AWS, Spot instances, CoreOS
>            Reporter: Yaron Rosenbaum
>            Priority: Blocker
>              Labels: AWS, DOS, Spot
>
> My program creates spot instances (see 'addNode' below).
> Recently, in some cases I've been getting "RequestResourceCountExceeded" error (see below).
> I am creating multiple nodes on multiple threads (3).
> It appears that jClouds is polling node status too frequently, and AWS perceives this as a DOS attack. AWS polling queries fail on "The amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service".
> jClouds interprets this as a failure to create, whereas this is just a failure to poll.
> The spot requests, however, are valid - and eventually fulfilled, but because of this error - they are never properly configured by jClouds, and in fact - are just left as orphans.
> This scenario, in turn, ends up creating more and more nodes - endlessly, all left as orphans (but costing dear $).
> So,
> 1) This specific error should not be interpreted as failure
> 2) The polling interval should be increased, or at least configurable
> 3) Maybe some sort of exponential backoff should be used in case this error happens
> 4) Failed spot requests should verify that the spot request is cancelled
> (the code below happens on 3 threads, concurrently)
> 	public ComputeMetadata addNode(Template template) throws RunNodesException {
> 		ComputeMetadata result = Iterables.getOnlyElement(getComputeService().createNodesInGroup(getGroupName(), 1, template));
> 		LOG.info("Node \"{}\" created", result.getName());
> 10:06:02.795 [main] INFO  io.multicloud.controller.AWSProvider - quorum is missing 3 nodes
> 10:06:03.077 [main] INFO  i.m.controller.CloudProviderImpl - Adding 3 nodes
> 10:06:04.113 [pool-3-thread-1] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gebgm2}])
> 10:06:04.342 [pool-3-thread-3] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02gez2h5}])
> 10:06:04.460 [pool-3-thread-2] WARN  jclouds.compute - << not api visible instances([{region=us-east-1, name=sir-02ghn9t2}])
> 10:06:17.853 [user thread 1] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gez2h5): 
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:06:17.872 [pool-3-thread-3] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-4f5a433b, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gez2h5:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='c5ffce45-02c3-4443-b4c2-42cfe3e2a7ef', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> 	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
> 	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> 	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.218 [user thread 0] ERROR jclouds.compute - << problem customizing node(us-east-1/sir-02gebgm2): 
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75) ~[sts-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87) ~[guava-17.0.jar:na]
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source) ~[na:na]
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53) ~[aws-ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56) ~[ec2-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111) ~[jclouds-core-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117) [jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49) [jclouds-compute-1.8.1.jar:1.8.1]
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [na:1.7.0_09]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:07:47.220 [pool-3-thread-1] ERROR i.m.controller.CloudProviderImpl - Failed to create node: 
> org.jclouds.compute.RunNodesException: error running 1 node group(AWS-East-1) location(us-east-1) image(ami-06ff446e) size(m1.medium) options({userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-0afda622, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]})
> Execution failures:
> 0 error[s]
> Node failures:
> 1) AWSResponseException on node us-east-1/sir-02gebgm2:
> org.jclouds.aws.AWSResponseException: request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='82736b80-b101-4111-acd0-69e3ce53d645', requestToken='null', code='RequestResourceCountExceeded', message='the amount of data you requested and frequency of requests together exceed the levels allowed by the Spot service', context='{Response=, Errors=}'}
> 	at org.jclouds.aws.handlers.ParseAWSErrorFromXmlContent.handleError(ParseAWSErrorFromXmlContent.java:75)
> 	at org.jclouds.http.handlers.DelegatingErrorHandler.handleError(DelegatingErrorHandler.java:65)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.shouldContinue(BaseHttpCommandExecutorService.java:128)
> 	at org.jclouds.http.internal.BaseHttpCommandExecutorService.invoke(BaseHttpCommandExecutorService.java:98)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.invoke(InvokeHttpMethod.java:90)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:73)
> 	at org.jclouds.rest.internal.InvokeHttpMethod.apply(InvokeHttpMethod.java:44)
> 	at org.jclouds.reflect.FunctionalReflection$FunctionalInvocationHandler.handleInvocation(FunctionalReflection.java:117)
> 	at com.google.common.reflect.AbstractInvocationHandler.invoke(AbstractInvocationHandler.java:87)
> 	at $Proxy70.describeSpotInstanceRequestsInRegion(Unknown Source)
> 	at org.jclouds.aws.ec2.compute.strategy.AWSEC2GetNodeMetadataStrategy.getRunningInstanceInRegion(AWSEC2GetNodeMetadataStrategy.java:53)
> 	at org.jclouds.ec2.compute.strategy.EC2GetNodeMetadataStrategy.getNode(EC2GetNodeMetadataStrategy.java:56)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:58)
> 	at org.jclouds.compute.predicates.internal.RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.refreshOrNull(RefreshNodeAndDoubleCheckOnFailUnlessStatusInvalid.java:38)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:57)
> 	at org.jclouds.compute.predicates.internal.RefreshAndDoubleCheckOnFailUnlessStatusInvalid.apply(RefreshAndDoubleCheckOnFailUnlessStatusInvalid.java:40)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:142)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull$1.apply(ComputeServiceTimeoutsModule.java:137)
> 	at org.jclouds.util.Predicates2$RetryablePredicate.apply(Predicates2.java:111)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:154)
> 	at org.jclouds.compute.config.ComputeServiceTimeoutsModule$RetryablePredicateGuardingNull.apply(ComputeServiceTimeoutsModule.java:118)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:63)
> 	at org.jclouds.compute.functions.PollNodeRunning.apply(PollNodeRunning.java:42)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:117)
> 	at org.jclouds.compute.strategy.CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.call(CustomizeNodeAndAddToGoodMapOrPutExceptionIntoBadMap.java:49)
> 	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 	at java.lang.Thread.run(Thread.java:722)
> 1 error[s]
> 	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:222) ~[jclouds-compute-1.8.1.jar:1.8.1]
> 	at org.jclouds.ec2.compute.EC2ComputeService.createNodesInGroup(EC2ComputeService.java:148) ~[ec2-1.8.1.jar:1.8.1]
> 	at io.multicloud.controller.CloudProviderImpl.addNode(CloudProviderImpl.java:65) ~[classes/:na]
> 	at io.multicloud.controller.CloudProviderImpl$CreateNewNodeTask.run(CloudProviderImpl.java:227) ~[classes/:na]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_09]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_09]
> 	at java.lang.Thread.run(Thread.java:722) [na:1.7.0_09]
> 10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Node "AWS-East-1-8b5f5f66" created
> 10:09:24.051 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Adding node "AWS-East-1-8b5f5f66" to load balancer
> 10:09:25.110 [pool-3-thread-2] INFO  i.m.controller.CloudProviderImpl - Creating node using template: {image={id=us-east-1/ami-06ff446e, providerId=ami-06ff446e, name=CoreOS-beta-444.5.0, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, os={family=unrecognized, arch=paravirtual, version=, description=595879546273/CoreOS-beta-444.5.0, is64Bit=true}, description=CoreOS beta 444.5.0 (PV), status=AVAILABLE[available], loginUser=root, userMetadata={owner=595879546273, rootDeviceType=ebs, virtualizationType=paravirtual, hypervisor=xen}}, hardware={id=m1.medium, providerId=m1.medium, processors=[{cores=1.0, speed=2.0}], ram=3750, volumes=[{type=LOCAL, size=10.0, device=/dev/sda1, bootDevice=true, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdb, bootDevice=false, durable=false}, {type=LOCAL, size=420.0, device=/dev/sdc, bootDevice=false, durable=false}], supportsImage=Predicates.and(Predicates.alwaysTrue(),Predicates.or(isWindows(),requiresVirtualizationType(paravirtual)),Predicates.alwaysTrue(),Predicates.alwaysTrue())}, location={scope=REGION, id=us-east-1, description=us-east-1, parent=aws-ec2, iso3166Codes=[US-VA]}, options={userMetadata={TYPE=SPOT}, keyPair=Yaron, userDataCksum=c242210b, subnetId=subnet-50cdf616, spotPrice=0.013, groupIds=[sg-ce6512ab, sg-d9cb77bc]}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message