jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Gaul (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCLOUDS-701) createNodesInGroup fails with IllegalStateException if we hit CPU quota on GCE
Date Tue, 27 Jan 2015 05:12:35 GMT

     [ https://issues.apache.org/jira/browse/JCLOUDS-701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Gaul updated JCLOUDS-701:
--------------------------------
    Component/s: jclouds-compute

> createNodesInGroup fails with IllegalStateException if we hit CPU quota on GCE
> ------------------------------------------------------------------------------
>
>                 Key: JCLOUDS-701
>                 URL: https://issues.apache.org/jira/browse/JCLOUDS-701
>             Project: jclouds
>          Issue Type: Bug
>          Components: jclouds-compute, jclouds-labs-google
>    Affects Versions: 1.8.0
>            Reporter: Sunil Shah
>
> When we call createNodesInGroup for a Google Compute Engine project that has come up
against its CPU quota, we see this exception:
> {code}
> 2014-09-02 15:55:03,640 ERROR [pool-3-thread-4] j.compute [SLF4JLogger.java:93] createNodesInGroup(nothing),
completed: 0/2, errors: 1, rate: 6586ms/op
> java.util.concurrent.ExecutionException: java.lang.IllegalStateException: operation failed.
Http Error Code: 403 HttpError: FORBIDDEN
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-17.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-17.0.jar:na]
> 	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-17.0.jar:na]
> 	at org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123) ~[jclouds-core-1.8.0.jar:1.8.0]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> Caused by: java.lang.IllegalStateException: operation failed. Http Error Code: 403 HttpError:
FORBIDDEN
> 	at org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.waitOperationDone(GoogleComputeEngineServiceAdapter.java:434)
~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> 	at org.jclouds.googlecomputeengine.compute.GoogleComputeEngineServiceAdapter.createNodeWithGroupEncodedIntoName(GoogleComputeEngineServiceAdapter.java:185)
~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> 	at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_60]
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_60]
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_60]
> 	at java.lang.Thread.run(Thread.java:745) [na:1.7.0_60]
> 	at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_60]
> 	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:127)
~[google-compute-engine-1.8.1-20140814.210916-6.jar:1.8.1-SNAPSHOT]
> 	at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210)
~[jclouds-compute-1.8.0.jar:1.8.0]
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_60]
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_60]
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_60]
> 	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_60]
> 	at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[guice-3.0.jar:na]
> 	at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
> 	at mesosphere.poseidon.providers.Google.createNodesFromTemplate(Google.scala:58) ~[classes/:na]
> {code}
> This is inconsistent with the error we get when the network quota is exceeded, which
is:
> {code}
> 2014-08-07 22:01:37,109 WARN [ApiWorker-2] m.p.p.Google [Provider.scala:142] development-24-88d79664.dcc.io
// Retrying after:
> com.google.common.util.concurrent.UncheckedExecutionException: java.lang.IllegalStateException:
Could not create network, operation failedOperation{kind=compute#operation, id=3290207288552397268,
name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24,
status=DONE, user=tobi@mesosphere.io, progress=100, insertTime=Thu Aug 07 22:01:35 UTC 2014,
startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07 22:01:36 UTC 2014, httpError={statusCode=403,
message=FORBIDDEN, headers={}}, operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota
'NETWORKS' exceeded.  Limit: 5.0}]}
>     at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2199) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache.get(LocalCache.java:3932) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3936) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4806)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4812)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4830)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.getOrCreateNetwork(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:137)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.compute.strategy.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:215)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_55]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
~[na:1.7.0_55]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_55]
>     at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_55]
>     at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.sun.proxy.$Proxy124.createNodesInGroup(Unknown Source) ~[na:na]
>     at mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:58)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at mesosphere.poseidon.providers.Google$$anonfun$1$$anonfun$apply$1.apply(Google.scala:55)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) [poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_55]
>     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_55]
>     at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]
> Caused by: java.lang.IllegalStateException: Could not create network, operation failedOperation{kind=compute#operation,
id=3290207288552397268, name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, selfLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/operations/operation-1407448895523-500113b7508b8-d2ab879d-4773f860,
name=operation-1407448895523-500113b7508b8-d2ab879d-4773f860, targetLink=https://www.googleapis.com/compute/v1/projects/modern-saga-648/global/networks/jclouds-development-24,
status=DONE, user=tobi@mesosphere.io, progress=100, insertTime=Thu Aug 07 22:01:35 UTC 2014,
startTime=Thu Aug 07 22:01:35 UTC 2014, endTime=Thu Aug 07 22:01:36 UTC 2014, httpError={statusCode=403,
message=FORBIDDEN, headers={}}, operationType=insert, errors=[Error{code=QUOTA_EXCEEDED, message=Quota
'NETWORKS' exceeded.  Limit: 5.0}]}
>     at com.google.common.base.Preconditions.checkState(Preconditions.java:150) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:95)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.functions.CreateNetworkIfNeeded.apply(CreateNetworkIfNeeded.java:46)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:59)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at org.jclouds.googlecomputeengine.compute.loaders.FindNetworkOrCreate.load(FindNetworkOrCreate.java:36)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3522)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2315) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2278)
~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2193) ~[poseidon-0.2-SNAPSHOT-jar-with-dependencies.jar:na]
>     ... 21 common frames omitted
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message