jclouds-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aled Sage (JIRA)" <j...@apache.org>
Subject [jira] [Created] (JCLOUDS-1283) IllegalArgumentException in OpenStack destroyNode, after failure to provision
Date Thu, 27 Apr 2017 21:20:04 GMT
Aled Sage created JCLOUDS-1283:
----------------------------------

             Summary: IllegalArgumentException in OpenStack destroyNode, after failure to
provision
                 Key: JCLOUDS-1283
                 URL: https://issues.apache.org/jira/browse/JCLOUDS-1283
             Project: jclouds
          Issue Type: Bug
          Components: jclouds-compute
    Affects Versions: 1.9.2
            Reporter: Aled Sage
            Priority: Trivial


As reported in https://issues.apache.org/jira/browse/BROOKLYN-295, with jclouds 1.9.2 (looking
at the line numbers, it looks like 2.0.1 would have the same behaviour).

I attempted to deploy an app to OpenStack. It failed to provision due to insufficient resources
(hence tricky to reproduce). But then jclouds tried and failed to destroy the VM.

It would be nice if jclouds handled the failure more gracefully. This exception hides the
message about the provisioning request failing.

The exception in jclouds was:

{noformat}
java.lang.IllegalArgumentException: id must be in format regionId/id
{noformat}

The provisioning error (shown in the OpenStack web-console) was:

{noformat}
Message  No valid host was found. Exceeded max scheduling attempts 3 for instance e74952b7-f495-4148-9a09-c4dee0d75e7a.
Last exception: [u'Traceback (most recent call last):\n', u' File "/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/c
Code     500
Details  File "/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py",
line 669, in build_instances instances[0].uuid) File "/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py",
line 172, in populate_retry raise exception.NoValidHost(reason=msg)
Created  June 8, 2016, 10:33 a.m.
{noformat}

The Brooklyn debug log shows:

{noformat}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation [brooklyn-execmanager-yA1Lh5GF-504]:
jclouds using template {image={id=RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77, providerId=e16a7bca-7363-45b1-bd6a-02479d4cea77,
name=CentOS 7, location={scope=REGION, id=RegionOne, description=RegionOne, parent=openstack-nova},
os={family=centos, name=CentOS 7, version=, description=CentOS 7, is64Bit=true}, description=CentOS
7, status=AVAILABLE, loginUser=root}, hardware={id=RegionOne/2, providerId=2, name=m1.small,
location={scope=REGION, id=RegionOne, description=RegionOne, parent=openstack-nova}, processors=[{cores=1.0,
speed=1.0}], ram=2048, volumes=[{type=LOCAL, size=20.0, bootDevice=true, durable=true}], supportsImage=ALWAYS_TRUE},
location={scope=REGION, id=RegionOne, description=RegionOne, parent=openstack-nova}, options={loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, brooklyn-user=aled,
brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, brooklyn-entity-id=rj29jfrfvy,
brooklyn-entity-name=swarm-node, brooklyn-server-creation-date=2016-06-08-1133}, networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f],
autoAssignFloatingIp=false, keyPairName=openstack, configDrive=false}} / options {loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, brooklyn-user=aled,
brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, brooklyn-entity-id=rj29jfrfvy,
brooklyn-entity-name=swarm-node, brooklyn-server-creation-date=2016-06-08-1133}, networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f],
autoAssignFloatingIp=false, keyPairName=openstack, configDrive=false} to provision machine
in openstack-nova:https:/acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}
2016-06-08 11:33:45,215 DEBUG o.a.b.l.j.JcloudsLocation [brooklyn-execmanager-yA1Lh5GF-504]:
NOTE: unused flags passed to obtain VM in openstack-nova:https://acme.com:5000/v2.0@VanillaSoftwareProcessImpl{id=rj29jfrfvy}:
{jclouds.keystone.credential-type=xxxxxxxx, machineCreationSemaphore=java.util.concurrent.Semaphore@78d9e87b[Permits
= 2147483643]}
2016-06-08 11:33:45,215 DEBUG jclouds.compute [brooklyn-execmanager-yA1Lh5GF-504]: >>
running 1 node group(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29) location(RegionOne)
image(RegionOne/e16a7bca-7363-45b1-bd6a-02479d4cea77) hardwareProfile(RegionOne/2) options({loginUser=centos,
loginPrivateKeyPresent=true, inboundPorts=[22, 3376, 2376], scriptPresent=true, securityGroups=[VPN_local],
userMetadata={Name=brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s, brooklyn-user=aled,
brooklyn-app-id=yfxfed97j6, brooklyn-app-name=Docker Swarm, brooklyn-entity-id=rj29jfrfvy,
brooklyn-entity-name=swarm-node, brooklyn-server-creation-date=2016-06-08-1133}, networks=[e472fd1f-01f8-493a-b284-b0ff0b0e305f],
autoAssignFloatingIp=false, keyPairName=openstack, configDrive=false})
2016-06-08 11:33:45,218 DEBUG o.j.l.s.f.RegionIdsFromConfiguration [brooklyn-execmanager-yA1Lh5GF-504]:
no jclouds.regions configured for provider openstack-nova

2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> GET https://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a
HTTP/1.1
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> Accept: application/json
2016-06-08 12:03:48,046 DEBUG jclouds.headers [user thread 8]: >> X-Auth-Token: 50f7eaf091ea4ca494bbfc591d7f61a7
2016-06-08 12:03:48,855 DEBUG o.j.h.i.JavaUrlHttpCommandExecutorService [user thread 8]: Receiving
response -2055078403: HTTP/1.1 200 OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << HTTP/1.1 200 OK
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Date: Wed, 08 Jun
2016 11:03:48 GMT
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Connection: close
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << X-Compute-Request-Id:
req-8c17bd92-eada-4d90-b002-4cd2ef697343
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Content-Type: application/json
2016-06-08 12:03:48,855 DEBUG jclouds.headers [user thread 8]: << Content-Length: 2709
2016-06-08 12:03:48,855 DEBUG jclouds.wire [user thread 8]: << "{"server": {"status":
"ERROR", "updated": "2016-06-08T10:33:51Z", "hostId": "8a47d0589f92964343fa21a82dfdc01abf880c069ffaf88c7bf8ba4a",
"OS-EXT-SRV-ATTR:host": "ds0017", "addresses": {}, "links": [{"href": 
"http://acme.com:8774/v2/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495-4148-9a09-c4dee0d75e7a",
"rel": "self"}, {"href": "http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/servers/e74952b7-f495
-4148-9a09-c4dee0d75e7a", "rel": "bookmark"}], "key_name": "openstack", "image": {"id": "e16a7bca-7363-45b1-bd6a-02479d4cea77",
"links": [{"href": "http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/images/e16a7bca-7363-45b1-bd6a-02479
d4cea77", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "error",
"OS-EXT-SRV-ATTR:instance_name": "instance-00000d8b", "OS-SRV-USG:launched_at": null, "OS-EXT-SRV-ATTR:hypervisor_hostname":
"ds0017.sng01.blueboxgrid.com", "flavor": {"id": "
2", "links": [{"href": "http://acme.com:8774/691aa6dd3e4e482aaaf7f73b16fffc18/flavors/2",
"rel": "bookmark"}]}, "id": "e74952b7-f495-4148-9a09-c4dee0d75e7a", "OS-SRV-USG:terminated_at":
null, "OS-EXT-AZ:availability_zone": "nova", "user_i
d": "111195567c864830ae4ade124a4a1efe", "name": "brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823",
"created": "2016-06-08T10:33:48Z", "tenant_id": "691aa6dd3e4e482aaaf7f73b16fffc18", "OS-DCF:diskConfig":
"MANUAL", "os-extended-volumes:volumes_attached": [], "
accessIPv4": "", "accessIPv6": "", "fault": {"message": "No valid host was found. Exceeded
max scheduling attempts 3 for instance e74952b7-f495-4148-9a09-c4dee0d75e7a. Last exception:
[u'Traceback (most recent call last):\\n', u'  File \"/opt/bbc/openstack-11.0-bbc173/n
ova/local/lib/python2.7/site-packages/nova/c", "code": 500, "details": "  File \"/opt/bbc/openstack-11.0-bbc173/nova/local/lib/python2.7/site-packages/nova/conductor/manager.py\",
line 669, in build_instances\n    instances[0].uuid)\n  File \"/opt/bbc/openstack-11.0-bbc
173/nova/local/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 172, in populate_retry\n
   raise exception.NoValidHost(reason=msg)\n", "created": "2016-06-08T10:33:51Z"}, "OS-EXT-STS:power_state":
0, "config_drive": "", "metadata": {"brooklyn-app-name": "Doc
ker Swarm", "brooklyn-entity-id": "rj29jfrfvy", "brooklyn-app-id": "yfxfed97j6", "brooklyn-user":
"aled", "brooklyn-entity-name": "swarm-node", "brooklyn-server-creation-date": "2016-06-08-1133",
"Name": "brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-zj5s", "jc
louds-group": "brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29"}}}"
2016-06-08 12:03:48,860 WARN  jclouds.compute [user thread 8]: Server brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29-823
was not created within 30 * 60ms so it will be destroyed.
2016-06-08 12:03:48,940 ERROR jclouds.compute [user thread 14]: createNodesInGroup(brooklyn-o8g809-aled-docker-swarm-yfxf-swarm-node-rj29),
completed: 0/1, errors: 1, rate: 1802112ms/op
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: id must be in
format regionId/id
        at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-16.0.1.jar:na]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-16.0.1.jar:na]
        at org.jclouds.concurrent.FutureIterables$1.run(FutureIterables.java:123) ~[jclouds-core-1.9.2.jar:1.9.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
Caused by: java.lang.IllegalArgumentException: id must be in format regionId/id
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:125) ~[guava-16.0.1.jar:na]
        at org.jclouds.openstack.nova.v2_0.domain.regionscoped.RegionAndId.fromSlashEncoded(RegionAndId.java:31)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.openstack.nova.v2_0.compute.functions.CleanupServer.apply(CleanupServer.java:66)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.destroyNode(NovaComputeServiceAdapter.java:269)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.openstack.nova.v2_0.compute.NovaComputeServiceAdapter.createNodeWithGroupEncodedIntoName(NovaComputeServiceAdapter.java:152)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.compute.strategy.impl.AdaptingComputeServiceStrategies.createNodeWithGroupEncodedIntoName(AdaptingComputeServiceStrategies.java:195)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:79)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet$AddNode.call(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:63)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_71]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_71]
        at java.lang.Thread.getStackTrace(Thread.java:1589) [na:1.7.0_71]
        at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:167)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.createNodeInGroupWithNameAndTemplate(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:161)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.compute.strategy.impl.CreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(CreateNodesWithGroupEncodedIntoNameThenAddToSet.java:122)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at org.jclouds.openstack.nova.v2_0.compute.strategy.ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.execute(ApplyNovaTemplateOptionsCreateNodesWithGroupEncodedIntoNameThenAddToSet.java:154)
~[openstack-nova-1.9.2-cloudsoft-20160121.jar:1.9.2-cloudsoft-20160121]
        at org.jclouds.compute.internal.BaseComputeService.createNodesInGroup(BaseComputeService.java:210)
~[jclouds-compute-1.9.2.jar:1.9.2]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_71]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.7.0_71]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_71]
        at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37)
~[guice-3.0.jar:na]
        at com.sun.proxy.$Proxy115.createNodesInGroup(Unknown Source) ~[na:na]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.obtainOnce(JcloudsLocation.java:732)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.location.jclouds.JcloudsLocation.obtain(JcloudsLocation.java:622)
~[brooklyn-locations-jclouds-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:406)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ObtainLocationTask.call(MachineLifecycleEffectorTasks.java:396)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.util.core.task.Tasks.withBlockingDetails(Tasks.java:98) ~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:380)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.entity.software.base.lifecycle.MachineLifecycleEffectorTasks$ProvisionMachineTask.call(MachineLifecycleEffectorTasks.java:364)
~[brooklyn-software-base-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.util.core.task.DynamicSequentialTask$DstJob.call(DynamicSequentialTask.java:359)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
~[brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_71]
        ... 3 common frames omitted
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message