cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dobropalka" <dobropa...@gmail.com>
Subject shared network with source NAT
Date Wed, 09 Jul 2014 15:59:17 GMT
Hey guys, please help me with one issue.

I'm working with 4.3 CS + 6.1 Xenserver cluster

We got 4 public ip, and I want to make vms to have access to internet from
guest net via NAT.

I understood how to make it with isolated net - I can just create net with
DefaultIsolatedNetworkOfferingWithSourceNatService offering

Everything goes smooth, until I need to get an access to this net from other
accounts. It's isolated, so it's not possible.

Ok, I try to create custom network offering with firewall, source nat, dns,
dhcp and user data.

Then create shared net. 

But I can't see this network in the list when I create VM.

If I try to add net to existing vm I am getting error:

 

 

2014-07-09 10:26:58,513 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-0de0852c ctx-a16637e0) Adding vm VM[User|testnat] to
network Ntwk[209|Guest|17]; requested nic profile
NicProfile[0-0-null-null-null

2014-07-09 10:26:58,516 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-0de0852c ctx-a16637e0) Allocating nic for vm
VM[User|testnat] in network Ntwk[209|Guest|17] with requested profile
NicProfile[0-0-null-null-null

2014-07-09 10:26:58,518 WARN  [c.c.n.IpAddressManagerImpl]
(Job-Executor-2:ctx-0de0852c ctx-a16637e0) Unable to get ip adress in  zone
id=1, vlanId id=[Ljava.lang.Object;@76dd522e, network id=209

2014-07-09 10:26:58,518 DEBUG [c.c.u.d.T.Transaction]
(Job-Executor-2:ctx-0de0852c ctx-a16637e0) Rolling back the transaction:
Time = 2 Name =  Job-Executor-2; called by
-TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-Transaction
Legacy.close:663-Transaction.execute:41-IpAddressManagerImpl.fetchNewPublicI
p:662-IpAddressManagerImpl.assignPublicIpAddress:650-IpAddressManagerImpl$11
.doInTransactionWithoutResult:1894-TransactionCallbackWithExceptionNoReturn.
doInTransaction:25-TransactionCallbackWithExceptionNoReturn.doInTransaction:
21-Transaction.execute:37-IpAddressManagerImpl.allocateDirectIp:1872-DirectN
etworkGuru$1.doInTransactionWithoutResult:244

2014-07-09 10:26:58,519 ERROR [c.c.a.ApiAsyncJobDispatcher]
(Job-Executor-2:ctx-0de0852c) Unexpected exception while executing
org.apache.cloudstack.api.command.user.vm.AddNicToVMCmd

com.cloud.utils.exception.CloudRuntimeException: Insufficient capacity when
adding NIC to VM[User|testnat]:
com.cloud.exception.InsufficientAddressCapacityException: Insufficient
address capacityScope=interface com.cloud.dc.DataCenter; id=1

        at
com.cloud.vm.UserVmManagerImpl.addNicToVirtualMachine(UserVmManagerImpl.java
:1047)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy169.addNicToVirtualMachine(Unknown Source)

        at
org.apache.cloudstack.api.command.user.vm.AddNicToVMCmd.execute(AddNicToVMCm
d.java:110)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 10:26:58,520 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-2:ctx-0de0852c) Complete async job-285, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"er
rorcode":530,"errortext":"Insufficient capacity when adding NIC to
VM[User|testnat]: com.cloud.exception.InsufficientAddressCapacityException:
Insufficient address capacityScope\u003dinterface com.cloud.dc.DataCenter;
id\u003d1"}

2014-07-09 10:26:58,529 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-2:ctx-0de0852c) Done executing
org.apache.cloudstack.api.command.user.vm.AddNicToVMCmd for job-285

2014-07-09 10:26:58,533 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(Job-Executor-2:ctx-0de0852c) Remove job-285 from job monitoring

 

Ok, I try to restart net so it create router.

 

Router trying to start and I'm getting following messages:

2014-07-09 10:42:58,084 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Waiting some more to make sure
there's no activity on VM[DomainRouter|r-216-VM]

 

They spam ~5 times a second for a half an hour and then errors with smth
like "network fails to restart"

Virtual router stuck in starting state.

VM isn't created on XEN.

 

Logs:

The task item for vm VM[DomainRouter|r-216-VM] has been inactive for 3660

2014-07-09 11:43:13,647 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Cleaning up because we're unable
to implement the network Ntwk[209|Guest|17]

2014-07-09 11:43:13,659 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Lock is acquired for network
Ntwk[209|Guest|17] as a part of network shutdown

2014-07-09 11:43:13,664 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Releasing 0 port forwarding rules
for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:43:13,664 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no rules to forward to
the network elements

2014-07-09 11:43:13,665 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Releasing 0 static nat rules for
network id=209 as a part of shutdownNetworkRules

2014-07-09 11:43:13,665 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no rules to forward to
the network elements

2014-07-09 11:43:13,665 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Revoking 0 Public load balancing
rules for network id=209

2014-07-09 11:43:13,665 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no Load Balancing Rules
to forward to the network elements

2014-07-09 11:43:13,666 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Revoking 0 Internal load
balancing rules for network id=209

2014-07-09 11:43:13,666 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no Load Balancing Rules
to forward to the network elements

2014-07-09 11:43:13,666 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Releasing 0 firewall ingress
rules for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:43:13,666 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no rules to forward to
the network elements

2014-07-09 11:43:13,666 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Releasing 0 firewall egress rules
for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:43:13,667 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) applying default firewall egress
rules

2014-07-09 11:43:13,673 WARN  [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Unable to apply firewall rules,
virtual router is not in the right state Starting

2014-07-09 11:43:13,673 WARN  [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Failed to apply default egress
rules for guest network due to

com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to apply firewall rules, virtual router is not in the
right state

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(Virtu
alNetworkApplianceManagerImpl.java:3928)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyFirewallRul
es(VirtualNetworkApplianceManagerImpl.java:3721)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.applyFirewallRules(Unknown Source)

        at
com.cloud.network.element.VirtualRouterElement.applyFWRules(VirtualRouterEle
ment.java:248)

        at
com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImp
l.java:567)

        at
com.cloud.network.IpAddressManagerImpl.applyRules(IpAddressManagerImpl.java:
502)

        at
com.cloud.network.firewall.FirewallManagerImpl.applyRules(FirewallManagerImp
l.java:521)

        at
com.cloud.network.firewall.FirewallManagerImpl.applyDefaultEgressFirewallRul
e(FirewallManagerImpl.java:671)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy174.applyDefaultEgressFirewallRule(Unknown
Source)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rkResources(NetworkOrchestrator.java:2704)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rkElementsAndResources(NetworkOrchestrator.java:2044)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rk(NetworkOrchestrator.java:1982)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
ork(NetworkOrchestrator.java:1006)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo
rkOrchestrator.java:1238)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:960)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwor
k(NetworkOrchestrator.java:2387)

        at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:
1847)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.restartNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(Res
tartNetworkCmd.java:92)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

       at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 11:43:13,674 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) There are no rules to forward to
the network elements

2014-07-09 11:43:13,675 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Found 0 static nat rules to apply
for network id 209

2014-07-09 11:43:13,679 WARN  [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Unable to apply ip association,
virtual router is not in the right state Starting

2014-07-09 11:43:13,679 DEBUG [c.c.n.IpAddressManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Resource is not available:
VirtualRouter

com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to apply ip association, virtual router is not in the
right state

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(Virtu
alNetworkApplianceManagerImpl.java:3928)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.associatePublicI
P(VirtualNetworkApplianceManagerImpl.java:3705)

        at
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.associatePubl
icIP(VpcVirtualNetworkApplianceManagerImpl.java:513)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.associatePublicIP(Unknown Source)

        at
com.cloud.network.element.VirtualRouterElement.applyIps(VirtualRouterElement
.java:476)

        at
com.cloud.network.IpAddressManagerImpl.applyIpAssociations(IpAddressManagerI
mpl.java:976)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rkResources(NetworkOrchestrator.java:2764)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rkElementsAndResources(NetworkOrchestrator.java:2044)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rk(NetworkOrchestrator.java:1982)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
ork(NetworkOrchestrator.java:1006)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo
rkOrchestrator.java:1238)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:960)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwor
k(NetworkOrchestrator.java:2387)

        at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:
1847)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.restartNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(Res
tartNetworkCmd.java:92)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 11:43:13,680 WARN  [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Unable to apply ip address
associations for Ntwk[209|Guest|17] as a part of shutdownNetworkRules

2014-07-09 11:43:13,680 WARN  [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Failed to cleanup network id=209
resources as a part of shutdownNetwork

2014-07-09 11:43:13,681 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Sending network shutdown to
VirtualRouter

2014-07-09 11:43:13,681 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Stopping router
VM[DomainRouter|r-216-VM]

2014-07-09 11:43:13,682 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Found an outstanding work item
for this vm VM[DomainRouter|r-216-VM] with state:Starting, work
id:e2dbfbf3-23c5-4a62-b954-9804717b815e

2014-07-09 11:43:13,683 WARN  [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Unable to complete shutdown of
the network elements due to element: VirtualRouter

com.cloud.utils.exception.CloudRuntimeException: We cannot stop
VM[DomainRouter|r-216-VM] when it is in state Starting

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl
.java:1402)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManager
Impl.java:1344)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl
.java:1312)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.stop(VirtualNetw
orkApplianceManagerImpl.java:2867)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.stop(VirtualNetw
orkApplianceManagerImpl.java:285)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.stop(Unknown Source)

        at
com.cloud.network.element.VirtualRouterElement.shutdown(VirtualRouterElement
.java:665)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rkElementsAndResources(NetworkOrchestrator.java:2069)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetwo
rk(NetworkOrchestrator.java:1982)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
ork(NetworkOrchestrator.java:1006)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo
rkOrchestrator.java:1238)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:960)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwor
k(NetworkOrchestrator.java:2387)

        at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:
1847)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.restartNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(Res
tartNetworkCmd.java:92)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 11:43:13,689 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Lock is released for network
Ntwk[209|Guest|17] as a part of network shutdown

2014-07-09 11:43:13,689 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Lock is released for network id
209 as a part of network implement

2014-07-09 11:43:13,689 ERROR [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Failed to start instance
VM[DomainRouter|r-216-VM]

com.cloud.exception.ConcurrentOperationException: There are concurrent
operations on VM[DomainRouter|r-216-VM]

        at
com.cloud.vm.VirtualMachineManagerImpl.changeToStartState(VirtualMachineMana
gerImpl.java:701)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:800)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
ork(NetworkOrchestrator.java:977)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo
rkOrchestrator.java:1238)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:960)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwor
k(NetworkOrchestrator.java:2387)

        at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:
1847)

       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.restartNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(Res
tartNetworkCmd.java:92)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 11:43:13,698 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Cleaning up resources for the vm
VM[DomainRouter|r-216-VM] in Starting state

2014-07-09 11:43:13,699 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-707e905c
ctx-edeb54d9) Seq 15-1492189375: Sending  { Cmd , MgmtId: 128437433941265,
via: 15(cloudstack-computin02-sar.tech-corps.com), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":fal
se,"vmName":"r-216-VM","wait":0}}] }

2014-07-09 11:43:13,699 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-707e905c
ctx-edeb54d9) Seq 15-1492189375: Executing:  { Cmd , MgmtId:
128437433941265, via: 15(cloudstack-computin02-sar.tech-corps.com), Ver: v1,
Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":fal
se,"vmName":"r-216-VM","wait":0}}] }

2014-07-09 11:43:13,699 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-41:ctx-525084f2) Seq 15-1492189375: Executing request

2014-07-09 11:43:13,712 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-3:null) Ping from 19

2014-07-09 11:43:13,716 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-4:null) Ping from 17

2014-07-09 11:43:13,777 INFO  [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-41:ctx-525084f2) VM does not exist on
XenServer90c04db1-a259-4740-9194-dc9288352141

2014-07-09 11:43:13,777 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-41:ctx-525084f2) Seq 15-1492189375: Response Received:

2014-07-09 11:43:13,777 DEBUG [c.c.a.t.Request]
(DirectAgent-41:ctx-525084f2) Seq 15-1492189375: Processing:  { Ans: ,
MgmtId: 128437433941265, via: 15, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not
exist","wait":0}}] }

2014-07-09 11:43:13,777 DEBUG [c.c.a.t.Request] (Job-Executor-1:ctx-707e905c
ctx-edeb54d9) Seq 15-1492189375: Received:  { Ans: , MgmtId:
128437433941265, via: 15, Ver: v1, Flags: 10, { StopAnswer } }

2014-07-09 11:43:13,782 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Successfully updated user
statistics as a part of domR VM[DomainRouter|r-216-VM] reboot/stop

2014-07-09 11:43:13,800 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Successfully released network
resources for the vm VM[DomainRouter|r-216-VM]

2014-07-09 11:43:13,800 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Successfully cleanued up
resources for the vm VM[DomainRouter|r-216-VM] in Starting state

2014-07-09 11:43:13,817 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) VM state transitted from
:Starting to Stopped with event: OperationFailedvm's original host id: null
new host id: null host id before state transition: 15

2014-07-09 11:43:13,819 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Hosts's actual total CPU: 15960
and CPU after applying overprovisioning: 15960

2014-07-09 11:43:13,819 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Hosts's actual total RAM:
20233051776 and RAM after applying overprovisioning: 20233052160

2014-07-09 11:43:13,819 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) release cpu from host: 15, old
used: 8490,reserved: 0, actual total: 15960, total with overprovisioning:
15960; new used: 7990,reserved:0; movedfromreserved:
false,moveToReserveredfalse

2014-07-09 11:43:13,819 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) release mem from host: 15, old
used: 7784628224,reserved: 0, total: 20233052160; new used:
7650410496,reserved:0; movedfromreserved: false,moveToReserveredfalse

2014-07-09 11:43:13,828 WARN  [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Failed to implement network
Ntwk[209|Guest|17] elements and resources as a part of network restart due
to

com.cloud.exception.AgentUnavailableException: Resource [Host:15] is
unreachable: Host 15: Unable to start instance due to There are concurrent
operations on VM[DomainRouter|r-216-VM]

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:1072)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwor
k(NetworkOrchestrator.java:2387)

        at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:
1847)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.restartNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(Res
tartNetworkCmd.java:92)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

Caused by: com.cloud.exception.ConcurrentOperationException: There are
concurrent operations on VM[DomainRouter|r-216-VM]

        at
com.cloud.vm.VirtualMachineManagerImpl.changeToStartState(VirtualMachineMana
gerImpl.java:701)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:800)

        at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp
l.java:761)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet
workApplianceManagerImpl.java:2845)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout
er(VirtualNetworkApplianceManagerImpl.java:1924)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir
tualNetworkApplianceManagerImpl.java:2024)

        at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou
terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:2002)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy240.deployVirtualRouterInGuestNetwork(Unknown
Source)

        at
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElemen
t.java:188)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
orkElementsAndResources(NetworkOrchestrator.java:1070)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetw
ork(NetworkOrchestrator.java:977)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(Netwo
rkOrchestrator.java:1238)

        at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManage
rImpl.java:960)

        ... 52 more

2014-07-09 11:43:13,829 WARN  [c.c.n.NetworkServiceImpl]
(Job-Executor-1:ctx-707e905c ctx-edeb54d9) Network id=209 failed to restart.

2014-07-09 11:43:13,838 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-1:ctx-707e905c) Complete async job-286, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"er
rorcode":530,"errortext":"Failed to restart network"}

 

 

After it  I can't delete this network:

 

2014-07-09 11:56:44,268 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-2:ctx-c5eff164) Executing AsyncJobVO {id:289, userId: 40,
accountId: 40, instanceType: None, instanceId: null, cmd:
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd, cmdInfo:
{"id":"19cab1df-7b01-4b77-955e-bb93a79968b3","response":"json","sessionkey":
"xuRxs4uf0PmKziiC/WE43JkdnxM\u003d","cmdEventType":"NETWORK.DELETE","ctxUser
Id":"40","httpmethod":"GET","_":"1404921388845","ctxAccountId":"40","ctxStar
tEventId":"1262"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 128437433941265, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null}

2014-07-09 11:56:44,284 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Lock is acquired for network
Ntwk[209|Guest|17] as a part of network shutdown

2014-07-09 11:56:44,289 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 port forwarding rules
for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:56:44,289 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no rules to forward to
the network elements

2014-07-09 11:56:44,290 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 static nat rules for
network id=209 as a part of shutdownNetworkRules

2014-07-09 11:56:44,290 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no rules to forward to
the network elements

2014-07-09 11:56:44,290 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoking 0 Public load balancing
rules for network id=209

2014-07-09 11:56:44,290 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no Load Balancing Rules
to forward to the network elements

2014-07-09 11:56:44,291 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoking 0 Internal load
balancing rules for network id=209

2014-07-09 11:56:44,291 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no Load Balancing Rules
to forward to the network elements

2014-07-09 11:56:44,291 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 firewall ingress
rules for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:56:44,291 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no rules to forward to
the network elements

2014-07-09 11:56:44,291 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 firewall egress rules
for network id=209 as a part of shutdownNetworkRules

2014-07-09 11:56:44,293 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) applying default firewall egress
rules

2014-07-09 11:56:44,297 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Router r-216-VM is in Stopped, so
not sending apply firewall rules commands to the backend

2014-07-09 11:56:44,297 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no rules to forward to
the network elements

2014-07-09 11:56:44,298 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Found 0 static nat rules to apply
for network id 209

2014-07-09 11:56:44,302 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Router r-216-VM is in Stopped, so
not sending apply ip association commands to the backend

2014-07-09 11:56:44,303 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Sending network shutdown to
VirtualRouter

2014-07-09 11:56:44,304 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Stopping router
VM[DomainRouter|r-216-VM]

2014-07-09 11:56:44,304 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) VM is already stopped:
VM[DomainRouter|r-216-VM]

2014-07-09 11:56:44,305 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Network id=209 is shutdown
successfully, cleaning up corresponding resources now.

2014-07-09 11:56:44,317 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Lock is released for network
Ntwk[209|Guest|17] as a part of network shutdown

2014-07-09 11:56:44,318 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 port forwarding rules
for network id=209

2014-07-09 11:56:44,318 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 static nat rules for
network id=209

2014-07-09 11:56:44,319 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no port forwarding
rules to apply for network id=209

2014-07-09 11:56:44,319 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no static nat rules to
apply for network id=209

2014-07-09 11:56:44,320 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully released rules for
network id=209 and # of rules now = 0

2014-07-09 11:56:44,320 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully cleaned up
portForwarding/staticNat rules for network id=209

2014-07-09 11:56:44,320 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Found 0 lb rules to cleanup

2014-07-09 11:56:44,320 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully cleaned up load
balancing rules for network id=209

2014-07-09 11:56:44,330 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 firewall rules for
network id=209

2014-07-09 11:56:44,330 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no firewall rules to
apply

2014-07-09 11:56:44,330 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully released firewall
rules for network id=209 and # of rules now = 0

2014-07-09 11:56:44,339 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully cleaned up
firewallRules rules for network id=209

2014-07-09 11:56:44,340 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully cleaned up
NetworkACLs for network id=209

2014-07-09 11:56:44,352 DEBUG [c.c.n.IpAddressManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoking all Firewallrules as a
part of public IP id=2 release...

2014-07-09 11:56:44,361 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 firewall rules for ip
id=2

2014-07-09 11:56:44,362 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no firewall rules to
apply

2014-07-09 11:56:44,362 DEBUG [c.c.n.f.FirewallManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully released firewall
rules for ip id=2 and # of rules now = 0

2014-07-09 11:56:44,371 DEBUG [c.c.n.IpAddressManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoking all
PortForwarding/StaticNat rules as a part of public IP id=2 release...

2014-07-09 11:56:44,372 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 port forwarding rules
for ip id=2

2014-07-09 11:56:44,372 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing 0 static nat rules for
ip id=2

2014-07-09 11:56:44,373 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no port forwarding
rules to apply for ip id=2

2014-07-09 11:56:44,373 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) There are no static nat rules to
apply for ip id=2

2014-07-09 11:56:44,373 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Source ip id=Ip[10.7.129.3-1] is
not one to one nat

2014-07-09 11:56:44,374 DEBUG [c.c.n.r.RulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Successfully released rules for
ip id=2 and # of rules now = 0

2014-07-09 11:56:44,374 DEBUG [c.c.n.IpAddressManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoking all LoadBalancing rules
as a part of public IP id=2 release...

2014-07-09 11:56:44,374 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Found 0 lb rules to cleanup

2014-07-09 11:56:44,374 DEBUG [c.c.n.IpAddressManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Cleaning up remote access vpns as
a part of public IP id=2 release...

2014-07-09 11:56:44,375 DEBUG [c.c.n.v.RemoteAccessVpnManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) there are no Remote access vpns
for public ip address id=2

2014-07-09 11:56:44,389 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Sending destroy to
com.cloud.network.element.VirtualRouterElement@6e1f669f

2014-07-09 11:56:44,390 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Attempting to destroy router 216

2014-07-09 11:56:44,392 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) VM is already stopped:
VM[DomainRouter|r-216-VM]

2014-07-09 11:56:44,401 DEBUG [c.c.c.CapacityManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) VM state transitted from :Stopped
to Expunging with event: ExpungeOperationvm's original host id: null new
host id: null host id before state transition: null

2014-07-09 11:56:44,402 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Destroying vm
VM[DomainRouter|r-216-VM]

2014-07-09 11:56:44,402 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Cleaning up NICS

2014-07-09 11:56:44,402 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Cleaning network for vm: 216

2014-07-09 11:56:44,411 DEBUG [c.c.n.NetworkModelImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Service SecurityGroup is not
supported in the network id=209

2014-07-09 11:56:44,413 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Asking VirtualRouter to release
Nic[685-216-null-10.2.2.1]

2014-07-09 11:56:44,413 DEBUG [c.c.n.g.DirectNetworkGuru]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Deallocate network: networkId:
209, ip: 10.2.2.1

2014-07-09 11:56:44,422 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Removed nic id=685

2014-07-09 11:56:44,423 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoving nic secondary ip entry
...

2014-07-09 11:56:44,441 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Removed nic id=686

2014-07-09 11:56:44,441 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoving nic secondary ip entry
...

2014-07-09 11:56:44,451 DEBUG [c.c.n.g.PublicNetworkGuru]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) public network deallocate
network: networkId: 200, ip: 10.7.129.3

2014-07-09 11:56:44,452 DEBUG [c.c.n.g.PublicNetworkGuru]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Deallocated nic:
NicProfile[687-216-null-null-null

2014-07-09 11:56:44,461 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Removed nic id=687

2014-07-09 11:56:44,462 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Revoving nic secondary ip entry
...

2014-07-09 11:56:44,462 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Cleaning up hypervisor data
structures (ex. SRs in XenServer) for managed storage

2014-07-09 11:56:44,463 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Cleaning storage for vm: 216

2014-07-09 11:56:44,485 INFO  [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Expunge volume with no data store
specified

2014-07-09 11:56:44,486 INFO  [o.a.c.s.v.VolumeServiceImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Volume 222 is not referred
anywhere, remove it from volumes table

2014-07-09 11:56:44,495 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Expunged
VM[DomainRouter|r-216-VM]

2014-07-09 11:56:44,511 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Network id=209 is destroyed
successfully, cleaning up corresponding resources now.

2014-07-09 11:56:44,512 DEBUG [c.c.n.g.DirectNetworkGuru]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Releasing ip 10.2.2.1 of
placeholder nic Nic[684-null-null-10.2.2.1]

2014-07-09 11:56:44,513 DEBUG [c.c.u.d.T.Transaction]
(Job-Executor-2:ctx-c5eff164 ctx-5f4d2d9a) Rolling back the transaction:
Time = 2 Name =  Job-Executor-2; called by
-TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-Transaction
Legacy.close:663-Transaction.execute:41-Transaction.execute:46-DirectNetwork
Guru.trash:322-NetworkOrchestrator$10.doInTransactionWithoutResult:2188-Tran
sactionCallbackNoReturn.doInTransaction:25-Transaction$2.doInTransaction:49-
Transaction.execute:37-Transaction.execute:46-NetworkOrchestrator.destroyNet
work:2183

2014-07-09 11:56:44,522 ERROR [c.c.a.ApiAsyncJobDispatcher]
(Job-Executor-2:ctx-c5eff164) Unexpected exception while executing
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd

java.lang.NullPointerException

        at
com.cloud.network.guru.DirectNetworkGuru$3.doInTransactionWithoutResult(Dire
ctNetworkGuru.java:329)

        at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCa
llbackNoReturn.java:25)

        at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)

        at com.cloud.utils.db.Transaction.execute(Transaction.java:37)

        at com.cloud.utils.db.Transaction.execute(Transaction.java:46)

        at
com.cloud.network.guru.DirectNetworkGuru.trash(DirectNetworkGuru.java:322)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator$10.doInTransa
ctionWithoutResult(NetworkOrchestrator.java:2188)

        at
com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCa
llbackNoReturn.java:25)

        at
com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)

        at com.cloud.utils.db.Transaction.execute(Transaction.java:37)

        at com.cloud.utils.db.Transaction.execute(Transaction.java:46)

        at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.destroyNetwor
k(NetworkOrchestrator.java:2183)

        at
com.cloud.network.NetworkServiceImpl.deleteNetwork(NetworkServiceImpl.java:1
810)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)

        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopU
tils.java:317)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint
(ReflectiveMethodInvocation.java:183)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:150)

        at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50
)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:161)

        at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Expos
eInvocationInterceptor.java:91)

        at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:172)

        at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)

        at com.sun.proxy.$Proxy199.deleteNetwork(Unknown Source)

        at
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd.execute(Dele
teNetworkCmd.java:77)

        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.ja
va:109)

        at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)

        at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext
(AsyncJobManagerImpl.java:509)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedCo
ntextRunnable.java:49)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(Defa
ultManagedContext.java:56)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithCon
text(DefaultManagedContext.java:103)

        at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithCont
ext(DefaultManagedContext.java:53)

        at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedCont
extRunnable.java:46)

        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
45)

        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
15)

        at java.lang.Thread.run(Thread.java:744)

2014-07-09 11:56:44,522 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-2:ctx-c5eff164) Complete async job-289, jobStatus: FAILED,
resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"er
rorcode":530}

2014-07-09 11:56:44,531 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-2:ctx-c5eff164) Done executing
org.apache.cloudstack.api.command.user.network.DeleteNetworkCmd for job-289

 

So, how can I make a shared net with NAT?

Yes, we can move all users from separate accounts to one big account. But
it's bad and can't work with, for example, LDAP auth

 

 

Regards, 

Ivan Derbenev

 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message