cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirk Kosinski <kirk.kosin...@shapeblue.com>
Subject RE: Reestablishing a VR when the VR was deleted
Date Mon, 19 Sep 2016 22:46:32 GMT
Hi, it's good to hear it's resolved.  The document you linked is correct in the context of
an upgrade.  The upgrade process looks at the name and description of the templates to find
the new system VM template, and will mark it as such in the database.  If, after an upgrade,
the system VM template is somehow removed I'm not sure how to fix it besides a database hack.

Best regards,
Kirk

kirk.kosinski@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HS
@shapeblue
  
 


-----Original Message-----
From: Mason Donahue [mailto:mdonahue@backstopsolutions.com] 
Sent: Monday, September 19, 2016 3:05 PM
To: users@cloudstack.apache.org
Subject: Re: Reestablishing a VR when the VR was deleted

Well, that was the bit of information that finally got us back up and running.

The original routerVM failed to start due to the template not being there, so I had redownloaded
the template according to the directions at https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.4.1/upgrade/upgrade-4.3.html
and that seems to have been the source of the problem.

Within the errors that accompanied the failure to start of the random VM I picked there was
an attempt to start a VR:

2016-09-19 15:29:48,962 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Lock is acquired for network id 204 as a part of router
startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,972 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Guest network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 15:29:48,977 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Adding nic for Virtual Router in Control network
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Found existing network configuration for offering [Network
Offering [3-Control-System-Control-Network]: Ntwk[28d967d8-3b75-4362-9234-b0d029b0d21b|Control|3]
2016-09-19 15:29:48,980 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Releasing lock for Acct[7500fc58-dcf6-11e2-b492-00219b9585d4-system]
2016-09-19 15:29:48,984 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Allocating the VR i=1703 in datacenter com.cloud.dc.DataCenterVO$$EnhancerByCGLIB$$9732e921@1with
the hypervisor type KVM
2016-09-19 15:29:48,988 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) KVM won't support system vm, skip it
2016-09-19 15:29:48,989 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Lock is released for network id 204 as a part of router
startup in Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod(1)-Cluster(7)-Host(25)-Storage()]
2016-09-19 15:29:48,989 INFO  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-4:ctx-21dac544
job-10167/job-10169 ctx-d441bcf6) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Can't find at least one running router!

The line ‘KVM won’t support system vm, skip it’ was the key, as googling it led to other
cloudstack-users mailing list posts with other people who had failed to add systemvm templates
back in. Shutting down the management server, updating the TYPE column in the database for
the systemvm to ‘SYSTEM’ instead of ‘USER’, and starting a VM again fixed it, and
everything came back up happily.

It seems like the documentation on importing systemVMs (for instance, https://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.9.0/upgrade/upgrade-4.4.html)
is incorrect, as it says that the template should be imported as ‘Routing: no’. Or is
there some other method in which a template is supposed to be set to SYSTEM instead of USER?

Thanks!

--Mason

On 9/19/16, 3:22 PM, "Kirk Kosinski" <kirk.kosinski@shapeblue.com> wrote:

    Hi, if you start a VM in a network that has no VR, the VR will be recreated.  So you can
stop/start an existing VM in the network, or deploy a new VM to the network.
    
    Best regards,
    
    
    kirk.kosinski@shapeblue.com 
    www.shapeblue.com
    53 Chandos Place, Covent Garden, London  WC2N 4HS
    @shapeblue
      
     
    
    
    -----Original Message-----
    From: Mason Donahue [mailto:mdonahue@backstopsolutions.com] 
    Sent: Monday, September 19, 2016 9:56 AM
    To: users@cloudstack.apache.org
    Subject: Reestablishing a VR when the VR was deleted
    
    Hi there,
    
    We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s outdated;
we were hoping to upgrade soon and then this happened.) We had an issue where the VR for one
of our Networks went down, and I mistakenly missed the ‘cleanup’ checkbox in the ‘Restart
network’ menu of the UI. We use the VR solely for DNS. We now have no VRs in our setup,
and the network can no longer be restarted due to a check that routers are running. Unfortunately,
to delete and re-add the network, the log output states that we’d have to expunge all of
our machines, which I am hoping to avoid doing. What are my other options? Can I un-mark the
router as deleted in the DB, and will that allow it to limp along to the point where it can
at least rebuild the network?
    
    Thanks,
    --Mason
    
    (relevant logs below)
    
    2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141) Executing AsyncJobVO {id:10141, userId: 2, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: {"response":"json","id":"448deced-7
    223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
    txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: IN_PROGRESS, processStatus:
0, resultCode: 0, result: null, initMsid: 144343483243, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
    2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d ctx-69ffa947)
===END===  192.168.42.156 -- GET  command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
    2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Restarting network 204...
    2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Shutting down the network id=204 as a part of network restart
    2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Releasing 0 port forwarding rules for network id=204 as a part of
shutdownNetworkRules
    2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
    2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Releasing 0 static nat rules for network id=204 as a part of shutdownNetworkRules
    2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
    2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Revoking 0 Public load balancing rules for network id=204
    2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
    2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Revoking 0 Internal load balancing rules for network id=204
    2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no Load Balancing Rules to forward to the network elements
    2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Releasing 0 firewall ingress rules for network id=204 as a part of
shutdownNetworkRules
    2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
    2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Releasing 0 firewall egress rules for network id=204 as a part of
shutdownNetworkRules
    2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) There are no rules to forward to the network elements
    2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Found 0 static nat rules to apply for network id 204
    2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Sending network shutdown to SecurityGroupProvider
    2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Sending network shutdown to VirtualRouter
    2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Implementing the network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
elements and resources as a part of network restart
    2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Asking SecurityGroupProvider to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
    2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Asking VirtualRouter to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
    2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Lock is acquired for network id 204 as a part of router startup in
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
    ost(null)-Storage()]
    2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Lock is released for network id 204 as a part of router startup in
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(null)-Cluster(null)-H
    ost(null)-Storage()]
    2016-09-19 11:46:51,053 WARN  [o.a.c.e.o.NetworkOrchestrator] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Failed to implement network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
elements and resources as a part of network restart due to
    com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable:
Can't find all necessary running routers!
            at com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
            at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
            at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
            at com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:606)
            at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
            at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
            at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
            at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
            at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
            at org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
            at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
            at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
            at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
            at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
            at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
            at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
            at java.util.concurrent.FutureTask.run(FutureTask.java:262)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
            at java.lang.Thread.run(Thread.java:745)
    2016-09-19 11:46:51,054 WARN  [c.c.n.NetworkServiceImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141 ctx-6dda2ed3) Network id=204 failed to restart.
    2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141) Complete async job-10141, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to restart network"}
    2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-8a42cc36
job-10141) Done executing org.apache.cloudstack.api.command.user.network.RestartNetworkCmd
for job-10141
    2016-09-19 11:46:51,107 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-5:ctx-8a42cc36
job-10141) Remove job-10141 from job monitoring
    
    

Mime
View raw message