cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ugo Vasi <ugo.v...@procne.it.INVALID>
Subject Re: Mysql got timeout and problems with UI
Date Wed, 06 Feb 2019 13:53:21 GMT
Hi all,
adding a new firewall rule I receive this error (note, after 1189 seconds):

2019-02-06 14:42:08,441 WARN  [o.a.c.f.j.i.AsyncJobMonitor] 
(Timer-0:ctx-c62615c7) (logid:37033038) Task (job-886) has been pending 
for 1189 seconds
2019-02-06 14:42:20,358 DEBUG [c.c.a.t.Request] 
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) 
Seq 7-4193977152988777787: Received:  { Ans: , MgmtId: 220777304233416, 
via: 7(cshp143), Ver: v1, Flags: 0, { GroupAnswer } }
2019-02-06 14:42:20,358 WARN  [c.c.n.f.FirewallManagerImpl] 
(API-Job-Executor-5:ctx-6106b5b5 job-886 ctx-ac682bcd) (logid:b4ed6266) 
Failed to apply firewall rules due to : Resource [DataCenter:1] is 
unreachable: Unable to apply firewall rules on router
2019-02-06 14:42:20,366 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-5:ctx-6106b5b5 job-886) (logid:b4ed6266) Complete 
async job-886, jobStatus: FAILED, resultCode: 530, result: 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed

to create firewall rule"}


The rule now is in "Deleting" state.


Il 06/02/19 10:59, Ugo Vasi ha scritto:
> Hi all,
> in a recente ACS 4.11.2.0 installation (Ubuntu 16.04 + KVM + Advanced 
> network) I have some problems in cloudstack usage.
>
> During the usual operations, I find a slowdown in the responses of the 
> UI and occasionally errors.
>
> This morning I created a new VM and I assigned an IP (xx.xx.xx.xx) of 
> a specific network (net-name).
>
> The  UI has stopped in the assignment phase even if in the logs (which 
> I report at the bottom of the email) I see that the job is executed in 
> a few seconds and updating the list of associated IPs in the network 
> list I see that the association is correct.
>
> I noticed that a series of these messages appear in the mysql log:
>
> 2019-02-06T00:50:58.758699Z 6298 [Note] Aborted connection 6298 to db: 
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets)
> 2019-02-06T00:51:59.352203Z 6269 [Note] Aborted connection 6269 to db: 
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets)
> 2019-02-06T00:51:59.421092Z 6289 [Note] Aborted connection 6289 to db: 
> 'dbname' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets)
> 2019-02-06T08:00:35.717673Z 6101 [Note] Aborted connection 6101 to db: 
> 'dbname_usage' user: 'username' host: 'localhost' (Got timeout reading 
> communication packets
>
>
> Note that the mysql server is used only by cloustack and is configured 
> according to the instructions in the cloudstack manual.
>
> I've already had a number of similar problems before (the version of 
> ACS was 4.11.1) and the situation has been getting worse and worse 
> until the interface and the whole cloudstack system has stopped 
> working (see "URGENT: Unable to schedule async job for command 
> com.cloud.vm.VmWorkStart, unexpected exception" and "urgent: Unable to 
> apply firewall rules on router").
>
>
> Has anyone had similar experiences or can help me understand where the 
> problem is?
>
>
>
> Job log (extracted from management-server.log):
>
> 2019-02-06 10:21:07,282 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:130cd0a6) Add job-876 
> into job monitoring 2019-02-06 10:21:07,294 DEBUG 
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a 
> job-876) (logid:ce8dc8fb) Executing AsyncJobVO {id:876, userId: 2, 
> accountId: 2, instanceType: IpAddress, instanceId: 31, cmd: 
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin, 
> cmdInfo: 
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface

> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface 
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},

> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, 
> result: null, initMsid: 220777304233416, completeMsid: null, 
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06 
> 10:21:07,295 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (qtp1096283470-521:ctx-31017793 ctx-63a1f58b) (logid:601bc8dd) submit 
> async job-876, details: AsyncJobVO {id:876, userId: 2, accountId: 2, 
> instanceType: IpAddress, instanceId: 31, cmd: 
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin, 
> cmdInfo: 
> {"response":"json","ctxUserId":"2","httpmethod":"GET","networkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","ctxStartEventId":"2308","id":"31","ctxDetails":"{\"interface

> com.cloud.network.IpAddress\":\"a710fbad-c2e5-409a-bd97-884e2d0ab7a5\",\"interface 
> com.cloud.network.Network\":\"bc0cc3a2-e21f-4904-a826-04477b1805c0\"}","ctxAccountId":"2","uuid":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","cmdEventType":"NET.IPASSIGN","_":"1549444854378"},

> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, 
> result: null, initMsid: 220777304233416, completeMsid: null, 
> lastUpdated: null, lastPolled: null, created: null} 2019-02-06 
> 10:21:07,354 DEBUG [c.c.n.IpAddressManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Associating ip Ip[xx.xx.xx.xx-1] to network 
> Ntwk[206|Guest|8] 2019-02-06 10:21:07,419 DEBUG 
> [c.c.r.ResourceLimitManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a 
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Updating resource Type = 
> public_ip count for Account = 2 Operation = increasing Amount = 1 
> 2019-02-06 10:21:07,426 DEBUG [c.c.n.IpAddressManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Successfully associated ip address xx.xx.xx.xx to 
> network Ntwk[206|Guest|8] 2019-02-06 10:21:07,439 DEBUG 
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a 
> job-876 ctx-462aefc2) (logid:ce8dc8fb) Complete async job-876, 
> jobStatus: SUCCEEDED, resultCode: 0, result: 
> org.apache.cloudstack.api.response.IPAddressResponse/ipaddress/{"id":"a710fbad-c2e5-409a-bd97-884e2d0ab7a5","ipaddress":"xx.xx.xx.xx","allocated":"2019-02-06T10:21:07+0100","zoneid":"f8d175fa-e4cc-4259-8c44-f88d7d429107","zonename":"somezone","issourcenat":false,"account":"admin","domainid":"165f090c-ffa3-11e8-bcb0-c8cbb8cb15cd","domain":"ROOT","forvirtualnetwork":true,"vlanid":"db86f7ae-6285-4beb-8e11-30e2a516e7e4","vlanname":"vlan://untagged","isstaticnat":false,"issystem":false,"associatednetworkid":"bc0cc3a2-e21f-4904-a826-04477b1805c0","associatednetworkname":"net-name","networkid":"0b8de733-b587-472b-b2c9-6944f45410d6","state":"Allocating","physicalnetworkid":"5cdff050-9cdc-4ffb-aa1f-78d2eeb7e28b","tags":[],"isportable":false,"fordisplay":true}

> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Publish async job-876 complete on message bus 
> 2019-02-06 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Wake up jobs related to job-876 2019-02-06 
> 10:21:07,441 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Update db status for job-876 2019-02-06 10:21:07,442 
> DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876 ctx-462aefc2) 
> (logid:ce8dc8fb) Wake up jobs joined with job-876 and disjoin all 
> subjobs created from job- 876 2019-02-06 10:21:07,446 DEBUG 
> [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-6:ctx-5d20ed8a 
> job-876) (logid:ce8dc8fb) Done executing 
> org.apache.cloudstack.api.command.admin.address.AssociateIPAddrCmdByAdmin 
> for job-876 2019-02-06 10:21:07,447 INFO [o.a.c.f.j.i.AsyncJobMonitor] 
> (API-Job-Executor-6:ctx-5d20ed8a job-876) (logid:ce8dc8fb) Remove 
> job-876 from job monitoring
>
>


-- 

*Ugo Vasi* / System Administrator
ugo.vasi@procne.it <mailto:ugo.vasi@procne.it>




*Procne S.r.l.*
+39 0432 486 523
via Cotonificio, 45
33010 Tavagnacco (UD)
www.procne.it <http://www.procne.it/>


Le informazioni contenute nella presente comunicazione ed i relativi 
allegati possono essere riservate e sono, comunque, destinate 
esclusivamente alle persone od alla Società sopraindicati. La 
diffusione, distribuzione e/o copiatura del documento trasmesso da parte 
di qualsiasi soggetto diverso dal destinatario è proibita sia ai sensi 
dell'art. 616 c.p., che ai sensi del Decreto Legislativo n. 196/2003 
"Codice in materia di protezione dei dati personali". Se avete ricevuto 
questo messaggio per errore, vi preghiamo di distruggerlo e di informare 
immediatamente Procne S.r.l. scrivendo all' indirizzo e-mail 
info@procne.it <mailto:info@procne.it>.


Mime
View raw message