cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Caviglione <c.alessan...@gmail.com>
Subject Re: Fresh 4.4.2 - Unable to get status on .... my XenServer
Date Sun, 07 Dec 2014 15:24:46 GMT
Yes, I do not believe that the issue is related to a host already
assigned...
Me too have this issue, CS4.4.1 and XS 6.2 hosts Zone installed from
scratch, "Unable to get status on..." error appear in management server
logs from the beginning.
However, I can work with my hosts, create instance, template, networks and
so on...
I can think that this is a bug... :-/

On Wed, Dec 3, 2014 at 4:50 PM, Matthew Midgett <
cloudstck@trick-solutions.com.invalid> wrote:

> I just finished installing XenServer again and started from a setup and I'm
> still getting the same errors that I was before about not being able to get
> a status. What to do now?
>
>
> JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ACw
>
> AAAfd0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk
>
> 1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQ
>
> B-AAsAAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ2
>
> 9udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AA
>
> sAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bH
>
> RNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfg
>
> ALAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdW
>
> x0TWFuYWdlZENvbnRleHRxAH4AGnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ACwAAADVxAH4AHXEAfg
>
> AadAAOcnVuV2l0aENvbnRleHRzcQB-AAsAAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbm
>
> FnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-ABZxAH4AF3NxAH4ACwAAAcxxAH
>
> 4AEXEAfgAScQB-ABdzcQB-AAsAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJF
>
> J1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-ABtzcQB-AAsAAAEGdAAfamF2YS51dG
>
> lsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgAXc3EAfgALAAAEeX
>
> QAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeG
>
> VjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgALAAACZ3QALmphdmEudXRpbC5jb25jdXJyZW50Ll
>
> RocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4ALHEAfgAXc3EAfgALAAAC6XQAEGphdmEubGFuZy
>
> 5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgAXc3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZG
>
> lmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAZ4cgAsamF2YS51dGlsLkNvbGxlY3Rpb2
>
> 5zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbG
>
> xlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAH
> cEAAAAAHhxAH4AOHg
> 2014-12-03 10:28:23,558 WARN  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-2d3ea815) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@7268fbbc: UPDATE
> op_dc_link_local_ip_address_alloc SET
> op_dc_link_local_ip_address_alloc.nic_id=123,
>
> op_dc_link_local_ip_address_alloc.reservation_id=_binary'97c33776-2f60-494b-
> 8f3c-7b20a0d7881f', op_dc_link_local_ip_address_alloc.taken='2014-12-03
> 15:28:23' WHERE op_dc_link_local_ip_address_alloc.id = 1806
> 2014-12-03 10:28:23,562 INFO  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-2d3ea815) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 13, will recycle it and start a new one
> 2014-12-03 10:28:23,697 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
> vm_instance13 and thread id 819805219
> 2014-12-03 10:28:37,991 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-077ec0bf) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:28:56,362 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-30:ctx-7833f032) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:29:37,990 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-40da93df) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:29:52,876 WARN  [o.a.c.f.j.i.AsyncJobMonitor]
> (Timer-1:ctx-db89f3ef) Task (job-393) has been pending for
> 90 seconds
> 2014-12-03 10:29:56,361 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-61:ctx-80573bda) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:30:02,536 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,560 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@571a2d18: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,568 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:30:02,570 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,693 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:30:02,721 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,721 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393 ctx-21971f3e) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,722 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Unable to complete
> AsyncJobVO {id:393, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:28:21 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-61:ctx-927aa31e job-388/job-393) Complete async job-393,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDc4NWRlMjlidXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:30:02,768 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-8e76fad8)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:02,770 WARN  [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-8e76fad8) Unexpected exception Job failed due to
> exception Resource [Host:1] is unreachable: Host 1: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@785de29b
> 2014-12-03 10:30:02,861 WARN  [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-c282e93d) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:30:05,371 INFO  [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Unable to
> acquire lock on VMTemplateStoragePool 54
> 2014-12-03 10:30:05,375 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Failed to
> start instance VM[SecondaryStorageVm|s-13-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,527 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,551 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394 ctx-55123741) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,551 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Unable to complete
> AsyncJobVO {id:394, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:28:23 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 54
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,553 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-62:ctx-21f67d3c job-384/job-394) Complete async job-394,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1NHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:30:05,596 WARN  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-00d2f7e3) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 54
> 2014-12-03 10:30:05,597 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:05,599 INFO  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-00d2f7e3) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 13, will recycle it and start a new one
> 2014-12-03 10:30:05,663 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-00d2f7e3) Was unable to find lock for the key
> vm_instance13 and thread id 819805219
> 2014-12-03 10:30:07,062 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-00d2f7e3)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:30:07,459 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-5b27f0f0) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:30:37,990 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-31:ctx-37ebcabd) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:30:56,361 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-62:ctx-f52b8d19) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:31:37,991 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-4133ae03) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:31:46,486 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,510 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@311d349d: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,517 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:31:46,519 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,617 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,642 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396 ctx-6d4609dc) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,642 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Unable to complete
> AsyncJobVO {id:396, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:30:02 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,644 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-63:ctx-d682f9f9 job-388/job-396) Complete async job-396,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDI1ZTg0ZWVldXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:31:46,689 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-c282e93d)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:46,691 WARN  [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-c282e93d) Unexpected exception Job failed due to
> exception Resource [Host:1] is unreachable: Host 1: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@25e84eee
> 2014-12-03 10:31:46,788 WARN  [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-5c69b42f) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:31:49,395 INFO  [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Unable to
> acquire lock on VMTemplateStoragePool 57
> 2014-12-03 10:31:49,399 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Failed to
> start instance VM[SecondaryStorageVm|s-14-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,554 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:31:49,578 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,578 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400 ctx-2ee9073d) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,579 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Unable to complete
> AsyncJobVO {id:400, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:30:07 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 57
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,580 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-65:ctx-9e1b1988 job-384/job-400) Complete async job-400,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA1N3VyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:31:49,620 WARN  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-5b27f0f0) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 57
> 2014-12-03 10:31:49,622 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:49,624 INFO  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-5b27f0f0) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 14, will recycle it and start a new one
> 2014-12-03 10:31:49,624 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-5b27f0f0)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:31:49,781 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-bc09a930) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:31:56,361 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-32:ctx-853d1e1b) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:32:37,989 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-fefa154b) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:32:56,361 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-65:ctx-51ba659a) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:33:37,990 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-e578dc8f) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:33:47,948 DEBUG [o.a.c.s.i.s.TemplateObject]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
> process event and answer
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:47,970 DEBUG [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) failed to
> create template on storage
> java.lang.RuntimeException: InvocationTargetException when invoking RPC
> callback for command: copyBaseImageCallback
> Caused by: java.lang.reflect.InvocationTargetException
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to
> process event
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@37aaee74: SELECT
> template_spool_ref.id, template_spool_ref.pool_id,
> template_spool_ref.template_id, template_spool_ref.created,
> template_spool_ref.last_updated, template_spool_ref.download_pct,
> template_spool_ref.download_state, template_spool_ref.local_path,
> template_spool_ref.error_str, template_spool_ref.job_id,
> template_spool_ref.install_path, template_spool_ref.template_size,
> template_spool_ref.marked_for_gc, template_spool_ref.update_count,
> template_spool_ref.updated, template_spool_ref.state FROM
> template_spool_ref
> WHERE template_spool_ref.pool_id = 2  AND template_spool_ref.template_id =
> 1
> ORDER BY RAND() LIMIT 1
> Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:47,976 WARN  [o.a.c.s.d.ObjectInDataStoreManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Template 1
> is not found on storage pool 2, so no need to delete
> 2014-12-03 10:33:47,979 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Failed to
> start instance VM[ConsoleProxy|v-10-VM]
> com.cloud.utils.exception.CloudRuntimeException: can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,085 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 4
> 2014-12-03 10:33:48,112 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:4] is unreachable: Host 4: Unable to start instance due to
> can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,113 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402 ctx-c2e18fb0) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:4]
> is unreachable: Host 4: Unable to start instance due to can't find mapping
> in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,113 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Unable to complete
> AsyncJobVO {id:402, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:31:46 EST 2014}, job origin:388
> com.cloud.exception.AgentUnavailableException: Resource [Host:4] is
> unreachable: Host 4: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> Caused by: com.cloud.utils.exception.CloudRuntimeException: can't find
> mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,115 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-66:ctx-38c8d928 job-388/job-402) Complete async job-402,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QA2kpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDo0XSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCA0OiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIGNhbid0IGZpbmQgbWFwcGlu
>
> ZyBpbiBPYmplY3RJbkRhdGFTdG9yZSB0YWJsZSBmb3I6IG9yZy5hcGFjaGUuY2xvdWRzdGFjay5z
>
> dG9yYWdlLmltYWdlLnN0b3JlLlRlbXBsYXRlT2JqZWN0QDRjN2JjNzc0dXIAHltMamF2YS5sYW5n
>
> LlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARM
>
> AAhmaWxlTmFtZXEAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZt
>
> V29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2JzcQB-
>
> AAsAAAH3dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNK
>
> b2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5qYXZhdAAMcnVuSW5Db250ZXh0
>
> c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdl
>
> ZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuYWdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EA
>
> fgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZh
>
> dWx0TWFuYWdlZENvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNx
>
> AH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVm
>
> YXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1x
>
> AH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5hcGFjaGUuY2xvdWRzdGFjay5t
>
> YW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHM
>
> cQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB13QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9y
>
> cyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEu
>
> dXRpbC5jb25jdXJyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAA
>
> BHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWRQb29s
>
> RXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVu
>
> dC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3NxAH4ACwAAAul0ABBqYXZhLmxh
>
> bmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlTGlzdPwPJTG17I4QAgABTAAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0
>
> aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9D
>
> b2xsZWN0aW9uO3hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAA
> AAB3BAAAAAB4cQB-ADh4
> 2014-12-03 10:33:48,158 ERROR [c.c.u.d.DbUtil]
> (consoleproxy-1:ctx-5c69b42f)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:48,160 WARN  [c.c.v.SystemVmLoadScanner]
> (consoleproxy-1:ctx-5c69b42f) Unexpected exception Job failed due to
> exception Resource [Host:4] is unreachable: Host 4: Unable to start
> instance
> due to can't find mapping in ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> java.lang.RuntimeException: Job failed due to exception Resource [Host:4]
> is
> unreachable: Host 4: Unable to start instance due to can't find mapping in
> ObjectInDataStore table for:
> org.apache.cloudstack.storage.image.store.TemplateObject@4c7bc774
> 2014-12-03 10:33:48,251 WARN  [c.c.u.d.Merovingian2]
> (consoleproxy-1:ctx-a282718d) Was unable to find lock for the key
> vm_instance10 and thread id 117755041
> 2014-12-03 10:33:51,376 INFO  [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Unable to
> acquire lock on VMTemplateStoragePool 60
> 2014-12-03 10:33:51,380 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Failed to
> start instance VM[SecondaryStorageVm|s-14-VM]
> com.cloud.utils.exception.CloudRuntimeException: Unable to acquire lock on
> VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,508 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) VM state
> transitted from :Starting to Stopped with event: OperationFailedvm's
> original host id: null new host id: null host id before state transition: 1
> 2014-12-03 10:33:51,533 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Invocation
> exception, caused by: com.cloud.exception.AgentUnavailableException:
> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
> Unable to acquire lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,534 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403 ctx-05dc78c7) Rethrow
> exception com.cloud.exception.AgentUnavailableException: Resource [Host:1]
> is unreachable: Host 1: Unable to start instance due to Unable to acquire
> lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,534 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Unable to complete
> AsyncJobVO {id:403, userId: 1, accountId: 1, instanceType: null,
> instanceId:
> null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
>
> rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9p
>
> ZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAlj
>
> bHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQA
>
> EkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-
>
> AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01h
>
> cDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAE
>
> SgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAAB
>
> AAAAAAAAAAEAAAAAAAAADnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBw
> cHBwcHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
> 0, result: null,
> initMsid: 223162458780723, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Wed Dec 03 10:31:49 EST 2014}, job origin:384
> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
> unreachable: Host 1: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 60
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> acquire lock on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,536 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-67:ctx-b6aee29d job-384/job-403) Complete async job-403,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAmUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBSZXNvdXJjZSBbSG9zdDoxXSBpcyB1bnJlYWNoYWJsZTog
>
> SG9zdCAxOiBVbmFibGUgdG8gc3RhcnQgaW5zdGFuY2UgZHVlIHRvIFVuYWJsZSB0byBhY3F1aXJl
>
> IGxvY2sgb24gVk1UZW1wbGF0ZVN0b3JhZ2VQb29sOiA2MHVyAB5bTGphdmEubGFuZy5TdGFja1Ry
>
> YWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1l
>
> bnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5h
>
> bWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JE
>
> aXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QA
>
> P29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdl
>
> ckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAA
>
> ADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0
>
> UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0
>
> AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFn
>
> ZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABn
>
> dABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5h
>
> Z2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5y
>
> dW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-
>
> ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFi
>
> bGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29u
>
> Y3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2
>
> YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9y
>
> LmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFk
>
> UG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVh
>
> ZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJs
>
> ZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5t
>
> b2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlv
>
> bjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAA
> eHEAfgA4eA
> 2014-12-03 10:33:51,576 WARN  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-bc09a930) Exception while trying
> to start secondary storage vm
> java.lang.RuntimeException: Job failed due to exception Resource [Host:1]
> is
> unreachable: Host 1: Unable to start instance due to Unable to acquire lock
> on VMTemplateStoragePool: 60
> 2014-12-03 10:33:51,577 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:51,579 INFO  [o.a.c.s.SecondaryStorageManagerImpl]
> (secstorage-1:ctx-bc09a930) Unable to start secondary storage vm for
> standby
> capacity, secStorageVm vm Id : 14, will recycle it and start a new one
> 2014-12-03 10:33:51,609 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-bc09a930) Was unable to find lock for the key
> vm_instance14 and thread id 819805219
> 2014-12-03 10:33:52,060 ERROR [c.c.u.d.DbUtil] (secstorage-1:ctx-bc09a930)
> RELEASE_LOCK() throws exception
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
> link
> failure
>         at
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3520)
> Caused by: java.io.EOFException: Can not read response from server.
> Expected
> to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
> 2014-12-03 10:33:52,397 WARN  [c.c.u.d.Merovingian2]
> (secstorage-1:ctx-0a61fb70) Was unable to find lock for the key
> vm_instance15 and thread id 819805219
> 2014-12-03 10:33:56,362 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-66:ctx-580f5bb1) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:35:31,403 ERROR [c.c.c.ClusterManagerImpl] (main:null) Unable
> to ping management server at 172.16.0.69:9090 due to ConnectException
> java.net.ConnectException: Connection refused
> 2014-12-03 10:35:31,591 WARN  [c.c.c.ConsoleProxyManagerImpl] (main:null)
> Empty console proxy domain, explicitly disabling SSL
> 2014-12-03 10:35:31,993 WARN  [c.c.s.d.DownloadMonitorImpl] (main:null)
> Only
> realhostip.com ssl cert is supported, ignoring self-signed and other certs
> 2014-12-03 10:35:49,150 INFO  [c.c.h.x.r.XenServerConnectionPool]
> (main:null) XenServer Connection Pool Configs:
> sleep.interval.on.error=10000
> 2014-12-03 10:35:49,732 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-c81f933c) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> commit
> or close the connection.
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 10:35:49,750 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-4:ctx-ebb2805b) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@30b1baa6: UPDATE op_ha_work SET
> op_ha_work.taken='2014-12-03 15:35:49',
> op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 4
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 10:35:49,808 WARN  [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Mark job as failed as its correspoding queue-item has been discarded. job
> id: 405
> 2014-12-03 10:35:49,808 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Complete async job-405, jobStatus: FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,813 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> job-405 no longer exists, we just log completion info here. FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,850 DEBUG [c.c.h.CheckOnAgentInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Unable to reach the agent for
> VM[SecondaryStorageVm|s-15-VM]: Resource [Host:4] is unreachable: Host 4:
> Host with specified id is not in the
> right state: Disconnected
> 2014-12-03 10:35:49,863 WARN  [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Mark job as failed as its correspoding queue-item has been discarded. job
> id: 411
> 2014-12-03 10:35:49,863 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> Complete async job-411, jobStatus: FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,859 DEBUG [c.c.h.CheckOnAgentInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) Unable to reach the agent for
> VM[ConsoleProxy|v-10-VM]: Resource [Host:4] is unreachable: Host 4: Host
> with specified id is not in the right
> state: Disconnected
> 2014-12-03 10:35:49,885 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (main:null)
> job-411 no longer exists, we just log completion info here. FAILED,
> resultCode: 0, result: Execution was cancelled because of server shutdown
> 2014-12-03 10:35:49,901 DEBUG [c.c.h.UserVmDomRInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) Not a User Vm, unable to determine state
> of VM[ConsoleProxy|v-10-VM] returning null
> 2014-12-03 10:35:49,902 DEBUG [c.c.h.UserVmDomRInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Not a User Vm, unable to determine state
> of VM[SecondaryStorageVm|s-15-VM] returning null
> 2014-12-03 10:35:49,921 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
> (HA-Worker-2:ctx-b2a9dc97 work-1) unable to determine state of
> VM[SecondaryStorageVm|s-15-VM] returning null
> 2014-12-03 10:35:49,922 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 10:35:49,929 DEBUG [c.c.h.ManagementIPSystemVMInvestigator]
> (HA-Worker-1:ctx-415457e8 work-4) unable to determine state of
> VM[ConsoleProxy|v-10-VM] returning null
> 2014-12-03 10:35:49,930 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-1:ctx-415457e8 work-4) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 10:35:50,055 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-1:ctx-415457e8 work-4) Was unable to find lock for the
> key vm_instance10 and thread id 1329615500
> 2014-12-03 10:35:50,059 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Was unable to find lock for the
> key vm_instance15 and thread id 9330563
> 2014-12-03 10:35:50,849 ERROR [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-375d995b) Unable to ping management server at
> 172.16.0.70:9090 due to ConnectException
> java.net.ConnectException: Connection refused
> 2014-12-03 10:35:50,850 WARN  [c.c.c.ClusterManagerImpl]
> (Cluster-Heartbeat-1:ctx-375d995b) Management node 4 is detected inactive
> by
> timestamp and also not pingable
> 2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 10:35:52,166 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Unable to
> cleanup VM: VM[ConsoleProxy|v-10-VM] ,since outstanding work item is not
> found
> 2014-12-03 10:35:52,166 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state
> Starting
> 2014-12-03 10:35:52,167 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415 ctx-a901375d) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,168 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Unable to
> complete AsyncJobVO {id:415, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAACnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 10:35:50 EST 2014}, job origin:413
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,173 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-1:ctx-e4c35b10 job-413/job-415) Complete async job-415,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xMC1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 10:35:52,175 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-15-VM] ,since outstanding work item is
> not found
> 2014-12-03 10:35:52,175 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in
> state Starting
> 2014-12-03 10:35:52,176 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416 ctx-c11ef29e) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state
> Starting
> 2014-12-03 10:35:52,176 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Unable to
> complete AsyncJobVO {id:416, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 223162458780723, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 10:35:50 EST 2014}, job origin:412
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
> 2014-12-03 10:35:52,178 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:ctx-fd2be5c1 job-412/job-416) Complete async job-416,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 10:35:52,274 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-1:ctx-415457e8 work-4) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-10-VM] when it is in state Starting
> 2014-12-03 10:35:52,276 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-b2a9dc97 work-1) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-15-VM] when it is in state Starting
> 2014-12-03 10:35:52,394 WARN  [c.c.c.ClusterManagerImpl]
> (Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server leave
> event took 68 ms
> 2014-12-03 10:35:52,395 WARN  [o.a.c.alerts]
> (Cluster-Notification-1:ctx-e8ae6c9b)  alertType:: 14 // dataCenterId:: 0
> // podId:: 0 // clusterId:: null // message:: Management server node
> 172.16.0.69 is up
> 2014-12-03 10:35:52,418 WARN  [c.c.c.ClusterManagerImpl]
> (Cluster-Notification-1:ctx-e8ae6c9b) Notifying management server join
> event
> took 24 ms
> 2014-12-03 10:36:57,546 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-dada7b54) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:36:58,203 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-41be4ac5) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:37:57,548 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-85d0acb3) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:37:58,204 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-68aea530) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:38:09,839 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-3:ctx-c510e983 ctx-c0a51a4e) Received unknown parameters for
> command listRegions. Unknown parameters : listall
> 2014-12-03 10:38:18,800 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-13:ctx-0ca7247c ctx-7869b51b) Received unknown parameters
> for
> command listZones. Unknown parameters : listall
> 2014-12-03 10:38:18,987 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-18:ctx-85e4fffe ctx-819e52d4) Received unknown parameters
> for
> command listPods. Unknown parameters : listall
> 2014-12-03 10:38:19,118 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-17:ctx-cb8c1540 ctx-5a4a6061) Received unknown parameters
> for
> command listClusters. Unknown parameters : listall
> 2014-12-03 10:38:19,244 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-19:ctx-be72b381 ctx-d0874501) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:19,391 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-21:ctx-e2218038 ctx-eed3052d) Received unknown parameters
> for
> command listStoragePools. Unknown parameters : listall
> 2014-12-03 10:38:19,533 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-22:ctx-20552766 ctx-83dccf23) Received unknown parameters
> for
> command listImageStores. Unknown parameters : listall type
> 2014-12-03 10:38:19,655 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-24:ctx-968df022 ctx-d4de7d4e) Received unknown parameters
> for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:24,050 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-7:ctx-193b8617 ctx-c4ab4021) Received unknown parameters for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:24,557 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-11:ctx-0e796a3f ctx-6e24a560) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:26,590 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-3:ctx-c278bf56 ctx-e32c7268) Received unknown parameters for
> command listZones. Unknown parameters : listall
> 2014-12-03 10:38:26,693 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-12:ctx-1afb82ed ctx-051fbce8) Received unknown parameters
> for
> command listPods. Unknown parameters : listall
> 2014-12-03 10:38:26,781 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-14:ctx-aadb7b92 ctx-cb2ede4c) Received unknown parameters
> for
> command listClusters. Unknown parameters : listall
> 2014-12-03 10:38:26,878 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-15:ctx-96f8cd1d ctx-87a1426e) Received unknown parameters
> for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:27,032 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-16:ctx-3593bc62 ctx-bab3e4e6) Received unknown parameters
> for
> command listStoragePools. Unknown parameters : listall
> 2014-12-03 10:38:27,133 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-13:ctx-004ef7e9 ctx-528af3a7) Received unknown parameters
> for
> command listImageStores. Unknown parameters : listall type
> 2014-12-03 10:38:27,294 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-18:ctx-fee8e727 ctx-f817e3df) Received unknown parameters
> for
> command listSystemVms. Unknown parameters : listall
> 2014-12-03 10:38:28,720 WARN  [c.c.a.d.ParamGenericValidationWorker]
> (catalina-exec-5:ctx-4e425560 ctx-dc515ed7) Received unknown parameters for
> command listHosts. Unknown parameters : listall
> 2014-12-03 10:38:57,545 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-3c160491) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:38:58,202 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-f2f80cb3) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:39:57,545 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-954a2d0f) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:39:58,202 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-1a58d0d9) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:40:57,545 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-353674d2) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:40:58,203 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-9:ctx-3761b53b) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:41:57,547 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-c0120a72) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:41:58,203 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-10:ctx-f1fdf064) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:42:57,546 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-efcc44be) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:42:58,203 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-9f51f8e3) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:43:57,547 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-3e62beae) Unable to get current status on
> 1(xenhv2.pvdc.chlt.charlottecolo.com)
> 2014-12-03 10:43:58,204 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-13:ctx-3893f12e) Unable to get current status on
> 4(xenhv1.pvdc.chlt.charlottecolo.com)
> [root@CSTW1 ~]#
>
>
>
> -----Original Message-----
> From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
> Sent: Wednesday, December 03, 2014 6:39 AM
> To: users@cloudstack.apache.org
> Subject: RE: Fresh 4.4.2 - Unable to get status on .... my XenServer
>
> Thanks Geoff I really appreciate the response. I guess that every other
> time
> over the last year I've reinstalled XenServer and didn't remember it. Maybe
> we should add this to the Docs as I don't ever remember seeing it, I could
> be wrong though. I do have the patch management automated and it checks in
> on every reboot. I will have to use Ilo to reinstall the OS today which is
> not that big of a deal. I do admit that I need to PXE boot the servers I
> just hadn't had the time. Yesterday I spent working out a builder for CS
> and
> a network mirror. Then I noticed about 3 hours later that you guys posted a
> public repo for 4.4.2. :)
>
>
> CONFIDENTIAL: This electronic mail, including any attachment(s) may contain
> information that is privileged, confidential, and/or otherwise protected
> from disclosure to anyone other than its intended recipient(s).  Any
> dissemination or use of this electronic email or its contents (including
> any
> attachments) by persons other than the intended recipient(s) is strictly
> prohibited.  If you have received this message in error, please notify me
> immediately by reply email so that I may correct my internal records.
> Please then delete the original message (including any attachments) in its
> entirety.  Thank you.
>
> -----Original Message-----
> From: Matthew Midgett [mailto:cloudstck@trick-solutions.com.INVALID]
> Sent: Wednesday, December 03, 2014 1:11 AM
> To: users@cloudstack.apache.org
> Subject: Fresh 4.4.2 - Unable to get status on .... my xenservers
>
> First I am not sure what is causing this, I have been testing 4.3  and 4.4
> multiple times and didn't notice this problem till middle of last month
> when
> I processed all xenserver updates and then tried to go to 4.4.1. I was
> getting the same error that I am unable to get status of my xenservers. I
> wiped out everything but the xenservers, those I just removed from the pool
> and let them reboot and added them to a new pool. In ACS it says its trying
> to start the SSVM and the Console VM but they stay in the starting state.
> I'm sure it is because of this error. The agent show up in ACS so I don't
> know why it can't talk to the xenservers. I can ping them from the
> management servers.
>
> Last login: Wed Dec  3 00:57:04 2014 from 192.168.255.6
> [root@CSTW1 ~]# grep -i -E 'exception|unable|fail|invalid|leak|warn|error'
> /var/log/cloudstack/management/management-server.log
> 2014-12-03 00:00:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-a0a23e42) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:00:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-f78a59e4) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:01:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-11:ctx-2e791a18) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:01:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-7bca20e8) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:02:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-28d48099) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:02:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-bc593a63) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:03:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-615b171d) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:03:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-07f6ba50) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:04:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-13:ctx-c2287fb5) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:04:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-26:ctx-f7d92459) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:05:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-14:ctx-a4e62e4e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:05:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-28:ctx-01bb91e3) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:06:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-30f576ea) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:06:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-29:ctx-4d6690de) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:07:27,540 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-645639e7) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: DB Exception
> on:
> com.mysql.jdbc.JDBC4PreparedStatement@32602693: UPDATE op_ha_work SET
> op_ha_work.taken='2014-12-03 05:07:27',
> op_ha_work.mgmt_server_id=223162458780723 WHERE op_ha_work.id = 5
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:07:27,595 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-3:ctx-c7c94958 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:07:27,645 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-3:ctx-c7c94958 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 818653380
> 2014-12-03 00:07:29,425 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Vm-Operations-Cleanup-1:ctx-35d886d3) VM Operations failed due to
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.JDBC4PreparedStatement@61807bba: DELETE FROM async_job
> WHERE
> async_job.id= 401
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:07:30,702 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-c7c94958 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:07:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-c7dd62d6) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:07:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-30:ctx-99d14f4c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:08:27,621 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-94cc502e work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:08:27,678 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-94cc502e work-2) Was unable to find lock for the
> key vm_instance15 and thread id 52609881
> 2014-12-03 00:08:29,421 ERROR [c.c.v.VirtualMachineManagerImpl]
> (Vm-Operations-Cleanup-1:ctx-ed5f2ff6) VM Operations failed due to
> com.cloud.utils.exception.CloudRuntimeException: DB Exception on:
> com.mysql.jdbc.PreparedStatement@15fe9d6d: DELETE FROM async_job WHERE
> async_job.id= 406
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Cannot delete or update a parent row: a foreign key constraint fails
> (`cloud`.`async_job_join_map`, CONSTRAINT
> `fk_async_job_join_map__join_job_id` FOREIGN KEY (`join_job_id`) REFERENCES
> `async_job` (`id`))
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1040)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:08:30,736 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-94cc502e work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:08:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-acc87f49) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:08:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-32:ctx-efd0036e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:09:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-1d3eecd0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:09:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-3271eeea) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:10:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-17:ctx-27637ed9) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:10:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-34:ctx-b1f98667) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:11:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-18:ctx-45760fd3) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:11:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-36:ctx-3367d63d) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:12:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-ee2683aa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:12:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-37:ctx-d0b02603) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:13:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-19:ctx-ce10b0fa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:13:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-38:ctx-e7e4ae6b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:14:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-20:ctx-c158dd2c) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:14:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-40:ctx-7a662bea) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:15:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-0a8f1c90) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:15:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-41:ctx-4c163515) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:16:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-51c2dfd8) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:16:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-42:ctx-c670489b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:17:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-9df53feb) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:17:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-44:ctx-a921b13b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:18:29,546 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-1:ctx-4fdb5161 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:18:29,582 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-1:ctx-4fdb5161 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 2132920636
> 2014-12-03 00:18:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-a54d1732) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:18:32,638 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-1:ctx-4fdb5161 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:18:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-45:ctx-5d1c0c7e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:19:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 00:19:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 00:19:30,872 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411 ctx-959fb603) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:19:30,872 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Unable to
> complete AsyncJobVO {id:411, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:19:02 EST 2014}, job origin:408
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:19:30,874 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-3:ctx-d59db286 job-408/job-411) Complete async job-411,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:19:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-23:ctx-0414dcdd) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:19:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-46:ctx-fc02b226) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:20:32,282 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-821b5aca) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:20:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-48:ctx-70dbfde8) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:21:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-de7b1636) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:21:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-49:ctx-154bac45) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:22:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-5c37a417) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:22:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-50:ctx-a34d4acf) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:23:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-52:ctx-4ee4cdd0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:23:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-7:ctx-de025548) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:24:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-53:ctx-198b066a) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:24:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-27:ctx-e4b78a1b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:25:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-14:ctx-d0466cfa) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:25:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-55:ctx-6c330223) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:26:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-2:ctx-98d6b222) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:26:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-57:ctx-031552f0) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:27:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-29:ctx-a436d05f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:27:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-58:ctx-0f3496a2) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:28:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-15:ctx-40cf6a78) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:28:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-59:ctx-1537f233) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:29:30,874 ERROR [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-0:ctx-84af6112) Caught this throwable,
> com.cloud.utils.exception.CloudRuntimeException: Unable to execute take
> Caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to
> commit
> or close the connection.
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock
> found when trying to get lock; try restarting transaction
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1065)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
> 2014-12-03 00:29:30,881 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:29:30,882 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
> not found
> 2014-12-03 00:29:30,882 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
> state Starting
> 2014-12-03 00:29:30,883 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417 ctx-081f7cae) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
> Starting
> 2014-12-03 00:29:30,883 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Unable to
> complete AsyncJobVO {id:417, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:29:02 EST 2014}, job origin:414
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:29:30,896 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-4:ctx-b25792e9 job-414/job-417) Complete async job-417,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:29:30,919 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-4:ctx-10fea75f work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:29:30,967 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-4:ctx-10fea75f work-2) Was unable to find lock for the
> key vm_instance15 and thread id 139852089
> 2014-12-03 00:29:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-8:ctx-bc77e0da) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:29:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-61:ctx-de44c2a9) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:29:34,025 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-4:ctx-10fea75f work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:30:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-31:ctx-8417fc60) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:30:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-62:ctx-d118067c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:31:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-16:ctx-393a2cde) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:31:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-63:ctx-fd5d5e54) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:32:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-3:ctx-07ab5550) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:32:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-65:ctx-0345383e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:33:32,286 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-33:ctx-60af4723) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:33:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-66:ctx-0b95563b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:34:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-17:ctx-8e9c36a2) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:34:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-67:ctx-397176fb) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:35:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-9:ctx-3c967d9b) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:35:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-69:ctx-8bc1cfaf) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:36:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-35:ctx-8186e45f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:36:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-70:ctx-557403b1) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:37:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-18:ctx-8539dc77) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:37:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-71:ctx-61ea3221) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:38:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-4:ctx-0dc9a587) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:38:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-73:ctx-7fba328c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:39:31,006 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-0:ctx-7a356c50 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:39:31,051 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-0:ctx-7a356c50 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 84459079
> 2014-12-03 00:39:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-37:ctx-bc7a8f81) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:39:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-74:ctx-ebe4657b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:39:34,105 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-0:ctx-7a356c50 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:40:30,865 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:40:30,866 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 00:40:30,866 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426 ctx-a9b4ebed) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:40:30,866 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Unable to
> complete AsyncJobVO {id:426, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:40:02 EST 2014}, job origin:423
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:40:30,868 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-5:ctx-32edffa5 job-423/job-426) Complete async job-426,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:40:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-19:ctx-338898ef) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:40:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-75:ctx-790ca221) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:41:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-10:ctx-0e2f52eb) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:41:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-77:ctx-f8287f50) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:42:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-39:ctx-1af09356) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:42:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-78:ctx-e1b2675f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:43:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-20:ctx-c9aea423) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:43:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-79:ctx-c459450e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:44:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-5:ctx-f7fee74e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:44:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-81:ctx-bc615860) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:45:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-41:ctx-4f287927) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:45:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-82:ctx-092eb488) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:46:27,281 WARN  [o.a.c.alerts] (HA-1:ctx-b34246c5)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 00:46:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-21:ctx-62eca8f2) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:46:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-83:ctx-272e6406) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:47:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-11:ctx-23f28317) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:47:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-85:ctx-905ed14b) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:48:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-43:ctx-4a673855) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:48:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-86:ctx-1285bc0f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:49:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-22:ctx-60a35d36) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:49:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-87:ctx-687282ce) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 00:50:30,871 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Unable to
> cleanup VM: VM[SecondaryStorageVm|s-22-VM] ,since outstanding work item is
> not found
> 2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in
> state Starting
> 2014-12-03 00:50:30,871 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432 ctx-9392cc16) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state
> Starting
> 2014-12-03 00:50:30,871 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Unable to
> complete AsyncJobVO {id:432, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAFnQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 00:50:02 EST 2014}, job origin:429
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 00:50:30,873 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-6:ctx-46fe9139 job-429/job-432) Complete async job-432,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAe0pvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtTZWNvbmRhcnlTdG9yYWdlVm18cy0yMi1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFy
>
> dGluZ3VyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNy
>
> ABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5k
>
> ZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAA
>
> cnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hl
>
> ci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdv
>
> cmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwu
>
> amF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFu
>
> YWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVu
>
> bmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdl
>
> ZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENv
>
> bnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFn
>
> ZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250
>
> ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcu
>
> YXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVx
>
> AH4AFnEAfgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwu
>
> Y29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4A
>
> G3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFz
>
> ay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4
>
> ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAu
>
> amF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdz
>
> cQB-AAsAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51
>
> dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhy
>
> ACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgAB
>
> TAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHS
> HZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 00:50:31,095 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 00:50:31,129 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Was unable to find lock for the
> key vm_instance15 and thread id 818653380
> 2014-12-03 00:50:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-6:ctx-2758188b) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:50:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-89:ctx-eb95a1c2) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:50:34,182 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-3:ctx-0dbe91b2 work-2) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 00:51:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-45:ctx-7be8bae6) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:51:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-90:ctx-3f76c1d7) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:52:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-23:ctx-e902ca80) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:52:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-91:ctx-4d56948f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:53:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-12:ctx-0876aaa4) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:53:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-93:ctx-c057ee22) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:54:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-47:ctx-8083fccf) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:54:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-94:ctx-56cebf3c) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:55:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-24:ctx-36c965de) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:55:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-95:ctx-86b1db16) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:56:27,241 WARN  [o.a.c.alerts] (HA-2:ctx-627e96a3)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 00:56:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-1:ctx-6000a728) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:56:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-97:ctx-ffbe388e) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:57:32,279 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-49:ctx-a54e4fa0) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:57:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-98:ctx-34e48d1f) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:58:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-25:ctx-ffdd759e) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:58:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-99:ctx-0c1a9fa7) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 00:59:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-101:ctx-8442d937) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 00:59:32,963 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-51:ctx-9bd2b3cc) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:00:31,170 WARN  [o.a.c.f.j.AsyncJobExecutionContext]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Job is executed without a context, setup
> psudo job for the executing thread
> 2014-12-03 01:00:31,208 WARN  [c.c.u.d.Merovingian2]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Was unable to find lock for the
> key vm_instance22 and thread id 52609881
> 2014-12-03 01:00:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-102:ctx-138c6123) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:00:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-26:ctx-3e13656a) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:00:34,263 WARN  [c.c.h.HighAvailabilityManagerImpl]
> (HA-Worker-2:ctx-f17ce3b5 work-5) Encountered unhandled exception during HA
> process, reschedule retry
> java.lang.RuntimeException: Job failed due to exception Work item not
> found,
> We cannot stop VM[SecondaryStorageVm|s-22-VM] when it is in state Starting
> 2014-12-03 01:01:32,280 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-103:ctx-334d11fc) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
> transition the state but we're moving on because it's forced stop
> 2014-12-03 01:01:32,856 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Unable to
> cleanup VM: VM[ConsoleProxy|v-15-VM] ,since outstanding work item is not
> found
> 2014-12-03 01:01:32,856 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Invocation
> exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Work
> item not found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state
> Starting
> 2014-12-03 01:01:32,857 INFO  [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438 ctx-70e24e89) Rethrow
> exception com.cloud.utils.exception.CloudRuntimeException: Work item not
> found, We cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 01:01:32,857 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Unable to
> complete AsyncJobVO {id:438, userId: 1, accountId: 1, instanceType: null,
> instanceId: null, cmd: com.cloud.vm.VmWorkStop, cmdInfo:
>
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNj
>
> b20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlk
>
> TAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAABAAAAAAAAAAEAAAAA
> AAAAD3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwB, cmdVersion: 0, status:
> IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid:
> 68415445678895, completeMsid: null, lastUpdated: null, lastPolled: null,
> created: Wed Dec 03 01:01:03 EST 2014}, job origin:429
> com.cloud.utils.exception.CloudRuntimeException: Work item not found, We
> cannot stop VM[ConsoleProxy|v-15-VM] when it is in state Starting
> 2014-12-03 01:01:32,865 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-7:ctx-0a1f299d job-429/job-438) Complete async job-438,
> jobStatus: FAILED, resultCode: 0, result:
>
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFu
>
> Zy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARM
>
> AAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9s
>
> YW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7
>
> TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAdUpvYiBm
>
> YWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBXb3JrIGl0ZW0gbm90IGZvdW5kLCBXZSBjYW5ub3Qgc3Rv
>
> cCBWTVtDb25zb2xlUHJveHl8di0xNS1WTV0gd2hlbiBpdCBpcyBpbiBzdGF0ZSBTdGFydGluZ3Vy
>
> AB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZh
>
> LmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJp
>
> bmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNv
>
> bS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZh
>
> dAAGcnVuSm9ic3EAfgALAAAB93QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9i
>
> cy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQA
>
> DHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5j
>
> b250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUu
>
> amF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250
>
> ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQu
>
> amF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29u
>
> dGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EA
>
> fgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hl
>
> LmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEA
>
> fgAXc3EAfgALAAABzHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3Vy
>
> cmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4A
>
> CwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZh
>
> cQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9y
>
> dAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51
>
> dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsA
>
> AALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNv
>
> bGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZh
>
> LnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QA
>
> FkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0D
> AAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-12-03 01:01:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-52:ctx-76332a49) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:02:32,282 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-105:ctx-b73de834) Unable to get current
> status on 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:02:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-53:ctx-9b938dc5) Unable to get current status on
> 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:03:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-27:ctx-28735fd7) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:03:32,967 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-107:ctx-e290c580) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:04:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-54:ctx-a53679d7) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:04:32,964 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-108:ctx-e1d3829f) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:05:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-55:ctx-65e26d72) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:05:32,965 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-110:ctx-b755e6ac) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> 2014-12-03 01:06:27,239 WARN  [o.a.c.alerts] (HA-3:ctx-b686e09d)
> alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null //
> message:: No usage server process running
> 2014-12-03 01:06:32,281 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-28:ctx-230f4b8f) Unable to get current status on
> 11(xenhv1-c1.pvdc.chlt.CharlotteColo.com)
> 2014-12-03 01:06:32,966 WARN  [c.c.a.m.DirectAgentAttache]
> (DirectAgentCronJob-111:ctx-77cd282d) Unable to get current
> status on 8(xenhv2-c1.pvdc.chlt.charlottecolo.com)
> [root@CSTW1 ~]#
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message