cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From cs user <acldstk...@gmail.com>
Subject Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
Date Thu, 01 Oct 2015 13:58:56 GMT
Hello,

Found a better error in the logs:

ERROR [c.c.a.t.Request] (AgentManager-Handler-11:null) Caught problem with
[{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":26631,"name":"i-2-26631-VM","bootloader":"PyGrub","type":"Us
er","cpus":1,"minSpeed":1200,"maxSpeed":1200,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.6 (64-bit)","platformEmulator":"CentOS 5
(64-bit)","bootArgs":"","enableHA":false,"
limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"","params":{"memoryOvercommitRatio":"1","platform":"viridian:true;acpi:1;apic:true;pae:true;nx:t
rue","hypervisortoolsversion":"xenserver56","cpuOvercommitRatio":"4"},"uuid":"b37e76e8-e19a-4100-bf1c-1ee8eba674f2","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"8f51
e73f-9bfc-4a37-bf86-664067a2cf40","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b114d58e-a37a-6433-c0ff-07f3613a3c68","id":219,"poolType":"LVM","ho
st":"192.168.97.2","path":"lvm","port":0,"url":"LVM://
192.168.97.2/lvm/?ROLE\u003dPrimary\u0026STOREUUID\u003db114d58e-a37a-6433-c0ff-07f3613a3c68
"}},"name":"ROOT-26631","size":21474836480,"path":"48
fe494b-ee02-4f08-b7aa-975baecd7b3e","volumeId":119075,"vmName":"i-2-26631-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":119075,"deviceId":0,"cacheMode":"NONE","hypervisorType":"Xe
nServer"}},"diskSeq":0,"path":"48fe494b-ee02-4f08-b7aa-975baecd7b3e","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"192.168.97.2","volumeSize":"21474836480"}},{"data":{
"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":2000,"defaultNic":true,"pxe
Disable":false,"nicUuid":"aaef8c14-784c-403f-91fc-b6a3167f2595","uuid":"508d0384-5bf0-4434-8155-0b5bb1b33b43","ip":"192.168.8.112","netmask":"255.255.254.0","gateway":"192.168.9.254","mac":"06:68:80:
00:00:7f","dns1":"192.168.0.224","dns2":"192.168.0.224","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isSecurityGroupEnabled":true,"name":"CS-Guest-Public"}],"vcpuMaxLimit":
16},"volumeToFiler":{"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]":{"id":223,"uuid":"bc01ca2b-e6d8-a856-27b5-6f6b55a06b1d","host":"192.168.97.1","path":"lvm","port":0,"type":"L
VM"}},"contextMap":{"job":"job-347068/job-347069"},"wait":0}}]
com.google.gson.JsonParseException: Expecting object found:
"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
        at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
        at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
        at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
        at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
        at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
        at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:521)
        at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
        at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
        at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at
com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.cloud.agent.transport.Request.getCommands(Request.java:250)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
        at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
        at com.cloud.utils.nio.Task.run(Task.java:84)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
WARN  [c.c.u.n.Task] (AgentManager-Handler-11:null) Caught the following
exception but pushing on
com.google.gson.JsonParseException: Expecting object found:
"Vol[119075|ROOT|48fe494b-ee02-4f08-b7aa-975baecd7b3e|21474836480]"
        at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
        at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
        at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
        at
com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
        at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at
com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
        at
com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:521)
        at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
        at
com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
        at
com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at
com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at
com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
        at
com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
        at
com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.cloud.agent.transport.Request.getCommands(Request.java:250)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
        at
com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:666)
        at com.cloud.utils.nio.Task.run(Task.java:84)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)



Not sure if that is more helpful?

Cheers!


On Thu, Oct 1, 2015 at 10:02 AM, cs user <acldstkusr@gmail.com> wrote:

> Hi Koushik,
>
> We've managed to remove the error now regarding host id 27708.... this was
> related to a very old secondary storage vm which was in a status of
> disconnected but had not been removed. It had long since been deleted.
>
> However, we are still seeing the error, it is very similar to what is
> described here:
>
> https://issues.apache.org/jira/browse/CLOUDSTACK-3715
>
> Still seeing this:
>
> ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
> (Work-Job-Executor-6:ctx-8e19de38 job-346708/job-346709 ctx-65bccb97) Error
> while migrating vm VM[User|i-2-26631-VM] to host Host[-181-Routing]
> com.cloud.exception.OperationTimedoutException: Commands
> 964614745187418300 to Host 181 timed out after 1200
>
> The migration never happens, its as though the call to the xen host is
> never made, or never received by xen.
>
> What ports are used to make this call? I assume either 22 or 443?
>
> Cheers!
>
>
>
> On Wed, Sep 30, 2015 at 12:50 PM, Koushik Das <koushik.das@citrix.com>
> wrote:
>
>> As per the logs the host id 27708 is not in the right state and operation
>> is failing. Check in DB for the status of host id 27708.
>> Also grep for job-343376 and job-343377 in all the MS logs to trace in
>> which MS the job originated and where it is getting executed.
>>
>> -----Original Message-----
>> From: cs user [mailto:acldstkusr@gmail.com]
>> Sent: Wednesday, 30 September 2015 16:01
>> To: users@cloudstack.apache.org
>> Subject: Re: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>>
>> Hi Koushik,
>>
>> We see errors such as :
>>
>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>> the right state: Disconnected ERROR
>> [o.a.c.s.m.XenServerStorageMotionStrategy]
>> (Work-Job-Executor-159:ctx-6e40204e job-343376/job-343377 ctx-818f1a51)
>> copy failed ERROR [c.c.v.VmWorkJobHandlerProxy]
>> (Work-Job-Executor-159:ctx-6e40204e
>> job-343376/job-343377 ctx-818f1a51) Invocation exception, caused by:
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> INFO  [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-159:ctx-6e40204e
>> job-343376/job-343377 ctx-818f1a51) Rethrow exception
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>> VM[User|i-2-26383-VM] along with its volumes.
>> com.cloud.utils.exception.CloudRuntimeException: Failed to send command,
>> due to Agent:27708, com.cloud.exception.AgentUnavailableException: Resource
>> [Host:27708] is unreachable: Host 27708: Host with specified id is not in
>> the right state: Disconnected
>>
>> However, when I look at the hosts in cloudstack, all hosts are connected
>> and green, none are reported by cloudstack as being disconnected. So I'm
>> not really quite sure what is going wrong. As I say, when only one
>> cloudstack management server is running, everything works perfectly fine.
>>
>> Cheers
>>
>> On Wed, Sep 30, 2015 at 10:27 AM, Koushik Das <koushik.das@citrix.com>
>> wrote:
>>
>> > Check logs on all 4 MS in the cluster around the time the operation
>> > was done. If you see any error/exceptions that can point to the problem.
>> >
>> > -----Original Message-----
>> > From: cs user [mailto:acldstkusr@gmail.com]
>> > Sent: Tuesday, 29 September 2015 20:53
>> > To: users@cloudstack.apache.org
>> > Subject: 4.5.1 - Xen 6.5 SP1 - Live migrations failing - Local disk
>> >
>> > Hi Folks,
>> >
>> > We are seeing some problems when attempting live migrations in a
>> > clustered cloudstack environment. We have 4 management servers, and
>> > when we attempt a migration of an instance (local storage) from one
>> > host to another, the migration times out.
>> >
>> > However, if we only have one management server running, the migration
>> > works fine and we can move the instance around between hosts in the
>> > same pod without any problems at all.
>> >
>> > Just wanted to know if anyone else had come across this issue?
>> >
>> > Cheers
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message