cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kurt K <k...@ihnetworks.com>
Subject Re: Getting errors while adding ceph storage to cloudstack
Date Thu, 23 Mar 2017 11:03:41 GMT
Hi,

We are facing a problem while creating new instance. On checking the 
logs, it shows the below errors. It looks like the error is related to 
Domain router VM.


=====
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is 
unreachable: Host 1: Unable to start instance due to Unable to start 
VM[DomainRouter|r-9-VM] due to error in finalizeStart, not retrying

Resource [Host:1] is unreachable: Host 1: Unable to start instance due 
to Unable to start VM[DomainRouter|r-14-VM] due to error in 
finalizeStart, not retrying

com.cloud.exception.InsufficientServerCapacityException: Unable to 
create a deployment for VM[User|i-2-8-VM]Scope=interface 
com.cloud.dc.DataCenter; id=1

=====

Please let us know your thoughts

-kurt


On 03/22/2017 05:42 PM, Kurt K wrote:
> Hi,
>
> We have successfully added ceph storage to cloud stack. But we still 
> have a doubt regarding creating VMs via ceph primary.
>
> Any idea about the VM's data store location in ceph primary?
>
> /-Kurt/
>
> On 02/23/2017 11:20 AM, Kurt K wrote:
>> Hi Simon,
>>
>> Thanks for the reply.
>>
>> >> Also run ceph health and make sure your agent can talk to your 
>> ceph monitors.
>>
>> ceph health status shows fine and also we can connect our OSD's from 
>> monitor server. Snippets pasted below.
>>
>> ===
>> [root@mon1 ~]# ceph -s
>>     cluster ebac75fc-e631-4c9f-a310-880cbcdd1d25
>>      health HEALTH_OK
>>      monmap e1: 1 mons at {mon1=10.10.48.7:6789/0}
>>             election epoch 3, quorum 0 mon1
>>      osdmap e12: 2 osds: 2 up, 2 in
>>             flags sortbitwise,require_jewel_osds
>>       pgmap v3376: 192 pgs, 2 pools, 0 bytes data, 0 objects
>>             73108 kB used, 1852 GB / 1852 GB avail
>>                  192 active+clean
>> ==
>> [root@mon1 ~]# ceph osd tree
>> ID WEIGHT  TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY
>> -1 1.80878 root default
>> -2 0.90439     host osd4
>>  0 0.90439         osd.0          up  1.00000          1.00000
>> -3 0.90439     host osdceph3
>>  1 0.90439         osd.1          up  1.00000          1.00000
>> ====
>>
>> >> Which OS are you running on your hosts?
>>
>> Our cloudstack servers are on Centos 6 and ceph admin/mon/osd servers 
>> are running on Centos 7.
>>
>> After we have enabled the cloudstack agent log to DEBUG on hypervisor 
>> server, we are seeing below errors while adding the ceph primary.
>>
>> ==
>> 2017-02-22 21:01:00,444 DEBUG [kvm.storage.LibvirtStorageAdaptor] 
>> (agentRequest-Handler-4:null) <pool type='rbd'>
>> ==
>>
>> In a quick search, we found that the hypervisor KVM has no kernel 
>> module loaded for RBD. Then we have upgraded the kernel to elrepo and 
>> loaded the module with modprobe. When we tried to rebuild rbd module 
>> with existing libvirtd configuration, that didn't worked. Apart from 
>> this, we have custom compiled libvirtd with rbd support and we have 
>> no clues regarding how to connect custom libvirtd with the qemu image 
>> utility.
>>
>> =====
>> [root@hyperkvm3 ~]# libvirtd --version (default)
>> libvirtd (libvirt) 0.10.2
>> =
>> [root@hyperkvm3 ~]# lsmod | grep rbd
>> rbd                    56743  0
>> libceph               148605  1 rbd
>> =
>> [root@hyperkvm3 ~]# qemu-img -h  | grep "Supported formats"
>> Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat 
>> qcow2 qed vhdx parallels nbd blkdebug null host_cdrom host_floppy 
>> host_device file gluster
>> == // no rbd support
>> [root@hyperkvm3 ~]# /usr/bin/sbin/libvirtd  --version  (custom)
>> /usr/bin/sbin/libvirtd (libvirt) 1.3.5
>> =====
>>
>> Do you have any ideas/suggestions?
>>
>> -Kurt
>>
>> On 02/22/2017 08:20 PM, Simon Weller wrote:
>>> I agree,  agent logs would be good to look at.
>>>
>>>
>>> You can enable kvm agent debugging by running this: sed -i 
>>> 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>>
>>> Restart the agent and then tail -f /var/log/cloudstack/agent/agent.log
>>>
>>>
>>> Also run ceph health and make sure your agent can talk to your ceph 
>>> monitors.
>>>
>>> Which OS are you running on your hosts?
>>>
>>>
>>> - Si
>>>
>>> ________________________________
>>> From: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
>>> Sent: Wednesday, February 22, 2017 12:45 AM
>>> To: users@cloudstack.apache.org
>>> Subject: Re: Getting errors while adding ceph storage to cloudstack
>>>
>>> Take a look at the agent logs on kvm host there will be more clues.
>>>
>>>
>>>
>>>
>>> On 22/02/17, 8:10 AM, "Kurt K" <kurt@ihnetworks.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I have created a ceph cluster with one admin server, one monitor 
>>>> and two
>>>> osd's. The setup is completed. But when trying to add the ceph as
>>>> primary storage of cloudstack, I am getting the below error in 
>>>> error logs.
>>>>
>>>> Am I missing something ? Please help.
>>>>
>>>> ================
>>>> 2017-02-20 21:03:02,842 DEBUG
>>>> [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
>>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) In createPool Adding the
>>>> pool to each of the hosts
>>>> 2017-02-20 21:03:02,843 DEBUG [c.c.s.StorageManagerImpl]
>>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Adding pool null to host 1
>>>> 2017-02-20 21:03:02,845 DEBUG [c.c.a.t.Request]
>>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
>>>> Sending  { Cmd , MgmtId: 207381009036, via: 1(hyperkvm.xxxxx.com), 
>>>> Ver:
>>>> v1, Flags: 100011,
>>>> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":14,"uuid":"9c51d737-3a6f-3bb3-8f28-109954fc2ef0","host":"mon1.xxxx.com","path":"cloudstack","userInfo":"cloudstack:AQDagqZYgSSpOBAATFvSt4tz3cOUWhNtR-NaoQ==","port":6789,"type":"RBD"},"localPath":"/mnt//ac5436a6-5889-30eb-b079-ac1e05a30526","wait":0}}]

>>>>
>>>> }
>>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
>>>> (AgentManager-Handler-15:null) Seq 1-653584895922143294: 
>>>> Processing:  {
>>>> Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 10,
>>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:

>>>>
>>>> Failed to create storage pool:
>>>> 9c51d737-3a6f-3bb3-8f28-109954fc2ef0\n\tat
>>>> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)\n\tat

>>>>
>>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)\n\tat

>>>>
>>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)\n\tat

>>>>
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2823)\n\tat

>>>>
>>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1325)\n\tat

>>>>
>>>> com.cloud.agent.Agent.processRequest(Agent.java:501)\n\tat
>>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat
>>>> com.cloud.utils.nio.Task.run(Task.java:84)\n\tat
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat

>>>>
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat

>>>>
>>>> java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
>>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
>>>> Received:  { Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 
>>>> 10, {
>>>> Answer } }
>>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.m.AgentManagerImpl]
>>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Details from executing 
>>>> class
>>>> com.cloud.agent.api.ModifyStoragePoolCommand:
>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to create
>>>> storage pool: 9c51d737-3a6f-3bb3-8f28-109954fc2ef0
>>>>      at
>>>> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)

>>>>
>>>>      at
>>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)

>>>>
>>>>      at
>>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)

>>>>
>>>> ================
>>>>
>>>> Any suggestions?
>>>>
>>>> -Kurt
>>> abhinandan.prateek@shapeblue.com
>>> www.shapeblue.com<http://www.shapeblue.com>
>>> ShapeBlue - The CloudStack Company<http://www.shapeblue.com/>
>>> www.shapeblue.com
>>> Introduction Upgrading CloudStack can sometimes be a little daunting 
>>> - but as the 5P's proverb goes - Proper Planning Prevents Poor 
>>> Performance.
>>>
>>>
>>>
>>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> @shapeblue
>>>
>>>
>>>
>>>
>>
>>
>
>


-- 
*~Kurt *

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message