cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kurt K <k...@ihnetworks.com>
Subject Re: Getting errors while adding ceph storage to cloudstack
Date Wed, 22 Mar 2017 12:12:14 GMT
Hi,

We have successfully added ceph storage to cloud stack. But we still 
have a doubt regarding creating VMs via ceph primary.

Any idea about the VM's data store location in ceph primary?

/-Kurt/

On 02/23/2017 11:20 AM, Kurt K wrote:
> Hi Simon,
>
> Thanks for the reply.
>
> >> Also run ceph health and make sure your agent can talk to your ceph 
> monitors.
>
> ceph health status shows fine and also we can connect our OSD's from 
> monitor server. Snippets pasted below.
>
> ===
> [root@mon1 ~]# ceph -s
>     cluster ebac75fc-e631-4c9f-a310-880cbcdd1d25
>      health HEALTH_OK
>      monmap e1: 1 mons at {mon1=10.10.48.7:6789/0}
>             election epoch 3, quorum 0 mon1
>      osdmap e12: 2 osds: 2 up, 2 in
>             flags sortbitwise,require_jewel_osds
>       pgmap v3376: 192 pgs, 2 pools, 0 bytes data, 0 objects
>             73108 kB used, 1852 GB / 1852 GB avail
>                  192 active+clean
> ==
> [root@mon1 ~]# ceph osd tree
> ID WEIGHT  TYPE NAME         UP/DOWN REWEIGHT PRIMARY-AFFINITY
> -1 1.80878 root default
> -2 0.90439     host osd4
>  0 0.90439         osd.0          up  1.00000          1.00000
> -3 0.90439     host osdceph3
>  1 0.90439         osd.1          up  1.00000          1.00000
> ====
>
> >> Which OS are you running on your hosts?
>
> Our cloudstack servers are on Centos 6 and ceph admin/mon/osd servers 
> are running on Centos 7.
>
> After we have enabled the cloudstack agent log to DEBUG on hypervisor 
> server, we are seeing below errors while adding the ceph primary.
>
> ==
> 2017-02-22 21:01:00,444 DEBUG [kvm.storage.LibvirtStorageAdaptor] 
> (agentRequest-Handler-4:null) <pool type='rbd'>
> ==
>
> In a quick search, we found that the hypervisor KVM has no kernel 
> module loaded for RBD. Then we have upgraded the kernel to elrepo and 
> loaded the module with modprobe. When we tried to rebuild rbd module 
> with existing libvirtd configuration, that didn't worked. Apart from 
> this, we have custom compiled libvirtd with rbd support and we have no 
> clues regarding how to connect custom libvirtd with the qemu image 
> utility.
>
> =====
> [root@hyperkvm3 ~]# libvirtd --version (default)
> libvirtd (libvirt) 0.10.2
> =
> [root@hyperkvm3 ~]# lsmod | grep rbd
> rbd                    56743  0
> libceph               148605  1 rbd
> =
> [root@hyperkvm3 ~]# qemu-img -h  | grep "Supported formats"
> Supported formats: raw cow qcow vdi vmdk cloop dmg bochs vpc vvfat 
> qcow2 qed vhdx parallels nbd blkdebug null host_cdrom  host_floppy 
> host_device file gluster
> == // no rbd support
> [root@hyperkvm3 ~]# /usr/bin/sbin/libvirtd  --version  (custom)
> /usr/bin/sbin/libvirtd (libvirt) 1.3.5
> =====
>
> Do you have any ideas/suggestions?
>
> -Kurt
>
> On 02/22/2017 08:20 PM, Simon Weller wrote:
>> I agree,  agent logs would be good to look at.
>>
>>
>> You can enable kvm agent debugging by running this: sed -i 
>> 's/INFO/DEBUG/g' /etc/cloudstack/agent/log4j-cloud.xml
>>
>> Restart the agent and then tail -f /var/log/cloudstack/agent/agent.log
>>
>>
>> Also run ceph health and make sure your agent can talk to your ceph 
>> monitors.
>>
>> Which OS are you running on your hosts?
>>
>>
>> - Si
>>
>> ________________________________
>> From: Abhinandan Prateek <abhinandan.prateek@shapeblue.com>
>> Sent: Wednesday, February 22, 2017 12:45 AM
>> To: users@cloudstack.apache.org
>> Subject: Re: Getting errors while adding ceph storage to cloudstack
>>
>> Take a look at the agent logs on kvm host there will be more clues.
>>
>>
>>
>>
>> On 22/02/17, 8:10 AM, "Kurt K" <kurt@ihnetworks.com> wrote:
>>
>>> Hello,
>>>
>>> I have created a ceph cluster with one admin server, one monitor and 
>>> two
>>> osd's. The setup is completed. But when trying to add the ceph as
>>> primary storage of cloudstack, I am getting the below error in error 
>>> logs.
>>>
>>> Am I missing something ? Please help.
>>>
>>> ================
>>> 2017-02-20 21:03:02,842 DEBUG
>>> [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl]
>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) In createPool Adding the
>>> pool to each of the hosts
>>> 2017-02-20 21:03:02,843 DEBUG [c.c.s.StorageManagerImpl]
>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Adding pool null to host 1
>>> 2017-02-20 21:03:02,845 DEBUG [c.c.a.t.Request]
>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
>>> Sending  { Cmd , MgmtId: 207381009036, via: 1(hyperkvm.xxxxx.com), Ver:
>>> v1, Flags: 100011,
>>> [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":true,"pool":{"id":14,"uuid":"9c51d737-3a6f-3bb3-8f28-109954fc2ef0","host":"mon1.xxxx.com","path":"cloudstack","userInfo":"cloudstack:AQDagqZYgSSpOBAATFvSt4tz3cOUWhNtR-NaoQ==","port":6789,"type":"RBD"},"localPath":"/mnt//ac5436a6-5889-30eb-b079-ac1e05a30526","wait":0}}]

>>>
>>> }
>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
>>> (AgentManager-Handler-15:null) Seq 1-653584895922143294: Processing:  {
>>> Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 10,
>>> [{"com.cloud.agent.api.Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:

>>>
>>> Failed to create storage pool:
>>> 9c51d737-3a6f-3bb3-8f28-109954fc2ef0\n\tat
>>> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)\n\tat

>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)\n\tat

>>>
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)\n\tat

>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2823)\n\tat

>>>
>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1325)\n\tat

>>>
>>> com.cloud.agent.Agent.processRequest(Agent.java:501)\n\tat
>>> com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:808)\n\tat
>>> com.cloud.utils.nio.Task.run(Task.java:84)\n\tat
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat

>>>
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat

>>>
>>> java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.t.Request]
>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Seq 1-653584895922143294:
>>> Received:  { Ans: , MgmtId: 207381009036, via: 1, Ver: v1, Flags: 10, {
>>> Answer } }
>>> 2017-02-20 21:03:02,944 DEBUG [c.c.a.m.AgentManagerImpl]
>>> (catalina-exec-6:ctx-f293a10c ctx-093b4faf) Details from executing 
>>> class
>>> com.cloud.agent.api.ModifyStoragePoolCommand:
>>> com.cloud.utils.exception.CloudRuntimeException: Failed to create
>>> storage pool: 9c51d737-3a6f-3bb3-8f28-109954fc2ef0
>>>      at
>>> com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:524)

>>>
>>>      at
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:277)

>>>
>>>      at
>>> com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:271)

>>>
>>> ================
>>>
>>> Any suggestions?
>>>
>>> -Kurt
>> abhinandan.prateek@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> ShapeBlue - The CloudStack Company<http://www.shapeblue.com/>
>> www.shapeblue.com
>> Introduction Upgrading CloudStack can sometimes be a little daunting 
>> - but as the 5P's proverb goes - Proper Planning Prevents Poor 
>> Performance.
>>
>>
>>
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>
>


-- 
*~Kurt *

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message