cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kirk Kosinski <kirkkosin...@gmail.com>
Subject Re: [URGENT] Mounting issues from KVM hosts to secondary storage
Date Sat, 05 Oct 2013 00:53:16 GMT
> And how about virsh pool-list result which is not reflecting the
> actual IDs of the secondary storage we see from Cloudstack GUI, is it
> supposed to be the case? If not, how to rectify it?

This is normal.  More than one pool pointing to secondary storage can
exist at a time so they can't all be named after the ID in CloudStack.

Best regards,
Kirk

On 10/03/2013 01:30 PM, Indra Pramana wrote:
> Hi Marcus,
> 
> Good day to you, and thank you for your e-mail.
> 
> We will try to do what you have suggested and will see if we can fix the
> problem. I am looking into increasing the filesystem of the KVM hosts to
> see if it can resolve the problem for now.
> 
> Thank you very much for your help!
> 
> Cheers.
> 
> 
> 
> On Fri, Oct 4, 2013 at 2:32 AM, Marcus Sorensen <shadowsor@gmail.com> wrote:
> 
>> You'll have to go back further in the log then, and/or go to the agent
>> log, because the issue of the ghost mount is due to the filesystem
>> being full, and if that's your secondary storage I'm not sure why it
>> would be written to from the kvm host unless you were doing backups or
>> snapshots.
>>
>> On Thu, Oct 3, 2013 at 12:03 PM, Indra Pramana <indra@sg.or.id> wrote:
>>> Hi Marcus,
>>>
>>> It happens randomly, mounting fails, and then / gets filled. I suspect
>> this
>>> occurs when the mounting fails, any files which is supposed to be
>>> transferred to the mount point (e.g. templates used to create VMs) will
>>> fill-up the / partition of the KVM host instead. This doesn't happen in
>>> 4.1.1, this only happens after we upgraded to 4.2.0.
>>>
>>> And how about virsh pool-list result which is not reflecting the actual
>> IDs
>>> of the secondary storage we see from Cloudstack GUI, is it supposed to be
>>> the case? If not, how to rectify it?
>>>
>>> Looking forward to your reply, thank you.
>>>
>>> Cheers.
>>>
>>>
>>>
>>>
>>> On Fri, Oct 4, 2013 at 1:51 AM, Marcus Sorensen <shadowsor@gmail.com>
>> wrote:
>>>
>>>> It sort of looks like the out of space triggered the issue. Libvirt
>> shows
>>>>
>>>> 2013-10-03 15:38:57.414+0000: 2710: error : virCommandWait:2348 :
>>>> internal error Child process (/bin/umount
>>>> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669) unexpected exit status 32:
>>>> error writing /etc/mtab.tmp: No space left on device
>>>>
>>>> So that entry of the filesystem being mounted is orphaned in
>>>> /etc/mtab, since it can't be removed. That seems to be the source of
>>>> why 'df' shows the thing mounted when it isn't, at least.
>>>>
>>>>
>>>> On Thu, Oct 3, 2013 at 11:45 AM, Indra Pramana <indra@sg.or.id> wrote:
>>>>> Hi Marcus and all,
>>>>>
>>>>> I also find some strange and interesting error messages from the
>> libvirt
>>>>> logs:
>>>>>
>>>>> http://pastebin.com/5ByfNpAf
>>>>>
>>>>> Looking forward to your reply, thank you.
>>>>>
>>>>> Cheers.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Oct 4, 2013 at 1:38 AM, Indra Pramana <indra@sg.or.id>
wrote:
>>>>>
>>>>>> Hi Marcus,
>>>>>>
>>>>>> Good day to you, and thank you for your e-mail. See my reply inline.
>>>>>>
>>>>>> On Fri, Oct 4, 2013 at 12:29 AM, Marcus Sorensen <
>> shadowsor@gmail.com
>>>>> wrote:
>>>>>>
>>>>>>> Are you using the release artifacts, or your own 4.2 build?
>>>>>>>
>>>>>>
>>>>>> [Indra:] We are using the release artifacts from below repo since
we
>> are
>>>>>> using Ubuntu:
>>>>>>
>>>>>> deb http://cloudstack.apt-get.eu/ubuntu precise 4.2
>>>>>>
>>>>>>
>>>>>>> When you rebooted the host, did the problem go away or come back
the
>>>>>>> same?
>>>>>>
>>>>>>
>>>>>> [Indra:] When I rebooted the host, the problem go away for a while,
>> but
>>>> it
>>>>>> will come back again after some time. It will come randomly at the
>> time
>>>>>> when we need to create a new instance on that host, or start an
>> existing
>>>>>> stopped instance.
>>>>>>
>>>>>>
>>>>>>> You may want to look at 'virsh pool-list' to see if libvirt is
>>>>>>> mounting/registering the secondary storage.
>>>>>>>
>>>>>>
>>>>>> [Indra:] This is the result of the virsh pool-list command:
>>>>>>
>>>>>> root@hv-kvm-02:/var/log/libvirt# virsh pool-list
>>>>>> Name                 State      Autostart
>>>>>> -----------------------------------------
>>>>>> 301071ac-4c1d-4eac-855b-124126da0a38 active     no
>>>>>> 5230667e-9c58-3ff6-983c-5fc2a72df669 active     no
>>>>>> d433809b-01ea-3947-ba0f-48077244e4d6 active     no
>>>>>>
>>>>>> Strange thing is that none of my secondary storage IDs are there.
>> Could
>>>> it
>>>>>> be that the ID might have changed during Cloudstack upgrade? Here
is
>> the
>>>>>> list of my secondary storage (there are two of them) even though
they
>>>> are
>>>>>> on the same NFS server:
>>>>>>   c02da448-b9f4-401b-b8d5-83e8ead5cfde nfs://
>>>>>> 10.237.11.31/mnt/vol1/sec-storage NFS
>>>>>> 5937edb6-2e95-4ae2-907b-80fe4599ed87 nfs://
>>>>>> 10.237.11.31/mnt/vol1/sec-storage2 NFS
>>>>>>
>>>>>>> Is this happening on multiple hosts, the same way?
>>>>>>
>>>>>>
>>>>>> [Indra:] Yes, it is happening on all the two hosts that I have, the
>> same
>>>>>> way.
>>>>>>
>>>>>>
>>>>>>> You may want to
>>>>>>> look at /etc/mtab, if the system reports it's mounted, though
it's
>>>>>>> not, it might be in there. Look at /proc/mounts as well.
>>>>>>>
>>>>>>
>>>>>> [Indra:] Please find result of df, /etc/mtab and /proc/mounts below.
>> The
>>>>>> "ghost" mount point is on df and /etc/mtab, but not on /proc/mounts.
>>>>>>
>>>>>> root@hv-kvm-02:/etc# df
>>>>>>
>>>>>> Filesystem                                              1K-blocks
>>>>  Used
>>>>>> Available Use% Mounted on
>>>>>> /dev/sda1                                                 4195924
>>>>>> 4192372         0 100% /
>>>>>>
>>>>>> udev                                                    132053356
>>>> 4
>>>>>> 132053352   1% /dev
>>>>>> tmpfs                                                    52825052
>>>> 704
>>>>>> 52824348   1% /run
>>>>>>
>>>>>> none                                                         5120
>>>>>> 0      5120   0% /run/lock
>>>>>> none                                                    132062620
>>>> 0
>>>>>> 132062620   0% /run/shm
>>>>>> cgroup                                                  132062620
>>>> 0
>>>>>> 132062620   0% /sys/fs/cgroup
>>>>>> /dev/sda6                                                10650544
>>>>>> 2500460   7609056  25% /home
>>>>>> 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288   4195924
>>>>>> 4192372         0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669
>>>>>>
>>>>>> root@hv-kvm-02:/etc# cat /etc/mtab
>>>>>> /dev/sda1 / ext4 rw,errors=remount-ro 0 0
>>>>>> proc /proc proc rw,noexec,nosuid,nodev 0 0
>>>>>> sysfs /sys sysfs rw,noexec,nosuid,nodev 0 0
>>>>>> none /sys/fs/fuse/connections fusectl rw 0 0
>>>>>> none /sys/kernel/debug debugfs rw 0 0
>>>>>> none /sys/kernel/security securityfs rw 0 0
>>>>>> udev /dev devtmpfs rw,mode=0755 0 0
>>>>>> devpts /dev/pts devpts rw,noexec,nosuid,gid=5,mode=0620 0 0
>>>>>> tmpfs /run tmpfs rw,noexec,nosuid,size=10%,mode=0755 0 0
>>>>>> none /run/lock tmpfs rw,noexec,nosuid,nodev,size=5242880 0 0
>>>>>> none /run/shm tmpfs rw,nosuid,nodev 0 0
>>>>>> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
>>>>>> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
>>>>>> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
>>>>>> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
>>>>>> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
>>>>>> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
>>>>>> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
>>>>>> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
>>>>>> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0
0
>>>>>> /dev/sda6 /home ext4 rw 0 0
>>>>>> rpc_pipefs /run/rpc_pipefs rpc_pipefs rw 0 0
>>>>>> binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc
>> rw,noexec,nosuid,nodev
>>>> 0 0
>>>>>> 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288
>>>>>> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 nfs rw,addr=10.237.11.31
0
>> 0
>>>>>>
>>>>>> rootfs / rootfs rw 0 0
>>>>>> sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
>>>>>> proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
>>>>>> udev /dev devtmpfs
>>>> rw,relatime,size=132053356k,nr_inodes=33013339,mode=755
>>>>>> 0 0
>>>>>> devpts /dev/pts devpts
>>>>>> rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
>>>>>> tmpfs /run tmpfs rw,nosuid,relatime,size=52825052k,mode=755 0 0
>>>>>> /dev/disk/by-uuid/d21f4676-075d-44be-a378-92ef6aaf2496 / ext4
>>>>>> rw,relatime,errors=remount-ro,data=ordered 0 0
>>>>>> none /sys/fs/fuse/connections fusectl rw,relatime 0 0
>>>>>> none /sys/kernel/debug debugfs rw,relatime 0 0
>>>>>> cgroup /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0
>>>>>> none /sys/kernel/security securityfs rw,relatime 0 0
>>>>>> none /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0
0
>>>>>> none /run/shm tmpfs rw,nosuid,nodev,relatime 0 0
>>>>>> cgroup /sys/fs/cgroup/cpuset cgroup rw,relatime,cpuset 0 0
>>>>>> cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0
>>>>>> cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0
>>>>>> cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0
>>>>>> cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0
>>>>>> cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0
>>>>>> cgroup /sys/fs/cgroup/blkio cgroup rw,relatime,blkio 0 0
>>>>>> cgroup /sys/fs/cgroup/perf_event cgroup rw,relatime,perf_event 0
0
>>>>>> /dev/sda6 /home ext4 rw,relatime,data=ordered 0 0
>>>>>> rpc_pipefs /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
>>>>>> binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc
>>>>>> rw,nosuid,nodev,noexec,relatime 0 0
>>>>>>
>>>>>> If I tried to un-mount (umount) the "ghost" mount point, it will
fail
>>>> with
>>>>>> below error message saying that it is not found in /proc/mounts:
>>>>>>
>>>>>> root@hv-kvm-02:/etc# umount
>> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669
>>>>>> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in
>> /proc/mounts
>>>>>> /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669 was not found in
>> /proc/mounts
>>>>>>
>>>>>> Do you have any idea what could be the problem? Is it libvirt issue
>> or
>>>>>> CloudStack issue?
>>>>>>
>>>>>> Any help on this is greatly appreciated. Since I am upgrading a
>>>> production
>>>>>> setup, we only have 1-2 hours left to resolve the problem, otherwise
>> we
>>>>>> will be forced to revert back to 4.1.1 (for the fourth time). :(
>>>>>>
>>>>>> Urgently looking forward to your reply, thank you.
>>>>>>
>>>>>> Cheers.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 3, 2013 at 9:53 AM, Indra Pramana <indra@sg.or.id>
>> wrote:
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> We face a major problem after upgrading to 4.2.0. Mounting
from
>> KVM
>>>>>>> hosts
>>>>>>>> to secondary storage seems to fail, every time a new VM instance
>> is
>>>>>>>> created, it will use up the / (root) partition of the KVM
hosts
>>>> instead.
>>>>>>>>
>>>>>>>> Here is the df result:
>>>>>>>>
>>>>>>>> ====
>>>>>>>> root@hv-kvm-02:/home/indra# df
>>>>>>>> Filesystem                                              1K-blocks
>>>>>>>  Used
>>>>>>>> Available Use% Mounted on
>>>>>>>> /dev/sda1                                               
 4195924
>>>>>>>> 4195924         0 100% /
>>>>>>>> udev                                                    132053356
>>>>>>> 4
>>>>>>>> 132053352   1% /dev
>>>>>>>> tmpfs                                                   
52825052
>>>>>>> 440
>>>>>>>> 52824612   1% /run
>>>>>>>> none                                                    
    5120
>>>>>>>> 0      5120   0% /run/lock
>>>>>>>> none                                                    132062620
>>>>>>> 0
>>>>>>>> 132062620   0% /run/shm
>>>>>>>> cgroup                                                  132062620
>>>>>>> 0
>>>>>>>> 132062620   0% /sys/fs/cgroup
>>>>>>>> /dev/sda6                                               
10650544
>>>>>>> 2500424
>>>>>>>> 7609092  25% /home
>>>>>>>> 10.237.11.31:/mnt/vol1/sec-storage2/template/tmpl/2/288 
 4195924
>>>>>>>> 4195924         0 100% /mnt/5230667e-9c58-3ff6-983c-5fc2a72df669
>>>>>>>> ====
>>>>>>>>
>>>>>>>> The strange thing is that df shows that it seems to be mounted,
>> but
>>>> it's
>>>>>>>> actually not mounted. If you noticed, the total capacity
of the
>> mount
>>>>>>> point
>>>>>>>> is exactly the same as the capacity of the / (root) partition.
By
>>>> right
>>>>>>> it
>>>>>>>> should show 7 TB instead of just 4 GB.
>>>>>>>>
>>>>>>>> This caused VM creation to be error due to out of disk space.
This
>>>> also
>>>>>>>> affect the KVM operations since the / (root) partition becomes
>> full,
>>>>>>> and we
>>>>>>>> can only release the space after we reboot the KVM host.
>>>>>>>>
>>>>>>>> Anyone experience this problem before? We are at loss on
how to
>>>> resolve
>>>>>>> the
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> Looking forward to your reply, thank you.
>>>>>>>>
>>>>>>>> Cheers.
>>>>>>>
>>>>>>
>>>>>>
>>>>
>>
> 

Mime
View raw message