cloudstack-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ivan Kudryavtsev <kudryavtsev...@bw-sw.com>
Subject Re: Poor NVMe Performance with KVM
Date Fri, 17 May 2019 12:24:43 GMT
Host is Dell r620 with Dual e5-2690/256GB 1333 DDR3.

пт, 17 мая 2019 г., 19:22 Ivan Kudryavtsev <kudryavtsev_ia@bw-sw.com>:

> Nux,
>
> I use Ubuntu 16.04 with "none" scheduler and the latest kernel 4.15. Guest
> is Ubuntu 18.04 with Noop scheduler for scsi-virtio  and "none" for virtio.
>
> Thanks.
>
> пт, 17 мая 2019 г., 19:18 Nux! <nux@li.nux.ro>:
>
>> Hi,
>>
>> What HV is that? CentOS? Are you using the right tuned profile? What
>> about in the guest? Which IO scheduler?
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>> > From: "Ivan Kudryavtsev" <kudryavtsev_ia@bw-sw.com>
>> > To: "users" <users@cloudstack.apache.org>
>> > Sent: Friday, 17 May, 2019 10:13:50
>> > Subject: Poor NVMe Performance with KVM
>>
>> > Hello, colleagues.
>> >
>> > Hope, someone could help me. I just deployed a new VM host with Intel
>> P4500
>> > local storage NVMe drive.
>> >
>> > From Hypervisor host I can get expected performance, 200K RIOPS, 3GBs
>> with
>> > FIO, write performance is also high as expected.
>> >
>> > I've created a new KVM VM Service offering with virtio-scsi controller
>> > (tried virtio as well) and VM is deployed. Now I try to benchmark it
>> with
>> > FIO. Results are very strange:
>> >
>> > 1. Read/Write with large blocks (1M) shows expected performance (my
>> limits
>> > are R=1000/W=500 MBs).
>> >
>> > 2. Write with direct=0 leads to expected 50K IOPS, while write with
>> > direct=1 leads to very moderate 2-3K IOPS.
>> >
>> > 3. Read with direct=0, direct=1 both lead to 3000 IOPS.
>> >
>> > During the benchmark I see VM IOWAIT=20%, while host IOWAIT is 0% which
>> is
>> > strange.
>> >
>> > So, basically, from inside VM my NVMe works very slow when small IOPS
>> are
>> > executed. From the host, it works great.
>> >
>> > I tried to mount the volume with NBD to /dev/nbd0 and benchmark. Read
>> > performance is nice. Maybe someone managed to use NVME with KVM with
>> small
>> > IOPS?
>> >
>> > The filesystem is XFS, previously tried with EXT4 - results are the
>> same.
>> >
>> > This is the part of VM XML definition generated by CloudStack:
>> >
>> >  <devices>
>> >    <emulator>/usr/bin/kvm-spice</emulator>
>> >    <disk type='file' device='disk'>
>> >      <driver name='qemu' type='qcow2' cache='none' discard='unmap'/>
>> >      <source
>> > file='/var/lib/libvirt/images/6809dbd0-4a15-4014-9322-fe9010695934'/>
>> >      <backingStore type='file' index='1'>
>> >        <format type='raw'/>
>> >        <source
>> > file='/var/lib/libvirt/images/ac43742c-3991-4be1-bff1-7617bf4fc6ef'/>
>> >        <backingStore/>
>> >      </backingStore>
>> >      <target dev='sda' bus='scsi'/>
>> >      <iotune>
>> >        <read_bytes_sec>1048576000</read_bytes_sec>
>> >        <write_bytes_sec>524288000</write_bytes_sec>
>> >        <read_iops_sec>100000</read_iops_sec>
>> >        <write_iops_sec>50000</write_iops_sec>
>> >      </iotune>
>> >      <serial>6809dbd04a1540149322</serial>
>> >      <alias name='scsi0-0-0-0'/>
>> >      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
>> >    </disk>
>> >    <disk type='file' device='cdrom'>
>> >      <driver name='qemu' type='raw'/>
>> >      <backingStore/>
>> >      <target dev='hdc' bus='ide'/>
>> >      <readonly/>
>> >      <alias name='ide0-1-0'/>
>> >      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
>> >    </disk>
>> >    <controller type='scsi' index='0' model='virtio-scsi'>
>> >      <alias name='scsi0'/>
>> >      <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
>> > function='0x0'/>
>> >    </controller>
>> >
>> > So, what I see now, is that it works slower than couple of two Samsung
>> 960
>> > PRO which is extremely strange.
>> >
>> > Thanks in advance.
>> >
>> >
>> > --
>> > With best regards, Ivan Kudryavtsev
>> > Bitworks LLC
>> > Cell RU: +7-923-414-1515
>> > Cell USA: +1-201-257-1512
>> > WWW: http://bitworks.software/ <http://bw-sw.com/>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message