Recently I was posed with the question. Which will give better performance on Harvester, virtio or SATA disk types. Although speed largely depends on what disks you choose to use (SSD, NVME, HDD), there is a difference in performance when it comes to these two types.
First, let’s take a look at virtio. Virtio is a standard for network and disk device drivers where the guest's device driver "knows" that it is running in a virtual environment. So, it cooperates with the host to achieve efficient utilization of resources.
Virtio performs this task by leveraging a paravirtualized approach. In a full-virtualisation scenario, an entire device needs to be emulated. But with paravirtualization, only the parts of a device that are most difficult to virtualize are emulated. For all the rest, the guest Operating System uses the Virtio API to get direct access to the physical hardware.
This results in (supposedly) faster performance as compared to emulating everything, and is the reason why Virtio is commonly used in environments that rely heavily on virtualization. It's supported by many hypervisors (here we are talking about Harvester), including QEMU, KVM, and VirtualBox.
In other words, Virtio provides a more direct and efficient way for virtual machines to communicate with devices on the host machine, or to use resources provided by the host machine, (supposedly) improving the performance of the virtual machine.
IBM has a great article on virtio: https://developer.ibm.com/articles/l-virtio/
I won’t be touching on SATA since this is the type that people are most familiar with. Let’s move on to the actual data. Is virtio faster?
I decided to test this using fio. I found an article on GCP that walked me through some basic testing. I created 4 Harvester VMs. 2 SATA machines and 2 virtio machines. Each machine has 8 CPU, 12 GB RAM, and a 200 GB boot drive.
The first thing we want to test is write throughput which we test by performing sequential writes with multiple parallel streams (16+), using an I/O block size of 1 MB and an I/O depth of at least 64. The test looks like this:
sudo fio --name=write_throughput --directory=$TEST_DIR --numjobs=16 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=write \
--group_reporting=1 --iodepth_batch_submit=64 \
--iodepth_batch_complete_max=64
And here are the results:
virtio-01: vda: ios=0/30837, merge=0/2822, ticks=0/45182123, in_queue=45214469, util=94.49%
virtio-02: vda: ios=0/30088, merge=0/2793, ticks=0/51260920, in_queue=51293951, util=92.95%
sata-01: sda: ios=0/200556, merge=0/2872, ticks=0/3960191, in_queue=3960567, util=99.83%
sata-02: sda: ios=0/209553, merge=0/2750, ticks=0/3837358, in_queue=3837596, util=99.97%
(The hostnames of the machine are “virtio-01,02,sata-01,02”)
From the fio test results, we can determine the performance of the machines by comparing the provided statistics.
1. "ios" is the total number of read/write operations. For the "write" case, the lower number of operations (virtio-01, virtio-02 with ios=0/30837, 0/30088) might indicate higher throughput due to the bigger size of each operation.
2. "merge" is the number of operations that were merged into existing operations. A lower number indicates fewer operations needed to be merged, which might suggest better performance.
3. "ticks" is the total time the disk was busy with read/write operations. Lower ticks means higher disk speed.
4. "in_queue" indicates the amount of time operations spent waiting in the scheduler queue. Lower means operations spent less time waiting, hence better disk performance.
5. "util" is the percentage of CPU time during which I/O requests were issued to the device. This is a measure of the load put on the system by the file I/O. Lower values would typically mean better performance, but it also means CPU is less busy which might be not the case for I/O-bound systems.
Looking at these statistics, the virtio machines have higher queue times and busy times ("ticks" and "in_queue") which would suggest that they perform slower in this test. The SATA machines have higher I/O operations and lower queue times and busy times, suggesting they would perform better.
Next, we test write IOPS by performing random writes, using an I/O block size of 4 KB and an I/O depth of at least 256. The test looks like this:
sudo fio --name=write_iops --directory=$TEST_DIR --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=256 --rw=randwrite --group_reporting=1 \
--iodepth_batch_submit=256 --iodepth_batch_complete_max=256
Here are the results:
virtio-01: vda: ios=0/688301, merge=0/91296, ticks=0/15268667, in_queue=15268779, util=99.86%
virtio-02: vda: ios=0/642641, merge=0/87824, ticks=0/15040145, in_queue=15040188, util=99.82%
sata-01: sda: ios=0/688951, merge=0/81752, ticks=0/3382053, in_queue=3382114, util=99.90%
sata-02: sda: ios=0/725208, merge=0/94523, ticks=0/3487930, in_queue=3487980, util=99.91%
From these results, it seems that SATA machines sata-01 and sata-02 have performed better in this write IOPS test. Although the differences in I/O operations and merge numbers between the machines are relatively small, the much lower ticks and in_queue times for SATA machines indicate faster operations and processing times.
Up next we test read throughput by performing sequential reads with multiple parallel streams (16+), using an I/O block size of 1 MB and an I/O depth of at least 64. The test looks like this:
sudo fio --name=read_throughput --directory=$TEST_DIR --numjobs=16 \
--size=10G --time_based --runtime=60s --ramp_time=2s --ioengine=libaio \
--direct=1 --verify=0 --bs=1M --iodepth=64 --rw=read \
--group_reporting=1 \
--iodepth_batch_submit=64 --iodepth_batch_complete_max=64
Here are the results:
virtio-01: vda: ios=53131/264, merge=0/228, ticks=7561068/240051, in_queue=7801475, util=100.00%
virtio-02: vda: ios=41900/735, merge=0/237, ticks=7994353/571157, in_queue=8565934, util=99.30%
sata-01: sda: ios=271187/12019, merge=0/229, ticks=3508338/186120, in_queue=3694619, util=99.94%
sata-02: sda: ios=367103/12031, merge=0/226, ticks=3408316/109919, in_queue=3518314, util=99.08%
Comparing these results, it appears that the SATA machines exhibit better read throughput in these tests. They complete more operations (higher "ios" for reads), and they complete these operations faster (lower "ticks" and "in_queue" times). This suggests that for read-heavy workloads, the SATA machines might offer better performance.
Finally we test read IOPS by performing random reads, using an I/O block size of 4 KB and an I/O depth of at least 256. The test looks like this:
sudo fio --name=read_iops --directory=$TEST_DIR --size=10G \
--time_based --runtime=60s --ramp_time=2s --ioengine=libaio --direct=1 \
--verify=0 --bs=4K --iodepth=256 --rw=randread --group_reporting=1 \
--iodepth_batch_submit=256 --iodepth_batch_complete_max=256
Here are the results:
virtio-01: vda: ios=1455976/4, merge=26/1, ticks=15771672/71, in_queue=15771746, util=99.91%
virtio-02: vda: ios=1602142/4, merge=25/1, ticks=15682020/51, in_queue=15682076, util=99.93%
sata-01: sda: ios=1075334/30, merge=21/37, ticks=2299071/149, in_queue=2299365, util=99.80%
sata-02: sda: ios=1261792/46, merge=27/32, ticks=2253883/437, in_queue=2254416, util=99.89%
While the Virtio machines managed to perform a higher number of operations, the SATA machines seem to have processed these operations more efficiently. This could make the SATA machines slightly faster in terms of read IOPS for this specific test.
So what did we learn?
Write Throughput Test: Here, the SATA machines showcased better performance. They had lower queue times and busy times, suggesting higher write speeds.
Write IOPS Test: In this scenario, SATA machines seemed to perform better, although the differences were relatively small. They showcased more I/O operations and lower queue and busy times, suggesting faster processing.
Read Throughput Test: The SATA machines seemed to be completing significantly more read operations than the Virtio machines, and they completed these operations faster, suggesting better read throughput performance.
Read IOPS Test: Despite the Virtio machines performing a higher number of operations, the SATA machines appeared to process these operations more efficiently, hence possibly giving a better read IOPS performance. However, the differences were not significant.
In conclusion, these tests indicate that the SATA machines have generally performed better than the Virtio machines.
I will be honest, I was not expecting this to be the case. I also only want to remind people that this was run on Harvester machines. This test may not be relevant for other hypervisors. I just find it interesting that for lots of writes and reads, SATA is the way to go.
Cheers,
Joe