Measuring Boot Latency

How to measure NEMU boot latency?

NEMU aims at being a slim, simple and efficient hypervisor and boot time latencies are an important metric.

What do we want to measure?

When starting a QEMU VM, we're generally interested about how long it takes before our application can start, i.e. how long does it take for the VM to start user space. We can split that time into 3 parts:

• How much time does QEMU spend before actually launching the virtual machine (VM)? This is the time QEMU spends preparing and launching the VM itself, the QEMU boot time: tQEMU
• When the VM starts through a BIOS/firmware, how much time do we spend in firmware, before starting the VM's kernel? That is the firmware boot time: tFW.
• How much time does it take for the VM kernel to boot and switch to user space? That is the kernel boot time: tKernel.

Overall tQEMU + tFW + tKernel gives us the boot to user space latency.

Methodology

There are many ways to measure boot time with QEMU, but here we're using a very simple one.

Our methodology is based on adding VM exit points in various components and using a QEMU specific device that actually terminates the QEMU process and thus the VM itself when handling those VM exits. Since the VM terminates right at those exit points, we can simply use generic tools like time to measure the time spent between the QEMU startup and termination moments. By triggering those VM exits at the right moments, we can easily measure tQEMU, tFW and tKernel.

The QEMU device used for those measurement is called isa-debug-exit for the PC machine type and sysbus-debugexit for the virt machine type. These devices take 2 arguments iobase and iosize. iobase basically specifies which IO port we need to write into to have these devices eventually handle the corresponding VM exit. If for example we start NEMU with the following argument:

-device sysbus-debugexit,iobase=0xf4

then any IO write to 0xf4 will terminate the QEMU process and the corresponding VM.

By properly instrumenting the QEMU, firmware and kernel code bases with the right IO port writes, we will force QEMU process terminations at different point of the boot process.

Putting it all together

We're going to use a combination of NEMU, qboot and a 4.18 Linux kernel to do our measurements.

NEMU

We use the topic/virt-x86 NEMU branch to build our QEMU binary with support for the virt machine type. See Running NEMU for more details.

qboot

qboot is a minimal x86 firmware that can boot a Linux kernel. We have added support for the virt machine type to it, please run the following commands to build a virt enabled qboot binary:

# git clone https://github.com/sameo/qboot
# cd qboot
# git branch --track virt-benchmarks origin/topic/virt-benchmarks
# git checkout virt-benchmarks
# BENCHMARK_HACK=1 make

You will now have a qboot/bios.bin binary that you can use as your VM firmware

Linux

Apply the following patch to a 4.18 kernel:

diff --git a/init/main.c b/init/main.c
index 5e13c544bbf4..468e60bf8098 100644
--- a/init/main.c
+++ b/init/main.c
@@ -1070,6 +1070,8 @@ static int __ref kernel_init(void *unused)
 
        rcu_end_inkernel_boot();
 
+       outb(2, 0xf6);
+
        if (ramdisk_execute_command) {
                ret = run_init_process(ramdisk_execute_command);
                if (!ret)

And build your kernel

# cd linux-4.18
# wget https://gist.github.com/sameo/dbbe3ad34a128a721cad5305c6aab307 -O .config
# make oldconfig
# make bzImage -j`nproc`

Measurements

We will use a script to run a number of VM startup iterations:

# wget https://gist.github.com/sameo/f9409091fc09c76858d20e0168dbc6f7 -O nemu-boot-loop.sh

QEMU boot time (tQEMU) for 1000 iterations

time ./nemu-boot-loop.sh --iterations=1000 --exit=0xf5 --firmware=~/qboot/bios.bin --binary=~/build-x86_64_virt/x86_64_virt-softmmu/qemu-system-x86_64_virt --kernel=~/linux-4.18/arch/x86/boot/bzImage

QEMU + firmware boot time (tQEMU + tFW) for 1000 iterations

time ./nemu-boot-loop.sh --iterations=1000 --exit=0xf4 --firmware=~/qboot/bios.bin --binary=~/build-x86_64_virt/x86_64_virt-softmmu/qemu-system-x86_64_virt --kernel=~/linux-4.18/arch/x86/boot/bzImage

QEMU + firmware + kernel boot time (tQEMU + tFW + tKernel) for 100 iterations

time ./nemu-boot-loop.sh --iterations=100 --exit=0xf6 --firmware=~/qboot/bios.bin --binary=~/build-x86_64_virt/x86_64_virt-softmmu/qemu-system-x86_64_virt --kernel=~/linux-4.18/arch/x86/boot/bzImage