When I went to Kernel Recipes earlier this year, I watched a very interesting presentation on using QEmu for Kernel development.
A few hours ago, working on Eudyptula challenge, I was getting a bit annoyed :
The nvidia module was not working on my custom kernel, I needed to do a mkinitcpio each time I compiled a kernel based on the Archlinux config, and occasionally my development module would crash the system. Thus I went on a quest to get QEmu working.
The subject is fairly easy, but unfortunately, documentation is sparse, so here’s a Howto that will allow you to get running in just a few minutes.
If you really are in a hurry, jump to the end of this article. I provide the finished scripts to build the system.
Principles
What people usually do when building a custom system is re-use a system, like debootstrap (it builds a debian system inside a directory), or openembedded, buildroot, ptxdist (mostly used in the embedded world). In our case however, we want to do a really really small system, so even those build systems are to much. We want something completely bare.
So what we are going to create is a build script that will :
- Compile your Kernel and modules
- Build an initramfs/initrd image, of about 3-4MB containing :
- busybox
- your kernel modules
- a custom init script
Building your kernel
I am not going to explain how to configure your kernel. If you don’t want to worry too much, just reuse your distributions “.config” file. It should be located in /proc/config.gz.
Compiling is easy :
ROOT_DIR=$(pwd) # adjust accordingly KERNEL_DIR=$ROOT_DIR/linux # Compile your kernel cd $KERNEL_DIR make make modules cd $ROOT_DIR
Building an Initramfs
First a little reminder : what is an initramdisk?
An initramdisk is a (small) CPIO archive that is loaded in ram as a / filesystem.
It is usually used in distros to do a 2-stage boot. The first boot will load the vital modules, mount the real file system (probably from a disk) and start the second stage from the disk.
This allows distros to provide a precompiled linux kernel with everything as modules for everyone (including motherboard drivers), and generate on installation an initrd that provides the modules your specific system will need to do a basic boot (ie: your “/” filesystem, essential motherboard drivers, …).
In our case, we don’t care about that second stage, we just want to boot into a shell.
Installing kernel modules
Make modules_install will install your modules in /lib/modules/<kernel_name>/…
You can just set INSTALL_MOD_PATH, and the modules will be installed in $INSTALL_MOD_PATH/lib/modules/<kernel_name>/… instead
INITRAMFS_DIR=$ROOT_DIR/initramfs # remove old modules rm -fr $INITRAMFS_DIR/lib/modules # install new modules cd $KERNEL_DIR make INSTALL_MOD_PATH=$INITRAMFS_DIR modules_install cd $ROOT_DIR
Installing Busybox
You can just install Busybox on your host system (provided you want to emulate the same architecture), and run
busybox --install $INITRAMFS_DIR/bin/
This will install all binaries that busybox can emulate into /bin of your initramfs. You only need to do this once, so no need to include it in your build script.
If you want, you can compile Busybox with specific options, to tell it you want some special command available. I find the one provided by Archlinux sufficient.
Your Init script
Here things are starting to be interesting. After the kernel boots, it will launch “/init” as process 0. On your distribution, this is probably systemd.
We don’t need something that huge, so we are just going to write our own as a shell script (remember busybox: it provides us with /bin/sh !)
What do you want to do in your init?
Initializing the system
We want to mount our /proc and /sys (add debugfs if you do kernel debugging), and populate /dev a bit (create /dev/zero, ttys, and other required devices)
#!/bin/sh # Don't forget the shebang on the first line /bin/mount -t proc none /proc /bin/mount -t sysfs sysfs /sys echo "> Populating /dev/" /bin/mdev -s
Loading modules
Well obviously, modules are not going to load themselves right? Use /bin/modprobe (provided by Busybox)
# ehci = qemu usb 2.0
# uhci = qemu usb 1.1
# add other modules as you see fit
MODULES="ehci-pci ehci-hcd uhci-hcd"
for module in $MODULES; do
echo "> Loading module $module"
/bin/modprobe $module
done
Give you control
Let’s not forget the most important part, right? If the init script exited your system would end.
/bin/sh
Generating your Initramfs
We need the cpio command
cd $INITRAMFS_DIR find . -print0 | cpio --null -ov --format=newc | gzip - > $ROOT_DIR/initramfs.img cd $ROOT_DIR
This will list all files in your ramfs directory, send them to cpio, and compress the image generated by cpio with gzip.
Launching QEmu
qemu-system-x86_64 \ -m 1024 \ -kernel linux/arch/x86_64/boot/bzImage \ -initrd initramfs.img \ -append 'console=ttyS0' \ -nographic \ -usb \ -device usb-ehci,id=ehci \ -device usb-host,bus=usb-bus.0,vendorid=0x046d,productid=0xc52b \ -device usb-tablet
Let’s inspect each line :
- -m 1024 will give 1GB to the system. This is definitely not required, ignore it or adapt it as you want
- -kernel and -initrd are ovious : you want to provide the generated initramfs and kernel
- -append ‘console=ttyS0’ and -nographic makes QEmu not create a new window, and redirect all the output to the terminal you used to launch it. If you want to access the QEmu console, use the shortcut <Ctl-a c>
- -usb enables usb. It will create ” usb-bus” hub to connect your USB 1.1 devices to.
- -device usb-ehci,id=ehci will create a “ehci” hub to connect your USB 2.0 devices to.
- -device usb-tablet creates a QEmu special “tablet” pointer. It will connect to your ehci hub automatically
- -device usb-host,bus=usb-bus.0,vendorid=0x046d,productid=0xc52b will pass the control from an usb device connected to your host, to the virtual machine. We provide the vendorid and productid of the device (as returned by lsusb), and tell it to connect to the first port of the usb-bus (USB 1.1) hub.
If your computer is recent enough and provides virtualization instructions, you could add the -enable-kvm option. Without it, the system takes about 4 seconds to boot on my computer.