Bring back the BIOS

… all is forgiven.

ARM booting sucks in several very concrete ways:

• As there is no standard display hardware nor standard serial port, it’s essentially impossible for a random firmware or kernel to output debug messages. (Like text mode + 16550 on a PC). The most likely outcome for the poor developer is a black screen of nothingness. This could be easily fixed if ARM allocated a standard serial port address and put it into every shipping ARM core.
• ARM instruction sets aren’t standardized. There’s no core of instructions that will work on any ARM (compare to 8086 on a PC). So there’s no way to write a bootloader that works everywhere. [For privileged instructions — unprivileged instructions are usually backwards compatible, except when they’re not (Thumb), but that doesn’t help when writing a bootloader.]
• There is a standard bootloader of sorts, or at least one which is very frequently used, called U-Boot. That’s good, but everyone and their uncle modifies it so it’s forked all over the place. And make sure you’ve got the exact fork for your hardware, otherwise you’ll brick it.
• U-Boot is generally bloody confusing, uses a binary blob + special tools for boot configuration, is hard to compile on x86, etc.
• U-Boot may not be replaceable on your hardware, or there may be some other, closed firmware underneath it (hello, R-Pi).
• The hardware isn’t discoverable and self-describing (unlike, say, PCI, USB and every recent PC peripheral). Thus you have Device Trees, big files that describe every piece of hardware, must be written specially for each piece of hardware, and must be correct else you’ll be looking at a black screen of death.
• Kernel signing. No standard toolset for signing kernels. Unfortunately this particular strain of madness is now infecting PCs too.

Anyhow, this explains how I’ve gone from 3 working Fedora ARM systems to 0 in the space of two days.

This is also the reason I think EFI is a terrible mistake. Having now used UEFI a lot more, I think it’s a vast improvement on the ARM bootloader zoo.

libguestfs launch times

Indulge me while a make a “note to self” about efforts to reduce the time taken by guestfs_launch which boots up the libguestfs appliance.

Time (s)	Operation
2s	Create supermin appliance: This has crept up over time from originally taking about 1/5th of a second to around 2s. Needs attention. Fixed see this note about cpio blocksize and update below.
2-3s	qemu startup: The time is mainly spent reading in the large -kernel and particularly -initrd files specified on the command line. The released qemu code is quite rubbish, but luckily kraxel beat me to fixing the problem with this patch.
3s	BIOS waits for keypress: As discussed yesterday, I’ve posted a patch. In the meantime the qemu devs have abandoned the old bochs BIOS for SeaBIOS, which I haven’t tested yet,
3s	Kernel boot time: Not very many easy wins here, since the kernel is already pretty efficient. We could try to remove some busy waits and sleeps — for example the kernel waits ¼s for the serial ports, which we don’t use.
1-2s	Userspace boot time: This is mainly time spent on udev and partition detection. I have never really understood why the kernel needs to pause so long on partition detection. Not much easy meat here, but improving the speed of udev in itself would be worthwhile.

I spotted a nice tool for turning strace output into timelines.

Update — bash globbing

Having solved the cpio problem, the largest bottleneck in creating the supermin appliance becomes globbing.

It’s probably not a well-known fact, but if you do:

ls *.c *.h

then bash reads the directory twice. It treats the two globs on the command line as completely separate entities. This is not so bad for a few globs, but when we make the supermin appliance we need to do over 120 globs, which takes bash about 0.8s to complete, contributing about 10% to the overall launch time. Bash is literally reading the same directory over and over again, 50 or more times.

At the moment it’s not obvious to me how to solve this.

Quick win

I got the libguestfs launch time down from 12 seconds to 9 seconds today, 25% faster!

It turns out that the appliance’s BIOS was waiting for 3 seconds for someone to hit [F12] on the imaginary keyboard. A simple patch to bochs BIOS fixes that … This patch benefits everyone using Fedora and virtualization, since all boot times will be reduced by 3 seconds.

This is only the start of the optimizations. I’m pretty certain we can get it down to a 4 or 5 second launch time.