Tag Archives: virt-tools

libguestfs appliance boot in under 1s

$ time LIBGUESTFS_BACKEND=direct LIBGUESTFS_HV=~/d/qemu/x86_64-softmmu/qemu-system-x86_64 guestfish -a /dev/null run

real	0m0.966s
user	0m0.623s
sys	0m0.281s

However I had to patch qemu to enable DMA loading of the kernel and initrd.


1 Comment

Filed under Uncategorized

Getting the libguestfs appliance boot time down to 1.2s

libguestfs can securely mount any disk image, but to do this it requires a small appliance to be run. The appliance is a very cut down Linux distro, but it still takes time to boot. For a while that time has floated around 3-5 seconds. This excludes libguestfs from some important use cases — one being the ability to monitor 1000s of VMs every few minutes (simple maths: 1000×3 > 5×60, so you cannot monitor 1000 VMs every 5 minutes without using a lot of parallel appliances).

Last year Intel announced Clear Containers. You may be forgiven for being confused (unclear?) by what Clear Containers actually is, but Intel’s demo is quite neat. (You can run these commands as non-root, and at time of writing they won’t damage your machine.)

$ wget https://download.clearlinux.org/demos/containers/clear-containers-demo.tar.xz
$ mv clear-containers-demo.tar.xz clear-containers-demo.tar.bz2
$ bunzip clear-containers-demo.tar.bz2
$ cd containers
$ bash ./boot.sh

It’s a complete Linux guest that boots in a fraction of a second. I take that as a challenge!

The first step is to have a good idea what all the parts are doing and what is taking the time. Booting an appliance involves several actors — qemu, BIOS, the guest kernel — and without being able to measure how much time each one spends doing things, it’s rather hard to say what needs work or if we’re making improvements. This was why I spent last week unsuccessfully looking at QEMU tracing. I have now settled on a simpler approach which is to time boot messages. The new boot analysis program produces quite clear output:

Warming up the libguestfs cache ...                                                                              
Running the tests in 5 passes ...                                                                                
    pass 1: 798 events collected in 1347184178 ns                                                                
    pass 2: 798 events collected in 1324153548 ns                                                                
    pass 3: 798 events collected in 1342126721 ns                                                                
    pass 4: 798 events collected in 1279500931 ns                                                                
    pass 5: 798 events collected in 1317457653 ns                                                                
Analyzing the results ...                                                                                        
0.000000s: ▲ run mean:1.321973s ±24.0ms (100.0%)                                                              
0.000065s: │ ▲ supermin:build mean:0.010523s ±0.1ms (0.8%)                                                  
           │ │                                                                                               
0.010588s: │ ▼                                                                                               
0.010612s: │ ▲ qemu:feature-detect mean:0.149075s ±4.2ms (11.3%)                                            
           │ │                                                                                               
0.159687s: │ ▼                                                                                               
0.161412s: │ ▲ ▲ qemu mean:1.160562s ±22.6ms (87.8%) qemu:overhead mean:0.123142s ±4.5ms (9.3%)          
           │ │ │                                                                                           
0.263153s: │ │ │ ▲ seabios mean:0.241488s ±2.8ms (18.3%)                                                
           │ │ │ │                                                                                       
0.284554s: │ │ ▼ │                                                                                       
0.284554s: │ │   │ ▲ bios:overhead mean:0.220087s ±2.8ms (16.6%)                                        
           │ │   │ │                                                                                     
0.504641s: │ │   ▼ ▼                                                                                     
0.504641s: │ │ ▲ ▲ kernel mean:0.817332s ±21.4ms (61.8%) kernel:overhead mean:0.374896s ±5.2ms (28.4%) 
           │ │ │ │                                                                                       
0.879537s: │ │ │ ▼                                                                                       
0.879537s: │ │ │ ▲ supermin:mini-initrd mean:0.086014s ±7.9ms (6.5%)                                    
           │ │ │ │                                                                                       
0.881863s: │ │ │ │ ▲ supermin: internal insmod crc32-pclmul.ko mean:0.001399s ±0.1ms (0.1%)           
           │ │ │ │ │                                                                                   
0.883262s: │ │ │ │ ▼                                                                                   
0.883262s: │ │ │ │ ▲ supermin: internal insmod crc32c-intel.ko mean:0.000226s ±0.5ms (0.0%)           
0.883488s: │ │ │ │ ▼                                                                                   
0.883488s: │ │ │ │ ▲ supermin: internal insmod crct10dif-pclmul.ko mean:0.000882s ±0.4ms (0.1%)       
0.884370s: │ │ │ │ ▼                                                                                   
0.884370s: │ │ │ │ ▲ supermin: internal insmod crc32.ko mean:0.001121s ±0.0ms (0.1%)                  
           │ │ │ │ │                                                                                   
0.885490s: │ │ │ │ ▼                                                                                   
0.885490s: │ │ │ │ ▲ supermin: internal insmod virtio.ko mean:0.001634s ±0.5ms (0.1%)                 
           │ │ │ │ │                                                                                   
0.887124s: │ │ │ │ ▼                                                                                   
0.887124s: │ │ │ │ ▲ supermin: internal insmod virtio_ring.ko mean:0.000581s ±0.7ms (0.0%)            
0.887706s: │ │ │ │ ▼                                                                                   
0.887706s: │ │ │ │ ▲ supermin: internal insmod virtio_blk.ko mean:0.001115s ±0.0ms (0.1%)             
           │ │ │ │ │                                                                                   
0.888821s: │ │ │ │ ▼                                                                                   
0.888821s: │ │ │ │ ▲ supermin: internal insmod virtio-rng.ko mean:0.000884s ±0.4ms (0.1%)             
0.889705s: │ │ │ │ ▼                                                                                   
0.889705s: │ │ │ │ ▲ supermin: internal insmod virtio_console.ko mean:0.001923s ±0.4ms (0.1%)         
           │ │ │ │ │                                                                                   
0.891627s: │ │ │ │ ▼                                                                                   
0.891627s: │ │ │ │ ▲ supermin: internal insmod virtio_net.ko mean:0.001483s ±0.4ms (0.1%)             
           │ │ │ │ │                                                                                   
0.893111s: │ │ │ │ ▼                                                                                   
0.893111s: │ │ │ │ ▲ supermin: internal insmod virtio_scsi.ko mean:0.000686s ±0.6ms (0.1%)            
0.893797s: │ │ │ │ ▼                                                                                   
0.893797s: │ │ │ │ ▲ supermin: internal insmod virtio_balloon.ko mean:0.000663s ±0.5ms (0.1%)         
0.894460s: │ │ │ │ ▼                                                                                   
0.894460s: │ │ │ │ ▲ supermin: internal insmod virtio_input.ko mean:0.000875s ±0.4ms (0.1%)           
0.895336s: │ │ │ │ ▼                                                                                   
0.895336s: │ │ │ │ ▲ supermin: internal insmod virtio_mmio.ko mean:0.001097s ±0.0ms (0.1%)            
           │ │ │ │ │                                                                                   
0.896433s: │ │ │ │ ▼                                                                                   
0.896433s: │ │ │ │ ▲ supermin: internal insmod virtio_pci.ko mean:0.050700s ±7.8ms (3.8%)             
           │ │ │ │ │                                                                                   
0.947133s: │ │ │ │ ▼                                                                                   
0.947133s: │ │ │ │ ▲ supermin: internal insmod crc-ccitt.ko mean:0.001144s ±0.6ms (0.1%)              
           │ │ │ │ │                                                                                   
0.948277s: │ │ │ │ ▼                                                                                   
0.948277s: │ │ │ │ ▲ supermin: internal insmod crc-itu-t.ko mean:0.000001s ±0.0ms (0.0%)              
0.948278s: │ │ │ │ ▼                                                                                   
0.948278s: │ │ │ │ ▲ supermin: internal insmod crc8.ko mean:0.001368s ±0.3ms (0.1%)                   
           │ │ │ │ │                                                                                   
0.949646s: │ │ │ │ ▼                                                                                   
0.949646s: │ │ │ │ ▲ supermin: internal insmod libcrc32c.ko mean:0.001043s ±0.9ms (0.1%)              
           │ │ │ │ │                                                                                   
0.950689s: │ │ │ │ ▼                                                                                   
           │ │ │ │                                                                                       
0.965551s: │ │ │ ▼                                                                                       
0.965551s: │ │ │ ▲ ▲ /init mean:0.318045s ±18.0ms (24.1%) bash:overhead mean:0.015855s ±3.1ms (1.2%) 
           │ │ │ │ │                                                                                   
0.981407s: │ │ │ │ ▼                                                                                   
           │ │ │ │                                                                                       
1.283597s: │ │ │ ▼                                                                                       
1.283597s: │ │ │ ▲ guestfsd mean:0.019151s ±1.9ms (1.4%)                                                
           │ │ │ │                                                                                       
1.294818s: │ │ │ │ ▲ shutdown mean:0.027156s ±4.1ms (2.1%)                                            
           │ │ │ │ │                                                                                   
1.302747s: │ │ │ ▼ │                                                                                   
           │ │ │   │                                                                                     
1.321973s: │ │ ▼   │                                                                                     
1.321973s: ▼ ▼     ▼                                                                                       

Armed with this analysis I made a good start on reducing the boot time. It’s now down to 1.2s (on my laptop) and there is scope for sub-second boots.

Some of the things I’ve changed to get to 1.2s:

Some of the things that may reduce boot times further:

  • Stop SeaBIOS from probing the entire PCI space looking for a boot device it will never use.
  • Implement DAX so that the appliance can execute files directly from backing disk instead of loading them into RAM.
  • A much more detailed look at the qemu and kernel startup process, taking a knife to anything that unnecessarily sleeps or wastes time.

By the way: Even if you never use libguestfs, but you do use virtualized linux, this benefits you too.


Filed under Uncategorized

New in libguestfs: Filesystem forensics support

Thanks to patches supplied by Matteo Cafasso, libguestfs, the library for accessing and modifying disk images is gradually getting support for filesystem forensics.

Initially I have added a Fedora libguestfs-forensics subpackage, which pulls The Sleuth Kit (TSK) into virt-rescue.

Parts of TSK will also be made available as libguestfs APIs so they are callable from other programs.


Filed under Uncategorized

Tip: FUSE-mount a disk image with Windows drive letters

guestmount is the libguestfs tool for taking a disk image and mounting it under the host filesystem. This works great for Linux disk images:

$ virt-builder centos-7.2
$ mkdir /tmp/mnt
$ guestmount -a centos-7.2.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ guestunmount /tmp/mnt

Those files under /tmp/mnt are inside the centos-7.2.img disk image file, and you can read and write them.

guestmount is fine for Windows disk images too, except when Windows has multiple drives, C:, D:, etc., because in that case you’ll only “see” the contents of the C: drive.

But guestmount is nowadays just a wrapper around the “mount-local” API in libguestfs, and you can use that API directly if you want to do anything a bit more complicated … such as exposing Windows drive letters.

Here is a Perl script which uses the mount-local API directly to do this:

#!/usr/bin/perl -w
use strict;
use Sys::Guestfs;
$| = 1;
die "usage: $0 mountpoint disk.img" if @ARGV < 2;
my $mp = shift @ARGV;
my $g = new Sys::Guestfs;
$g->add_drive_opts ($_) foreach @ARGV;
my @roots = $g->inspect_os;
die "$0: no operating system found" if @roots != 1;
my $root = $roots[0];
die "$0: not Windows" if $g->inspect_get_type ($root) ne "windows";
my %map = $g->inspect_get_drive_mappings ($root);
foreach (keys %map) {
    $g->mkmountpoint ("/$_");
    eval { $g->mount ($map{$_}, "/$_") };
    warn "$@ (ignored)\n" if $@;
$g->mount_local ($mp);
print "filesystem mounted on $mp\n";

You can use it like this:

$ mkdir /tmp/mnt
$ ./drive-letters.pl /tmp/mnt windows7.img
filesystem ready on /tmp/mnt

in another window:

$ cd /tmp/mnt
$ ls
C  D
$ cd C
$ ls
Documents and Settings
Program Files
$ cd ../..
$ guestunmount /tmp/mnt

(Thanks to Pino Toscano for working out the details)

Leave a comment

Filed under Uncategorized

CentOS 7.2, with added virt-v2v

CentOS 7.2 is out. New in this release, you can simply install virt-v2v and all the dependencies (including libguestfs-winsupport!) are included*.

* = except for libguestfs-xfs which you might also need for converting RHEL 7 guests.

# yum install virt-v2v
Dependencies Resolved

 Package                         Arch   Version                   Repository
 virt-v2v                        x86_64 1:1.28.1-1.55.el7.centos  base     12 M
Installing for dependencies:
 attr                            x86_64 2.4.46-12.el7             base     66 k
 augeas-libs                     x86_64 1.4.0-2.el7               base    355 k
 boost-system                    x86_64 1.53.0-25.el7             base     39 k
 boost-thread                    x86_64 1.53.0-25.el7             base     57 k
 bridge-utils                    x86_64 1.5-9.el7                 base     32 k
 bzip2                           x86_64 1.0.6-13.el7              base     52 k
 celt051                         x86_64             base     53 k
 cryptsetup                      x86_64 1.6.7-1.el7               base    119 k
 cyrus-sasl                      x86_64 2.1.26-19.2.el7           base     88 k
 cyrus-sasl-md5                  x86_64 2.1.26-19.2.el7           base     56 k
 device-mapper-event             x86_64 7:1.02.107-5.el7          base    167 k
 device-mapper-event-libs        x86_64 7:1.02.107-5.el7          base    169 k
 device-mapper-persistent-data   x86_64 0.5.5-1.el7               base    350 k
 dosfstools                      x86_64 3.0.20-9.el7              base    101 k
 flac-libs                       x86_64 1.3.0-5.el7_1             base    169 k
 fuse                            x86_64 2.9.2-6.el7               base     84 k
 fuse-libs                       x86_64 2.9.2-6.el7               base     93 k
 gdisk                           x86_64 0.8.6-5.el7               base    187 k
 genisoimage                     x86_64 1.1.11-23.el7             base    298 k
 glusterfs                       x86_64 3.7.1-16.el7              base    422 k
 glusterfs-api                   x86_64 3.7.1-16.el7              base     60 k
 glusterfs-client-xlators        x86_64 3.7.1-16.el7              base    732 k
 glusterfs-libs                  x86_64 3.7.1-16.el7              base    321 k
 gnutls-dane                     x86_64 3.3.8-12.el7_1.1          base     32 k
 gnutls-utils                    x86_64 3.3.8-12.el7_1.1          base    228 k
 gperftools-libs                 x86_64 2.4-7.el7                 base    272 k
 gsm                             x86_64 1.0.13-11.el7             base     30 k
 gtk2                            x86_64 2.24.28-8.el7             base    3.4 M
 hexedit                         x86_64 1.2.13-5.el7              base     39 k
 hivex                           x86_64 1.3.10-5.7.el7            base    100 k
 icoutils                        x86_64 0.31.0-3.el7              base     82 k
 ipxe-roms-qemu                  noarch 20130517-7.gitc4bce43.el7 base    559 k
 iscsi-initiator-utils           x86_64          base    418 k
 iscsi-initiator-utils-iscsiuio  x86_64          base     81 k
 ldns                            x86_64 1.6.16-7.el7              base    473 k
 libaio                          x86_64 0.3.109-13.el7            base     24 k
 libasyncns                      x86_64 0.8-7.el7                 base     26 k
 libcgroup                       x86_64 0.41-8.el7                base     64 k
 libconfig                       x86_64 1.4.9-5.el7               base     59 k
 libguestfs                      x86_64 1:1.28.1-1.55.el7.centos  base    1.7 M
 libguestfs-tools-c              x86_64 1:1.28.1-1.55.el7.centos  base    2.2 M
 libguestfs-winsupport           x86_64 7.2-1.el7                 base    2.1 M
 libibverbs                      x86_64 1.1.8-8.el7               base     56 k
 libicu                          x86_64 50.1.2-15.el7             base    6.9 M
 libiscsi                        x86_64 1.9.0-6.el7               base     60 k
 libogg                          x86_64 2:1.3.0-7.el7             base     24 k
 libosinfo                       x86_64 0.2.12-3.el7              base    256 k
 librados2                       x86_64 1:0.80.7-3.el7            base    1.5 M
 librbd1                         x86_64 1:0.80.7-3.el7            base    350 k
 librdmacm                       x86_64 1.0.21-1.el7              base     64 k
 libreport-filesystem            x86_64 2.1.11-32.el7.centos      base     37 k
 libseccomp                      x86_64 2.2.1-1.el7               base     49 k
 libsndfile                      x86_64 1.0.25-10.el7             base    149 k
 libunwind                       x86_64 1.1-5.el7                 base     56 k
 libusal                         x86_64 1.1.11-23.el7             base    135 k
 libvirt-client                  x86_64 1.2.17-13.el7_2.2         updates 4.3 M
 libvirt-daemon                  x86_64 1.2.17-13.el7_2.2         updates 584 k
 libvirt-daemon-driver-interface x86_64 1.2.17-13.el7_2.2         updates 161 k
 libvirt-daemon-driver-network   x86_64 1.2.17-13.el7_2.2         updates 301 k
 libvirt-daemon-driver-nodedev   x86_64 1.2.17-13.el7_2.2         updates 160 k
 libvirt-daemon-driver-nwfilter  x86_64 1.2.17-13.el7_2.2         updates 184 k
 libvirt-daemon-driver-qemu      x86_64 1.2.17-13.el7_2.2         updates 569 k
 libvirt-daemon-driver-secret    x86_64 1.2.17-13.el7_2.2         updates 154 k
 libvirt-daemon-driver-storage   x86_64 1.2.17-13.el7_2.2         updates 327 k
 libvirt-daemon-kvm              x86_64 1.2.17-13.el7_2.2         updates 117 k
 libvorbis                       x86_64 1:1.3.3-8.el7             base    204 k
 libxslt                         x86_64 1.1.28-5.el7              base    242 k
 lsof                            x86_64 4.87-4.el7                base    331 k
 lvm2                            x86_64 7:2.02.130-5.el7          base    1.0 M
 lvm2-libs                       x86_64 7:2.02.130-5.el7          base    872 k
 lzop                            x86_64 1.03-10.el7               base     54 k
 mailcap                         noarch 2.1.41-2.el7              base     31 k
 make                            x86_64 1:3.82-21.el7             base    420 k
 mdadm                           x86_64 3.3.2-7.el7               base    393 k
 mtools                          x86_64 4.0.18-5.el7              base    203 k
 netcf-libs                      x86_64 0.2.8-1.el7               base     69 k
 netpbm                          x86_64 10.61.02-9.el7            base    180 k
 netpbm-progs                    x86_64 10.61.02-9.el7            base    1.9 M
 nmap-ncat                       x86_64 2:6.40-7.el7              base    201 k
 numad                           x86_64 0.5-14.20140620git.el7    base     34 k
 openssl                         x86_64 1:1.0.1e-51.el7_2.1       updates 711 k
 perl-Business-ISBN              noarch 2.06-2.el7                base     25 k
 perl-Business-ISBN-Data         noarch 20120719.001-2.el7        base     24 k
 perl-Compress-Raw-Bzip2         x86_64 2.061-3.el7               base     32 k
 perl-Compress-Raw-Zlib          x86_64 1:2.061-4.el7             base     57 k
 perl-Digest                     noarch 1.17-245.el7              base     23 k
 perl-Digest-MD5                 x86_64 2.52-3.el7                base     30 k
 perl-Encode-Locale              noarch 1.03-5.el7                base     16 k
 perl-File-Listing               noarch 6.04-7.el7                base     13 k
 perl-HTML-Parser                x86_64 3.71-4.el7                base    115 k
 perl-HTML-Tagset                noarch 3.20-15.el7               base     18 k
 perl-HTTP-Cookies               noarch 6.01-5.el7                base     26 k
 perl-HTTP-Daemon                noarch 6.01-5.el7                base     20 k
 perl-HTTP-Date                  noarch 6.02-8.el7                base     14 k
 perl-HTTP-Message               noarch 6.06-6.el7                base     82 k
 perl-HTTP-Negotiate             noarch 6.01-5.el7                base     17 k
 perl-IO-Compress                noarch 2.061-2.el7               base    260 k
 perl-IO-HTML                    noarch 1.00-2.el7                base     23 k
 perl-IO-Socket-IP               noarch 0.21-4.el7                base     35 k
 perl-IO-Socket-SSL              noarch 1.94-3.el7                base    113 k
 perl-LWP-MediaTypes             noarch 6.02-2.el7                base     24 k
 perl-Net-HTTP                   noarch 6.06-2.el7                base     29 k
 perl-Net-LibIDN                 x86_64 0.12-15.el7               base     28 k
 perl-Net-SSLeay                 x86_64 1.55-3.el7                base    285 k
 perl-TimeDate                   noarch 1:2.30-2.el7              base     52 k
 perl-URI                        noarch 1.60-9.el7                base    106 k
 perl-WWW-RobotRules             noarch 6.02-5.el7                base     18 k
 perl-hivex                      x86_64 1.3.10-5.7.el7            base     41 k
 perl-libwww-perl                noarch 6.05-2.el7                base    205 k
 pulseaudio-libs                 x86_64 6.0-7.el7                 base    576 k
 python-chardet                  noarch 2.2.1-1.el7_1             base    227 k
 python-kitchen                  noarch 1.1.1-5.el7               base    267 k
 qemu-img                        x86_64 10:1.5.3-105.el7_2.1      updates 657 k
 qemu-kvm                        x86_64 10:1.5.3-105.el7_2.1      updates 1.8 M
 qemu-kvm-common                 x86_64 10:1.5.3-105.el7_2.1      updates 390 k
 radvd                           x86_64 1.9.2-9.el7               base     85 k
 rsyslog-mmjsonparse             x86_64 7.4.7-12.el7              base     29 k
 scrub                           x86_64 2.5.2-5.el7               base     40 k
 seabios-bin                     noarch 1.7.5-11.el7              base     74 k
 seavgabios-bin                  noarch 1.7.5-11.el7              base     32 k
 sgabios-bin                     noarch 1:0.20110622svn-4.el7     base    7.1 k
 spice-server                    x86_64 0.12.4-15.el7             base    380 k
 supermin5                       x86_64 5.1.10-1.2.el7            base    574 k
 syslinux                        x86_64 4.05-12.el7               base    990 k
 syslinux-extlinux               x86_64 4.05-12.el7               base    363 k
 unbound-libs                    x86_64 1.4.20-26.el7             base    296 k
 unzip                           x86_64 6.0-15.el7                base    166 k
 usbredir                        x86_64 0.6-7.el7                 base     44 k
 yajl                            x86_64 2.0.4-4.el7               base     39 k
 yum-utils                       noarch 1.1.31-34.el7             base    113 k

Transaction Summary
Install  1 Package (+130 Dependent packages)

Total download size: 59 M
Installed size: 172 M

Leave a comment

Filed under Uncategorized

Inspection, now with added Prolog

You can give libguestfs an unknown disk image, and it tries to guess what’s on it, in terms of operating systems, Linux distro, Windows drive letter mappings and so on, a process that we call inspection. This is an important part of many of the virt tools, because when you type a command like

$ virt-cat -a linux.img /var/log/messages

how is virt-cat to know that on this particular disk image the sysadmin put /var on a separate partition? Because, inspection.

Given that inspection is such an important part of many tools, and vital for standalone programs like virt-inspector you might wonder how it works.

The answer, right now, is 6000+ lines of hairy, intricate C code, which is difficult to maintain and a source of hard to fix bugs and hard to implement feature requests.

$ wc -l src/inspect*.c
   823 src/inspect-apps.c
   725 src/inspect.c
   777 src/inspect-fs.c
   543 src/inspect-fs-cd.c
  2092 src/inspect-fs-unix.c
   704 src/inspect-fs-windows.c
   600 src/inspect-icon.c
  6264 total

How can we make this better?

Getting back to basics, inspection is really a lot of heuristics. Things like:

  • If this filesystem contains a file /etc/fstab then it could be a Linux root filesystem. And:
  • If this thing we think is a Linux root filesystem contains /etc/debian_version then it could be a Debian root filesystem.

These heuristics can be expressed in a logic language inspired by Prolog:

LinuxRoot(fs) :-
    File(fs, "/etc/fstab").
DebianRoot(fs) :-
    File(fs, "/etc/debian_version").

What we’re doing here is collecting a set of facts (Prolog calls them “compound terms”), like:

File("/dev/sda1", "/etc/fstab").
File("/dev/sda1", "/etc/debian_version").

and deriving new facts using the rules:


(I should say at this point that I’m simplifying things a bit. If you want to get a flavour of what the inspection rules might finally look like, then take a look at this file.)

So far I have written a compiler that compiles inspection rules into fairly efficient C code (and hence to binaries), using a forward chaining strategy. It has some nice features like transparently embedding C code into the rules, allowing you to do more complicated operations directly in C:

Distro(fs, distro) :-
    LinuxRootWithOSRelease(fs), /* has /etc/os-release */
      int r;
      CLEANUP_FREE char *distro = NULL;
      if ((r = get_distro_from_os_release (fs, &distro))
           <= 0)
        return r;
      set_distro (distro);
      return 0;


BlockDevice(dev) :-
      CLEANUP_FREE_STRING_LIST char **devs =
        get_all_block_devices ();
      if (devs == NULL) return -1;
      for (size_t i = 0; devs[i] != NULL; ++i)
        set_dev (devs[i]);
      return 0;

My inspection rules run to < 500 lines of code so far, although it’s hard to compare that to the current code because (a) the inspection rules will likely double or triple in size once they are able to do everything that the current code can do, and (b) there’s a lot of supporting runtime code like get_all_block_devices above.

Nevertheless I hope the new rules system will be faster, more supportable and extensible, and easier to understand than the current code. It will also be 100% backwards compatible with existing libguestfs users (since we never break compatibility).

You can follow development in this branch on github.

Update: Hacker News discussion of this article.

Leave a comment

Filed under Uncategorized

Linux Kernel Library backend for libguestfs

For a very long time I’ve been keeping an eye on rump kernels. These are a fancy way of saying ordinary kernel code which has been ported or altered in some way so you can link it to userspace programs, usually as a library. There have been claims that rump kernels make libguestfs obsolete, which is a load of nonsense for reasons I’ll talk about in a minute. But rump kernels are interesting and finally I’ve been able to add a rump kernel library to libguestfs.

The first time I encountered rump kernels was in 2010 with Prof. Renzo Davoli’s virtualsquare View-OS project. The more recent rump kernel project was based on NetBSD. Since NetBSD doesn’t use Linux drivers, but reimplementations, it was never very interesting. We couldn’t have included it in RHEL, and it would be only a matter of time before someone found an important feature or data-corrupting difference between the real ext2/3 driver from Linux and the reimplementation in BSD. Last I looked, NetBSD didn’t even support ext4. The choice of NetBSD made it a non-starter for us.

libos arrived earlier this year and, being based on Linux, was a lot more interesting, but their aim was only the Linux network stack so it wasn’t directly relevant to libguestfs, which is only concerned with disk and filesystem access.

However last week Richard Weinberger (Linux UML maintainer) CC’d me on a much more interesting project. LKL (Linux Kernel Library) is a relatively small patchset on top of Linux that turns Linux into a library. Quite literally you just link your program with -llkl and you can start a Linux kernel and make Linux system calls. A few example programs can be found here.

Today I wrote a work-in-progress LKL backend for libguestfs. [Note for those keen to jump in and use it, it doesn’t do very much at the moment. There is a small to-do list. But it’s only a few hours more work to have it as a useful backend for libguestfs.]

What could an LKL-enabled libguestfs do? Well it could open a raw disk image that contains only a Linux filesystem, or possibly a partitioned disk image with a Linux filesystem. And it can do that very quickly – in about 0.03 seconds in my tests (compared to 2.5 seconds for the qemu-based libguestfs).

There are a long list of things that LKL cannot do however. It cannot open anything containing LVM (because LVM requires userspace tools, that would all have to be modified to use liblkl, and there would be all kinds of synchronization problems between the LVM-linked liblkl kernel and the libguestfs-linked kernel). It cannot run anything that requires a userspace tool. That includes btrfs, ntfs-3g (FUSE based), some MD devices, encrypted disks, dm-thinp and Windows Dynamic Disks.

It cannot create new LVs or filesystems or any of the other disk structures that libguestfs can create.

It cannot open disk formats like qcow2 and vmdk. The qcow2 code in particular is very specific to qemu, and cannot possibly be ported to the kernel.

It cannot open remote devices like Ceph, Gluster, https, ssh or ftp (although there is an nbd client in the kernel, so one day that may be possible).

It cannot run commands in the context of the guest.

It’s also less secure than libguestfs with the qemu backend, because it doesn’t isolate your host from exploits in the guest or filesystem using virtualization and sVirt.

All of those things can be done by libguestfs today, but may never be possible (or only in limited cases) with LKL.

But within the limits of what LKL can do, it should become an excellent choice (assuming it gets upstream). Adding an LKL backend to libguestfs brings to the table the large, but clean and well-tested libguestfs API and the language bindings and virt tools, so libguestfs gives you the best of both worlds: the performance of LKL by using that backend, or the massive feature set of libguestfs by switching over to the qemu or libvirt backend, which for most programs is a single line code change.


Filed under Uncategorized