nbdkit 1.24 & libnbd 1.6, new copying tool

As well as nbdkit 1.24 being released on Thursday, its sister project libnbd 1.6 was released at the same time. This comes with an enhanced copying tool called nbdcopy designed to replace some uses of qemu-img convert (note: it’s not a general replacement).

nbdcopy lets you copy from and to NBD servers (nbdkit, qemu-nbd, qemu-storage-daemon, nbd-server), local files, local block devices, pipes/sockets, and stdin/stdout. For example to stream the content of an NBD server:

$ nbdcopy nbd://localhost - | hexdump -C

The “-” character streams to stdout. nbd://localhost is an NBD URI referring to an NBD server that is already running. What if you don’t have an already running server? nbdcopy lets you run one from the command line (and cleans up after). For example this is one way to convert a qcow2 file to raw:

$ nbdcopy -- [ qemu-nbd -f qcow2 disk.qcow ] disk.raw

Here the [ ... ] section starts qemu-nbd as a captive NBD server, exposing privately an NBD endpoint, and nbdcopy copies this to local file disk.raw. (“--” is needed to stop nbdcopy trying to interpret qemu-nbd’s own command line arguments.)

However this post is really about the nbdkit release. How did I test and benchmark nbdcopy? Of course I wrote an nbdkit plugin called nbdkit-sparse-random-plugin. This plugin has two clever features for testing copying tools. Firstly it creates random disks which have the same “shape” as virtual machine disk images (but without the overhead of needing to bother with an actual VM). Secondly it can act as both a source and target for testing copies.

Let’s unpack those two things a bit further.

Virtual machine disk images (especially mostly empty ones) are mostly sparse. Here’s part of the sparse map from a Fedora 32 disk image:

$ virt-builder fedora-32
$ filefrag -e fedora-32.img 
 Filesystem type is: 58465342
 File size of fedora-32.img is 6442450944 (1572864 blocks of 4096 bytes)
  ext:     logical_offset:        physical_offset: length:   expected: flags:
    0:        0..       0:    2038672..   2038672:      1:            
    1:        1..      15:    2176040..   2176054:     15:    2038673:
    2:      256..     271:    2188819..   2188834:     16:    2176295:
    3:      512..    3135:    3650850..   3653473:   2624:    2189075:
    4:     3168..    4463:    3781763..   3783058:   1296:    3653506:
[...]

The new sparse-random plugin generates a disk image which has a similar shape — islands of random data in a sea of sparseness. The algorithm for doing this is quite neat. Because the plugin doesn’t need to store the data, unlike a real disk image, it can generate huge disk images (eg. a terabyte) while using almost no memory. We use a low-overhead, high-quality random number generator and are smart about seeds so that every run of sparse-random with the same seed produces identical output.

The other part of this plugin is how we can use it to test copying tools like nbdcopy and qemu-img convert. My idea was that the plugin could be used both as the source and the target of the copy:

$ nbdkit -U - sparse-random 1T --run ' nbdcopy "$uri" "$uri" '

Here we create a terabyte-sized sparse-random disk, and get nbdcopy to copy from the plugin to the plugin. On reads sparse-random supplies the sparseness and random data. On writes it checks if what is being written matches the content of the plugin, throwing -EIO errors if not. Assuming the copying tool is correctly handling errors, we can both validate the copying tool and benchmark it. And it works with qemu-img convert too:

$ nbdkit -U - sparse-random 1T --run ' qemu-img convert "$uri" "$uri" '

And now we can see which one is faster.

Try it, you may be surprised.

New in virt-sysprep: Set root and user passwords

New in virt-sysprep ≥ 1.23.13 is the ability to set root and user passwords in Linux guests (previous manual method is described here).

$ virt-sysprep \
  --root-password password:123456 \
  --password joe:file:/tmp/secret -a guest.img

You shouldn’t normally specify the cleartext password on the command line, although it’s useful for testing. You should usually provide a file containing the password, ensuring that it is not readable by other users on the system (ie. mode 0600).

virt-rescue

Virt-rescue is a new tool proposed for inclusion in libguestfs. It lets you get a rescue shell on your virtual machine, good for making quick, interactive, unstructured fixes:

$ virt-rescue F11x64

Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you will be able to examine them.

><rescue> /sbin/e2fsck /dev/vg_f11x64/lv_root
[...]
><rescue> mount /dev/vg_f11x64/lv_root /sysroot
><rescue> ls /sysroot/
bin   dev  home  lib64       media  opt   root  selinux  sys  usr
boot  etc  lib   lost+found  mnt    proc  sbin  srv      tmp  var
><rescue> sync
><rescue> umount /sysroot
><rescue> exit

Of course we encourage you to continue using libguestfs and guestfish for making properly structured changes through a stable, programmable API!

libguestfs – fun with tar

I’ve added tar support to libguestfs. Two examples:

Extract a directory subtree

This pulls out /usr/share/doc from the guest and saves it as a local file called /tmp/doc.tar.gz:

$ guestfish -a guest.img -m /dev/VolGroup00/LogVol00 

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> tgz-out /usr/share/doc /tmp/doc.tar.gz

Convert a tarball to an ext3 filesystem

$ cat /tmp/tar2ext3.sh
#!/bin/sh -
if [ $# -ne 2 ]; then
    echo "tar2ext3 input.tgz output.img"; exit 1
fi
guestfish <<EOF
alloc $2 100M
run
sfdisk /dev/sda 0 0 0 ,
mkfs ext3 /dev/sda1
mount /dev/sda1 /
tgz-in $1 /
sync
EOF
$ /tmp/tar2ext3.sh libguestfs-1.0.2.tar.gz test.img
$ ll test.img
-rw-rw-r--. 1 rjones rjones 104857600 2009-04-20 15:49 test.img

Now test.img contains all those files, inside an ext3 filesystem.

libguestfs 1.0.0 released, and Ruby bindings

libguestfs 1.0.0 is out, sources and binary RPMs here.

I also added Ruby bindings:

$ irb 
irb(main):002:0> require 'guestfs'
=> true
irb(main):003:0> g = Guestfs::create()
=> #<Guestfs::Guestfs:0x7feab981b630>
irb(main):004:0> g.add_drive("RHEL52PV32.img")
=> nil
irb(main):005:0> g.launch()
=> nil
irb(main):006:0> g.wait_ready()
=> nil
irb(main):007:0> g.lvs()
=> ["/dev/VolGroup00/LogVol00", "/dev/VolGroup00/LogVol01"]
irb(main):008:0> g.pvs()
=> ["/dev/sda2"]
irb(main):009:0> g.vgs()
=> ["VolGroup00"]
irb(main):010:0> g.list_partitions()
=> ["/dev/sda1", "/dev/sda2"]

libguestfs ocaml bindings

In addition to perl bindings, libguestfs now has OCaml bindings:

open Printf

let () =
  if Array.length Sys.argv <= 1 ||
    not (Sys.file_exists Sys.argv.(1)) then (
    eprintf "Usage: lvs guest.img\n";
    exit 1
  );

  let h = Guestfs.create () in
  Guestfs.add_drive h Sys.argv.(1);
  Guestfs.launch h;
  Guestfs.wait_ready h;

  let pvs = Guestfs.pvs h in
  printf "PVs found: [ %s ]\n"
    (String.concat "; " (Array.to_list pvs));

  let vgs = Guestfs.vgs h in
  printf "VGs found: [ %s ]\n"
    (String.concat "; " (Array.to_list vgs));

  let lvs = Guestfs.lvs h in
  printf "LVs found: [ %s ]\n"
    (String.concat "; " (Array.to_list lvs));

which produces the following output:

$ ./lvs RHEL52PV32.img
PVs found: [ /dev/sda2 ]
VGs found: [ VolGroup00 ]
LVs found: [ /dev/VolGroup00/LogVol00; /dev/VolGroup00/LogVol01 ]

libguestfs perl bindings

You can now use Perl to examine and modify virtual machine disk images through libguestfs, as in this example:

#!/usr/bin/perl -w
use strict;
use Sys::Guestfs;

# Look for LVM LVs, VGs and PVs in a guest image.

die "Usage: lvs.pl guest.img\n" if @ARGV != 1 || ! -f $ARGV[0];

my $h = Sys::Guestfs->new ();
$h->add_drive ($ARGV[0]);

print "Launching, this can take a few seconds\n";
$h->launch ();
$h->wait_ready ();

print "Looking for PVs on the disk image\n";
my @pvs = $h->pvs ();
print "PVs found: (", join (", ", @pvs), ")\n";

print "Looking for VGs on the disk image\n";
my @vgs = $h->vgs ();
print "VGs found: (", join (", ", @vgs), ")\n";

print "Looking for LVs on the disk image\n";
my @lvs = $h->lvs ();
print "LVs found: (", join (", ", @lvs), ")\n";

Which produces this output:

$ ./lvs.pl RHEL52PV32.img
Creating the libguestfs handle
Launching, this can take a few seconds
Looking for PVs on the disk image
PVs found: (/dev/sda2)
Looking for VGs on the disk image
VGs found: (VolGroup00)
Looking for LVs on the disk image
LVs found: (/dev/VolGroup00/LogVol00, /dev/VolGroup00/LogVol01)

More libguestfs progress

Now with added ls!

><fs> mount /dev/sda1 /
><fs> ll /
total 13
drwxr-xr-x  3 root root  1024 Apr  4 04:40 .
drwxr-xr-x 18 root root     0 Apr  4 11:36 ..
-rw-r--r--  1 root root     0 Apr  4 04:40 goodbye
-rw-r--r--  1 root root     0 Apr  3 17:23 hello
drwx------  2 root root 12288 Apr  3 15:27 lost+found
><fs> touch /testing-1-2-3
><fs> ll /
total 13
drwxr-xr-x  3 root root  1024 Apr  4 11:37 .
drwxr-xr-x 18 root root     0 Apr  4 11:36 ..
-rw-r--r--  1 root root     0 Apr  4 04:40 goodbye
-rw-r--r--  1 root root     0 Apr  3 17:23 hello
drwx------  2 root root 12288 Apr  3 15:27 lost+found
-rw-r--r--  1 root root     0 Apr  4 11:37 testing-1-2-3

The more serious point to that was to make sure that the RPC protocol we’re using (based on XDR) can transfer strings and lists of strings in both directions, which will be necessary for the device / LV enumeration APIs.

libguestfs modifying a guest image

Here we have a virtual machine image (test1.img) that contains an MBR-style partition with an ext2 filesystem on the first partition, and we create a file /hello on that filesystem. For more explanation see this posting. Very experimental code is in this git repository. Note that we aren’t building or running any of this as root.

$ guestfish 

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> help
    Command         Description
help            display a list of commands or help on a command
quit            quit guestfish
add             add a guest image to be examined or modified
cdrom           add a CD-ROM image to be examined
launch          launch the subprocess
mount           mount a guest disk at a position in the filesystem
sync            sync disks, writes are flushed through to the disk image
touch           update file timestamps or create a new file
    Use -h <cmd> / help <cmd> to show detailed help for a command.
><fs> add test1.img
><fs> launch
><fs> mount /dev/sda1 /
><fs> touch /hello
><fs> sync
><fs> exit

Why “minimal” is 225 MB

As I mentioned in the last post a “minimal” febootstrap Fedora install clocks in at a staggering 225 MB. When I say minimal, I mean just bash and the simplest command-line tools from coreutils:

$ ls /bin
arch      chgrp  cut   echo   fgrep  ls      mv    rmdir  stty   true
basename  chmod  date  egrep  grep   mkdir   nice  sh     su     uname
bash      chown  dd    env    link   mknod   pwd   sleep  sync   unlink
cat       cp     df    false  ln     mktemp  rm    sort   touch

Where does all the space go?

Thanks to KDE’s filelight tool, we can easily visualize the disk usage, in a nice interactive graphical way.

34% of the total disk space (76 MB) is taken up with a single file, /usr/lib/locale/locale-archive. We suspect this is an optional file that contains all locale information and is mapped into every glibc-using process. Since the minimal image I have in mind is non-interactive, there doesn’t seem to be much point in having locales at all, and this can be deleted. Obviously if you wanted an interactive, internationalized Fedora, you can’t just go and remove this file.

Another 34% is taken up with the yum cache, ie. the packages that we installed. This just needs to be deleted, and febootstrap should have an option to do this automatically.

6% (15 MB) are the locale files. As explained above, these can go.

3% (8 MB) is, extraordinarily, cracklib. It turns out that coreutils requires pam, which requires cracklib to test the strength of passwords. This is completely useless for us, because the virtual machine image won’t even have a login prompt, never mind the ability to change passwords.

A further 5% is documentation, man pages and i18n stuff that we don’t care about.

Just removing the above brings the image down to 38 MB. The next step will be to do some much more aggressive minimization, based on analyzing the binaries that we’re actually going to use and their dependencies.