Accessing Ceph (rbd), sheepdog, etc using libguestfs

Thanks to infernix who contributed this tip on how to use libguestfs to access Ceph (and in theory, sheepdog, gluster, iscsi and more) devices.

If you apply this small patch to libguestfs you can use these distributed filesystems straight away by doing:

$ guestfish
><fs> set-attach-method appliance
><fs> add-drive /dev/null
><fs> config -set drive.hd0.file=rbd:pool/volume
><fs> run

… followed by usual guestfish commands.

This is a temporary hack, until we properly model Ceph (etc) through the libguestfs stable API. Nevertheless it works as follows:

1. The add-drive /dev/null adds a drive, known to libguestfs.
2. Implicitly this means that libguestfs adds a -drive option when it runs qemu.
3. The custom qemu -set drive.hd0.file=... parameter modifies the preceding -drive option added by libguestfs so that the file is changed from /dev/null to whatever you want. In this case, to a Ceph rbd:... path.

Extracting filesystems from guest images, reconstructing guest images from filesystems, part 3

In part 1 I “exploded” a disk image into its constituent filesystems. In part 2 I “imploded” those filesystems back into a disk image without LVM. Now let’s see if we can get this thing to boot.

Since there is no boot sector nor grub, the disk image produced by virt-implode won’t boot normally. You could boot it using an external kernel and initrd, but let’s see if we can install grub first.

Using virt-rescue we can interactively run programs from the guest:

$ virt-rescue -a output.img
...
><rescue> mount /dev/sda2 /sysroot
><rescue> mount /dev/sda1 /sysroot/boot
><rescue> mount --bind /dev /sysroot/dev
><rescue> mount --bind /sys /sysroot/sys
><rescue> mount --bind /proc /sysroot/proc
><rescue> chroot /sysroot
sh: no job control in this shell
><rescue> cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
><rescue> vi /etc/fstab

You may need to fix /etc/fstab in the guest so that it points to the new partitions. For guests using LABELs or UUIDs, this won’t be necessary.

At this point, it should be simply a matter of running grub-install. But here’s where I remember how much I hate grub, because it just throws up peculiar, non-actionable error messages for every conceivable variation in the command.

So I give up that, and decide to extract the kernel and initrd and use the external boot method after all:

$ guestfish --ro -a output.img -i

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
/dev/sda2 mounted on /
/dev/sda1 mounted on /boot

><fs> ll /boot
total 4397
drwxr-xr-x.  5 root root    1024 Nov 15 13:49 .
drwxr-xr-x. 24 root root    4096 Nov 15 13:50 ..
-rw-r--r--.  1 root root  909034 Apr 20  2009 System.map-2.6.9-89.EL
drwxr-xr-x   3 root root    1024 Nov 15 13:49 boot
-rw-r--r--.  1 root root   45145 Apr 20  2009 config-2.6.9-89.EL
drwxr-xr-x.  2 root root    1024 Nov 15 13:58 grub
-rw-r--r--.  1 root root 1546843 Oct 27  2010 initrd-2.6.9-89.EL.img
drwx------   2 root root   12288 Oct 27  2010 lost+found
-rw-r--r--.  1 root root   23108 Aug  3  2005 message
-rw-r--r--.  1 root root   21282 Aug  3  2005 message.ja
-rw-r--r--.  1 root root   67352 Apr 20  2009 symvers-2.6.9-89.EL.gz
-rw-r--r--.  1 root root 1829516 Apr 20  2009 vmlinuz-2.6.9-89.EL
><fs> download /boot/vmlinuz-2.6.9-89.EL /tmp/vmlinuz-2.6.9-89.EL
><fs> download /boot/initrd-2.6.9-89.EL.img /tmp/initrd-2.6.9-89.EL.img

$ qemu-kvm -m 512 \
  -kernel vmlinuz-2.6.9-89.EL \
  -initrd initrd-2.6.9-89.EL.img \
  -append 'ro root=/dev/hda2' \
  -hda output.img

(I should really use libvirt, but this quick test proves that the guest works fine)

libguestfs on RHEL 5

I’ve spent the past couple of weeks [sic] getting a modern libguestfs into RHEL 5. RHEL 5 was first released in March 2007 and is thus over 5 years old.

It’s hard work.

Firstly RHEL 5 lacks virtio-serial, which is the mechanism used by the library to talk to the appliance. We used to use a TCP connection, but we gave that up since it’s slow, awkward and possibly insecure. To get RHEL 5 going again, I had to forward port the old TCP code into a modern branch of libguestfs.

Secondly RHEL 5 lacks a lot of features that libguestfs presents through the API. I got most of them working by carefully fixing and rewriting code, but some of them are impossible to do (eg. btrfs support).

Altogether I wrote about 30 patches, with about half going upstream, and the other half sitting in an “old Linux”-only side branch.

The release notes are attached below, and the packages should be available in EPEL 5 in a few days.

---

libguestfs for EPEL 5
by Richard W.M. Jones

This package contains modern libguestfs for EPEL 5. Although libguestfs is supplied in RHEL 5, the version supplied is very old and is only intended to be used with virt-v2v from RHEL 5. It is only available in the RHEL-5-V2V channel.

Therefore there exists a need for a modern libguestfs for RHEL 5 users. This package supplies this through the community-supported EPEL repository.

(1) Installation

After enabling the EPEL repository, install libguestfs in the normal way using yum. No post-installation configuration should be needed.

You must have a non-Xen kernel installed on the host. Note that the non-Xen kernel just needs to be installed, it does *not* need to be running (you can run libguestfs under Xen if you want). The non-Xen kernel is used to boot the libguestfs appliance.

You must use this package together with the version of qemu-kvm from RHEL 5 (usually called kvm-83-NNN.el5). Alternate versions of qemu, KVM, using $LIBGUESTFS_QEMU or QEMU wrappers, will probably not work.

Hardware virtualization will greatly improve the performance of libguestfs, but it is not required. libguestfs cannot use hardware virtualization when run in a Xen domain (not even dom0). For more performance tips, read guestfs-performance(1).

(2) Getting help

This package is supported voluntarily by the EPEL community and by the upstream libguestfs maintainers.

It is not supported by Red Hat.

It cannot be used with the virt-v2v package from RHEL 5. If you need to use virt-v2v on RHEL 5, use the libguestfs package from the same RHN channel. However we would advise you to use RHEL 6 or later because that version of virt-v2v is substantially more powerful.

To get help with this package, contact the libguestfs maintainers. See http://libguestfs.org for links to the mailing list, IRC, and how to file bugs.

(3) Features not available

The following features are not compatible with RHEL 5 and have been removed completely:

• PHP bindings
• GObject, GObject introspection, anything that uses GObject Introspection such as the Javascript bindings
• Erlang bindings
• The btrfs filesystem.
• guestfs_fstrim
• guestfs_wipefs
• Various APIs that take UUIDs (eg. guestfs_mkswap_U)
• virtio-scsi
• Internationalization of any non-C programs.

(4) Features that may not work

The following features may not work in this version of libguestfs:

• LUKS (encrypted filesystems within guests)
• MD (RAID devices within guests)
• FUSE, guestmount, the mount-local API. These appear to have some problems because of the older version of FUSE in RHEL 5.
• virt-sysprep --script option (because it requires FUSE).
• The ntfsclone_* APIs only partially work. In particular, the metadataonly flag is not working, and maybe other parts of this API won’t work.
• guestfs_vfs_label cannot read labels from NTFS filesystems.
• guestfs_blkid may return fewer fields than expected on a more modern Linux system.
• guestfs_txz_in and guestfs_txz_out don’t work.
• guestfs_utimens doesn’t let you set the time on a symbolic link (this is a limitation of the RHEL 5 kernel).
• guestfs_mkswap_label followed by guestfs_swapon_label does not work. The reasons are not well understood.
• Adding drives using non-virtio-blk interfaces.

(5) Other notes

An artificial pause of 1 second has been added after unmounting any filesystem.

*Without* the 1 second pause, the following test case will fail sometimes:

  guestfish <<EOF
  sparse test1.img 1G
  run
  part-disk /dev/sda mbr
  mkfs msdos /dev/sda1
  mount /dev/sda1 /
  touch /foo
  umount /
  mkfs ntfs /dev/sda1
  # Next command would fail:
  mount /dev/sda1 /
  vfs-type /dev/sda1 -x
  EOF

With the 1 second pause, this case passes. The reason is not understood.

Tip: Using mount-local API from C

Previously if you wanted to mount a disk image on the host from libguestfs, you had to use a separate program called guestmount.

A couple of months ago, we added FUSE support directly into the libguestfs API, and rewrote guestmount to use it. This also means you can use FUSE from your own libguestfs programs.

Yesterday I wrote a short example of using the mount-local API from C. This program creates a new disk image, formats it etc using libguestfs, then gives you a subshell so you can copy your own files, directories etc. in. When you quit the subshell, the disk image is synced and closed, and you end up with a virtual disk image containing all the files you just added. (Nothing that you couldn’t easily do before, but a nice little demonstration anyway).

Here it is in use:

$ gcc -Wall mount_local.c -o mount_local -lguestfs
$ ./mount_local /tmp/test.img

This is the 'mount-local' demonstration program.  Follow the
instructions on screen.

Creating and formatting the disk image, please wait a moment ...

The _current directory_ is a FUSE filesystem backed by the disk
image which is managed by libguestfs.  Any files or directories
you copy into here (up to 512 MB) will be saved into the disk
image.  You can also delete files, create certain special files
and so on.

When you have finished adding files, hit ^D or exit to exit the
shell and return to the mount-local program.

mount-local-shell> ls
lost+found  PUT_FILES_AND_DIRECTORIES_HERE

From the subshell, I copy in some files:

mount-local-shell> cp -a /usr/share/doc/libguestfs-devel-1.17.40/ .
mount-local-shell> ls
libguestfs-devel-1.17.40  lost+found  PUT_FILES_AND_DIRECTORIES_HERE
mount-local-shell> ls libguestfs-devel-1.17.40/
AUTHORS			       example-ubuntu.xml
BUGS			       example-windows-2003-x64-cd.xml
ChangeLog		       example-windows-2003-x86-cd.xml
copy_over.c		       example-windows.xml
create_disk.c		       example-windows-xp-cd.xml
display_icon.c		       HACKING
example-debian-netinst-cd.xml  inspect_vm.c
example-debian.xml	       README
example-fedora-dvd.xml	       RELEASE-NOTES
example-fedora-netinst-cd.xml  ROADMAP
example-fedora.xml	       TODO
example-rhel-6-dvd.xml	       virt-dhcp-address.c
example-rhel-6-netinst-cd.xml  virt-inspector.rng
example-ubuntu-live-cd.xml

After copying in my files, I exit from the subshell:

mount-local-shell> exit

Any files or directories that you copied in have been saved into
the disk image called '/tmp/test.img'.

Try opening the disk image with guestfish to see those files:

  guestfish -a /tmp/test.img -m /dev/sda1

Here is the disk image that was created (note it is sparse, so it’s not really so large as it appears):

$ ll /tmp/test.img
-rw-r--r--. 1 rjones rjones 536870912 May 14 12:03 /tmp/test.img
$ du -sh /tmp/test.img
18M	/tmp/test.img

We can use guestfish to look inside it:

$ guestfish -a /tmp/test.img -m /dev/sda1

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> ll /
total 18
drwxr-xr-x  4 root root  1024 May 14 12:03 .
drwxr-xr-x 23 1000 1000  4096 May 14 12:18 ..
-rw-r--r--  1 root root     0 May 14 12:03 PUT_FILES_AND_DIRECTORIES_HERE
drwxr-xr-x  2 root root  1024 May 14 08:37 libguestfs-devel-1.17.40
drwx------  2 root root 12288 May 14 12:03 lost+found

Download the test program here: https://github.com/libguestfs/libguestfs/blob/master/examples/mount_local.c

virt-format: Erase and make blank disks

There is a new tool in libguestfs ≥ 1.15.17: virt-format lets you erase disks and make blank disks.

Usage is quite simple:

$ virt-format -a disk.img

(Note that erases any data on disk.img!)

This works for any format of disk, eg. qcow2, or you can run it on host partitions, LVs, USB storage etc. By default it just creates a partition, but using other options you can make empty filesystems and logical volumes.

This is a simple tool that doesn’t cover everything you might want to do. For more complex requirements, see virt-make-fs or guestfish.

guestfish tutorial in Linux Format #152

It was out a couple of weeks ago. Previously …

Maximum qcow2 disk size

I don’t see this documented anywhere obvious, so I examined the source of qemu and performed some tests.

qemu stores the size in a 64 bit unsigned integer. However the qemu-img command line parsing code refuses to parse any number larger than 2⁶³-513 (9223372036854774784), and therefore that appears to be the largest disk size you can create:

$ qemu-img create -f qcow2 test1.img $((2**63-513))
Formatting 'test1.img', fmt=qcow2 size=9223372036854774784 encryption=off cluster_size=65536 
$ ll -h test1.img
-rw-r--r--. 1 rjones rjones 192K Oct  3 17:27 test1.img
$ guestfish -a test1.img run : blockdev-getsize64 /dev/sda
9223372036854774784

Interestingly things go horribly wrong as soon as you try to partition this:

$ virt-rescue test1.img
><rescue> parted /dev/vda print
Error: /dev/vda: unrecognised disk label                                  
Model: Virtio Block Device (virtblk)
Disk /dev/vda: 9223372TB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
><rescue> parted /dev/vda mklabel gpt
$ # bang! back to the prompt!

https://bugs.launchpad.net/qemu/+bug/865518

libguestfs tutorial to appear in Linux Format

Chris Brown has written a short tutorial on libguestfs and guestfish which will appear in Linux Format (a UK magazine) #152 in December/January.

libguestfs packages for Ubuntu 10.04LTS

Here you can find libguestfs packages for Ubuntu 10.04LTS:

http://libguestfs.org/download/binaries/ubuntu1004-packages/

Make sure you read the README first.

For a list of places to find binaries for other versions and distros, see here.

Playing with oz-install

oz-install (see also: Oz) is a command line program for building operating system instances. It can automatically install a fairly wide variety of OSes, including Windows.

Using it is pretty effortless. You first need to write a small template file, which just describes the name and version of the OS you want to install and where to get the ISO from. Here is one for Fedora 14:

<template>
  <name>fedora14_x86_64</name>
  <os>
    <name>Fedora</name>
    <version>14</version>
    <arch>x86_64</arch>
    <install type='iso'>
      <iso>file:///mnt/media/installers/Fedora-14-x86_64-DVD/Fedora-14-x86_64-DVD.iso</iso>
    </install>
  </os>
  <description>Fedora 14 x86_64 template</description>
</template>

By default oz-install will write the destination VM to /var/lib/libvirt/images, so you either need to check you’ve got enough free disk space under there, or else change the output path in /etc/oz/oz.cfg.

Using oz-install is simple, just do:

# oz-install fedora.xml
Libvirt XML was written to fedora14_x86_64Aug_30_2011-13:53:07

(oz-install requires root, but there is a feature request to remove this limitation)

oz-install writes a disk image and a libvirt XML file that you can use to immediately boot the guest:

# virsh define fedora14_x86_64Aug_30_2011-13:53:07
Domain fedora14_x86_64 defined from fedora14_x86_64Aug_30_2011-13:53:07

# virsh start fedora14_x86_64
Domain fedora14_x86_64 started

# virt-viewer fedora14_x86_64

I don’t know what root password is given to new Oz guests, but I was easily able to edit it out using guestfish.

Note that serious Oz users will want to further customize the guest using oz-customize, rather than booting it like I did. The workflow is that you use oz-install once to create the operating system template. Then you can duplicate the template, customize it (oz-customize) and deploy it. The copy-and-customize step is much quicker than installing a whole new guest from scratch each time.

Update: Chris tells me that the default root password is ozrootpw. You can also set it in the template description file by adding a <rootpw> element.