qemu 1.5.0 released, with ssh block device support

qemu 1.5.0 has been released, featuring ssh support so you can access remote disks over ssh, including from libguestfs.

Here’s how to use this from guestfish:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -a ssh://onuma/mnt/scratch/winxp.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 May 20 19:47 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
-rwxrwxrwx  1 root root      47564 Apr 14  2008 NTDETECT.COM
drwxrwxrwx  1 root root       4096 Oct 11  2011 Program Files
drwxrwxrwx  1 root root       4096 Oct 11  2011 System Volume Information
drwxrwxrwx  1 root root      28672 Oct 11  2011 WINDOWS
-rwxrwxrwx  1 root root        211 Oct 11  2011 boot.ini
-rwxrwxrwx  1 root root     250048 Apr 14  2008 ntldr
-rwxrwxrwx  1 root root 1610612736 Oct 11  2011 pagefile.sys

Fedora 19 virtualization test day 2013-05-28

Put it in your calendars .. May 28th is Fedora 19 virtualization test day.

New features include nested virtualization on Intel, new Boxes, new libosinfo, new qemu, KMS-based spice driver, live storage migration and virtio RNG.

Every day is libguestfs test day. Just follow the instructions here.

Scanning offline guests using OpenSCAP and guestmount

OpenSCAP is a project that lets you scan physical machines looking for known vulnerabilities or configuration problems (like public-writable directories).

Obviously it would be good to use this to scan guests, especially in a cloud scenario where you want to help naive users not to deploy guests that are just going to get pwned the minute they go online.

New upstream in OpenSCAP is the ability to scan chroots. You can use this to scan containers, or using guestmount, scan offline guests.

Usage with guestmount is described here or here.

(Thanks Daniel Kopecek and Peter Vrabec)

Testing exabyte-sized filesystems using qcow2 and guestfish

You can use qcow2 backing files as a convenient way to test what happens when you try to create exabyte-sized filesystems. Just to remind you, 1 exabyte is a million terabytes, or a pile of ordinary hard disks stacked 8 miles high.

There is a bug in qemu that prevents you from creating very large disks unless you adjust the cluster_size option (thanks Kevin Wolf):

$ qemu-img create -f qcow2 huge.qcow2 \
      $((1024*1024))T -o cluster_size=2M
Formatting 'huge.qcow2', fmt=qcow2 size=1152921504606846976 encryption=off cluster_size=2097152 lazy_refcounts=off 

After that you can just attach the disk to guestfish and start playing with huge filesystems.

[I should note that virt-rescue is probably a better choice of tool here, especially for people who need to experiment with unusual filesystem or LVM options]

$ guestfish -a huge.qcow2

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> run
><fs> blockdev-getsize64 /dev/sda
1152921504606846976
><fs> part-disk /dev/sda gpt

Ext4 (according to Wikipedia) is supposed to support 1 exabyte disks, but I couldn’t get that to work, possibly because there was not enough RAM:

><fs> mkfs ext4 /dev/sda1
libguestfs: error: mkfs: ext4: /dev/sda1: mke2fs 1.42.5 (29-Jul-2012)
/dev/sda1: Not enough space to build proposed filesystem while setting up superblock

XFS could create a filesystem, but I didn’t let it run to completion because it would need about 5 petabytes to store the filesystem metadata:

><fs> mkfs xfs /dev/sda1
[ disks churn for many minutes while qcow2 file grows
and grows and grows ... ]

LVM2 PVs are possible, but creating a VG requires us to adjust the extent size:

><fs> pvcreate /dev/sda1
><fs> vgcreate VG /dev/sda1
libguestfs: error: vgcreate:   PV /dev/sda1 too large for extent size 4.00 MiB.
  Format-specific setup of physical volume '/dev/sda1' failed.
  Unable to add physical volume '/dev/sda1' to volume group 'VG'.
><fs> debug sh "vgcreate -s 1G VG /dev/sda1"
  Volume group "VG" successfully created
><fs> lvcreate LV VG 1000000000
><fs> lvs-full
[0] = {
  lv_name: LV
[...]
  lv_size: 1048576536870912
}

Previously …

Using libguestfs over HTTP (and FTP)

New in libguestfs upstream and 1.21.39 is the ability to access disks over FTP, FTPS, HTTP, HTTPS and TFTP (read-only).

You can use it like this:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -a http://x.x.x.x/scratch/winxp.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 May 11 18:45 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
-rwxrwxrwx  1 root root      47564 Apr 14  2008 NTDETECT.COM
drwxrwxrwx  1 root root       4096 Oct 11  2011 Program Files
drwxrwxrwx  1 root root       4096 Oct 11  2011 System Volume Information
drwxrwxrwx  1 root root      28672 Oct 11  2011 WINDOWS
-rwxrwxrwx  1 root root        211 Oct 11  2011 boot.ini
-rwxrwxrwx  1 root root     250048 Apr 14  2008 ntldr
-rwxrwxrwx  1 root root 1610612736 Oct 11  2011 pagefile.sys

Apart from being a tiny bit slower, it just works as if the disk was local.

Using libguestfs to access iSCSI disks

In libguestfs ≥ 1.21.38 you can access at least some iSCSI disks.

On my server (RHEL 6 in this case) I create an iSCSI target backed by a Windows XP disk image:

# service tgtd start
Starting SCSI target daemon:                       [  OK  ]
# tgtadm --lld iscsi --op new --mode target --tid 1 \
      -T iqn.1994-05.com.redhat
# chcon system_u:object_r:tgtd_var_lib_t:s0 /tmp/winxp.img
# tgtadm --lld iscsi --op new --mode logicalunit --tid 1 \
      --lun 1 -b /tmp/winxp.img
# tgt-admin -s
...

Previously I opened port 3250 on the server. Because libguestfs doesn’t yet support authentication against the iSCSI server, I had to bypass that:

# tgtadm --lld iscsi --mode target --op bind --tid 1 -I ALL

Now on the client, I can connect to the iSCSI target using libguestfs like this:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --format=raw -a iscsi://x.x.x.x/iqn.1994-05.com.redhat/1 -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 May 11 17:16 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
-rwxrwxrwx  1 root root      47564 Apr 14  2008 NTDETECT.COM
drwxrwxrwx  1 root root       4096 Oct 11  2011 Program Files
drwxrwxrwx  1 root root       4096 Oct 11  2011 System Volume Information
drwxrwxrwx  1 root root      28672 Oct 11  2011 WINDOWS
-rwxrwxrwx  1 root root        211 Oct 11  2011 boot.ini
-rwxrwxrwx  1 root root     250048 Apr 14  2008 ntldr
-rwxrwxrwx  1 root root 1610612736 Oct 11  2011 pagefile.sys

Using libguestfs to read Hyper-V VHDX disk images

(Thanks to Jeff Cody for both writing the code and helping me to get it to work).

New in qemu 1.5 is read-only support for Hyper-V‘s native disk format, vhdx. You can now open vhdx files in libguestfs just like any other:

$ guestfish --ro --format=vhdx -a /tmp/f18x64.vhdx -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 18 (Spherical Cow)
/dev/fedora/root mounted on /
/dev/sda1 mounted on /boot

Notes:

1. As stated above, you will need the very latest qemu for this to work.
2. You must specify --format=vhdx at the moment because of a missing feature in libvirt.
3. You must specify --ro because writing is not supported (by the qemu driver).

Tip: Convert a Windows DVD ISO to a bootable USB key using guestfish

Situation: You have a Windows DVD (or ISO), but like any sane person in 2013 you don’t have a DVD drive on the computer. You want to convert the Windows DVD into a bootable USB key. There are many recipes for this online, but they all require another Windows machine and of course cannot be automated.

However with guestfish (and the always brilliant SYSLINUX doing most of the heavy lifting), this script will unpack the ISO and turn it into a bootable USB key.

Notes:

1. I am not going to support this script. You will need to read the script, look up the commands in the guestfish man page, and understand what it does. Any requests for help will be deleted unread.
2. You need to edit the USB key device before using the script.
3. You need libguestfs ≥ 1.21 with SYSLINUX support compiled in.

#!/bin/bash -

guestfish <<'EOF'
trace on

add-ro en_microsoft_hyper-v_server_2012_x64_dvd_915600.iso

# NB: The next line MUST be changed to your USB drive.
# ANYTHING ON THIS DRIVE WILL BE OVERWRITTEN WITHOUT WARNING.
add /dev/sdX

run

# Inside the appliance, /dev/sda = DVD, /dev/sdb = USB.
# THESE ARE NOT RELATED TO HOST DISK NAMES.

echo "Partitioning the USB disk ..."
part-init /dev/sdb mbr
part-add /dev/sdb p 63 -1
part-set-mbr-id /dev/sdb 1 0xb
part-set-bootable /dev/sdb 1 true
mkfs vfat /dev/sdb1

echo "Copying the contents of the DVD to the USB key ..."
mkmountpoint /cd
mkmountpoint /usb
mount /dev/sda /cd
mount /dev/sdb1 /usb
# XXX We should add cp-r command XXX
debug sh "cp -rP /sysroot/cd/* /sysroot/usb"
#glob cp-a /cd/* /usb
umount /cd
umount /usb
rmmountpoint /cd
rmmountpoint /usb

echo "Making the USB key bootable using SYSLINUX ..."
syslinux /dev/sdb1
mount /dev/sdb1 /
upload /usr/share/syslinux/chain.c32 /chain.c32
write /syslinux.cfg "DEFAULT windows\n\nLABEL windows\nCOM32 chain.c32\nAPPEND fs ntldr=/bootmgr\n"
umount /dev/sdb1
upload /usr/share/syslinux/mbr.bin /dev/sdb

echo "Finished."

EOF

Using libguestfs remotely with Python and rpyc

libguestfs has high quality Python bindings. Using rpyc you can make a remote libguestfs server with almost no effort at all.

Firstly start an rpyc server:

$ /usr/lib/python2.7/site-packages/rpyc/servers/classic_server.py
[SLAVE      INFO       13:21:17 tid=140019939981120] server started on 0.0.0.0:18812
[SLAVE      INFO       13:21:17 tid=140019784894208] started background auto-register thread (interval = 60)
[REGCLNT    INFO       13:21:17] registering on 255.255.255.255:18811
[REGCLNT    WARNING    13:21:19] no registry acknowledged

Now, possibly from the same machine or some other machine, you can connect to this server and use Python objects remotely as if they were local:

$ python
Python 2.7.3 (default, Aug  9 2012, 17:23:57) 
[GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpyc
>>> c = rpyc.classic.connect('localhost')

You can now create a libguestfs handle, following the example here.

>>> g = c.modules.guestfs.GuestFS()
>>> g.version()
{'release': 36L, 'major': 1L, 'minor': 21L, 'extra': 'fedora=20,release=1.fc20,libvirt'}
>>> g.add_drive('/dev/fedora/f18x64', readonly=True)
>>> g.launch()
>>> roots = g.inspect_os()
>>> g.inspect_get_product_name(roots[0])
'Fedora release 18 (Spherical Cow)'
>>> g.inspect_get_mountpoints(roots[0])
[('/', '/dev/mapper/fedora-root'), ('/boot', '/dev/sda1')]

As you can see, the g object is transparently remoted without you needing to do anything.

Brian Steven’s keynote at OpenStack Summit

Brian is Red Hat’s CTO, and hence my boss’s boss’s boss (or something like that). This is a pretty good (and honest) talk about Red Hat’s plans for OpenStack.

Edit: By the way, the thumbnail (the one I see at any rate) is not Brian.