Tag Archives: libvirt

Extracting filesystems from guest images, reconstructing guest images from filesystems, part 3

In part 1 I “exploded” a disk image into its constituent filesystems. In part 2 I “imploded” those filesystems back into a disk image without LVM. Now let’s see if we can get this thing to boot.

Since there is no boot sector nor grub, the disk image produced by virt-implode won’t boot normally. You could boot it using an external kernel and initrd, but let’s see if we can install grub first.

Using virt-rescue we can interactively run programs from the guest:

$ virt-rescue -a output.img
...
><rescue> mount /dev/sda2 /sysroot
><rescue> mount /dev/sda1 /sysroot/boot
><rescue> mount --bind /dev /sysroot/dev
><rescue> mount --bind /sys /sysroot/sys
><rescue> mount --bind /proc /sysroot/proc
><rescue> chroot /sysroot
sh: no job control in this shell
><rescue> cat /etc/redhat-release
Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
><rescue> vi /etc/fstab

You may need to fix /etc/fstab in the guest so that it points to the new partitions. For guests using LABELs or UUIDs, this won’t be necessary.

At this point, it should be simply a matter of running grub-install. But here’s where I remember how much I hate grub, because it just throws up peculiar, non-actionable error messages for every conceivable variation in the command.

So I give up that, and decide to extract the kernel and initrd and use the external boot method after all:

$ guestfish --ro -a output.img -i

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux AS release 4 (Nahant Update 8)
/dev/sda2 mounted on /
/dev/sda1 mounted on /boot

><fs> ll /boot
total 4397
drwxr-xr-x.  5 root root    1024 Nov 15 13:49 .
drwxr-xr-x. 24 root root    4096 Nov 15 13:50 ..
-rw-r--r--.  1 root root  909034 Apr 20  2009 System.map-2.6.9-89.EL
drwxr-xr-x   3 root root    1024 Nov 15 13:49 boot
-rw-r--r--.  1 root root   45145 Apr 20  2009 config-2.6.9-89.EL
drwxr-xr-x.  2 root root    1024 Nov 15 13:58 grub
-rw-r--r--.  1 root root 1546843 Oct 27  2010 initrd-2.6.9-89.EL.img
drwx------   2 root root   12288 Oct 27  2010 lost+found
-rw-r--r--.  1 root root   23108 Aug  3  2005 message
-rw-r--r--.  1 root root   21282 Aug  3  2005 message.ja
-rw-r--r--.  1 root root   67352 Apr 20  2009 symvers-2.6.9-89.EL.gz
-rw-r--r--.  1 root root 1829516 Apr 20  2009 vmlinuz-2.6.9-89.EL
><fs> download /boot/vmlinuz-2.6.9-89.EL /tmp/vmlinuz-2.6.9-89.EL
><fs> download /boot/initrd-2.6.9-89.EL.img /tmp/initrd-2.6.9-89.EL.img
$ qemu-kvm -m 512 \
  -kernel vmlinuz-2.6.9-89.EL \
  -initrd initrd-2.6.9-89.EL.img \
  -append 'ro root=/dev/hda2' \
  -hda output.img

(I should really use libvirt, but this quick test proves that the guest works fine)

1 Comment

Filed under Uncategorized

Thursday is Fedora virt test day

Thursday (1st Nov) is Fedora virtualization test day. Help us out by testing libguestfs!

Fedora 18 has definitely been a struggle. It is possibly the most delayed Fedora release ever. In libguestfs (in Fedora only) we switched to using libvirt to launch the appliance, revealing a lot of bugs and problems in libvirt in the process.

At the same time we’ve added dozens of major new features to libguestfs.

So there’s likely to be a lot of bugs, and you can make a difference.

Leave a comment

Filed under Uncategorized

Hotplugging in libguestfs

I just posted a new set of patches upstream which add hotplugging support for libguestfs.

Since the project started, most libguestfs programs have looked like this:

/* create the handle */
g = guestfs_create ();
/* add 1 or more disks to examine */
guestfs_add_drive (g, "disk.img");
/* launch the handle */
guestfs_launch (g);

It was not possible to add further disks after launch, because of the way libguestfs works: You could not add disks to the small appliance that we use after the appliance has started up.

Except that you can: Linux and qemu have supported hotplugging disks for a long time, but we didn’t expose this through libguestfs.

Now that you can use libvirt to run the appliance it becomes relatively easy for us to add hotplugging, implemented via libvirt’s virDomainAttachDevice API. (The actual implementation inside libvirt is very complex, hence the reason why we didn’t try to reimplement it in libguestfs). From the point of view of libguestfs callers, you can now call the add-drive* APIs after launch.

All good news? Apart from the obvious limitation that you have to be using the libvirt backend for it to work at all, is it a good idea to use hotplugging?

From a security point of view, there is a trade-off: If you have to modify lots of guests, it’s faster to use hotplugging because you save having to start a new appliance for each guest. But it’s also less secure (significantly so) because one guest may be able to exploit and hijack the appliance and interfere with the disks of other appliances.

In virt-df we accept this risk. It is largely mitigated because the disks are read-only so an exploit from one guest cannot make a permanent change to any other guest.

But if you are considering using hotplugging to modify lots of mutually untrustworthy disks, don’t do it. Either use one appliance per group of mutually trusting guests, or (easier) just launch an appliance per guest. (Reading this page may help you to get the best performance in this case).

Leave a comment

Filed under Uncategorized

New in libguestfs: Use libvirt to launch the appliance

libvirt logoFrom libguestfs ≥ 1.19.24 you can, if you want, use libvirt to manage the libguestfs appliance. To enable this, set the environment variable:

export LIBGUESTFS_ATTACH_METHOD=libvirt

and (hopefully) any libguestfs program, guestfish, virt tool etc will just work as normal. As well as the absolute latest libguestfs you’ll also need libvirt ≥ 0.9.13 and a recent qemu.

What’s the point of this change? There are a couple of advantages. Firstly libvirt confines guests using sVirt mandatory access control (using SELinux or AppArmor). This will [it's not working right now] provide additional defences against rogue disk images subverting the appliance or qemu.

Secondly libvirt already does disk hotplugging, and it will allow us to add this easily to libguestfs.

2 Comments

Filed under Uncategorized

Using libguestfs live without libvirt

You should use libguestfs live with libvirt because it takes the pain out of it.

However if for some reason you don’t want to use libvirt, here is how to use it directly. Note normally you should use libvirt and you do not need to do any of this!

(1) Your guest needs to have guestfsd -r installed and running. [Normally you just install libguestfs-live-service which does everything necessary]

(2) You need to start up qemu (the guest) with these special options, or the equivalent. [Normally you just set this up through libvirt]

-chardev pty,id=charserial0
-device isa-serial,chardev=charserial0,id=serial0
-chardev socket,id=charchannel0,path=/tmp/socket,server,nowait
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.libguestfs.channel.0

(3) Start guestfish and use the attach-method command to point to the socket. [Normally using '-d Guest --live' does all this automatically for you by getting the necessary information out of libvirt]

$ guestfish
><fs> attach-method unix:/tmp/socket
><fs> run
><fs> ll /
[use libguestfs command as normal here]

Leave a comment

Filed under Uncategorized

Tip: Detecting guest activity: three methods

(1) Is the guest generating disk and network activity (a.k.a are the lights flashing)?

I don’t have nice code for this, but you can see the technique that Oz uses here. Oz uses the libvirt monitoring APIs to look for disk and network activity, and signals when it hasn’t seen any after a certain timeout period.

(2) When was the last time a user logged in?

If this is the sort of “activity” you’re after, then you can use virt-cat on Linux, or virt-win-reg [sorry, no actual example yet] on Windows.

(3) What was the last file updated in a guest?

Finally, if you’re interested in the newest file updated in a guest, see this technique which will work for any Linux or Windows guest.

Are there other kinds of “activity” that it’s interesting to find from guests?

Leave a comment

Filed under Uncategorized

Half-baked idea: Autodetect dependencies

For more half-baked ideas, see the ideas tag.

Large “coordination” libraries like libvirt unify a lot of disparate features through one API, and as a result they depend on many other libraries and external programs. Libvirt directly links to 20 libraries and requires countless other external programs.

We want users to be able to compile libvirt even when they don’t have the full set of libraries (you don’t need, say, PolKit, libvirt will still work with reduced functionality). The issue though is you end up with code which looks like:

switch (cred[i].type) {
case VIR_CRED_EXTERNAL: {
    if (STRNEQ(cred[i].challenge, "PolicyKit"))
        return -1;

#if defined(POLKIT_AUTH)
    if (virConnectAuthGainPolkit(cred[i].prompt) < 0)
        return -1;
#else
    /* Ignore & carry on. Although we can't auth
     * directly, the user may have authenticated
     * themselves already outside context of libvirt
     */
#endif
    break;
}

This code is fragile because (a) it’s hard to reason about all the pathways and (b) it’s combinatorially difficult to test all the different permutations of available libraries. This fragility leads to bug reports and possibly worse.

Before I get to the half-baked idea, I’ll throw in another thought: at the moment we do most of this detection at compile time using a long configure script. It might be better to do it at run time. You could imagine how this could work if you were a very patient programmer who liked writing tedious boilerplate:

libaudit = dlopen ("libaudit.so", 0);
//...
if (libaudit) {
  int (*audit_add_watch) (struct audit_rule_data **rulep,
                          const char *path);
  audit_add_watch = dlsym (libaudit, "audit_add_watch");
  if (audit_add_watch)
    r = audit_add_watch (rule, path);
  else
    goto no_func;
} else {
 no_func:
  // no libaudit, do something else
}

The half-baked idea is this: Write the code as if all the functions exist. Then transform the code into the runtime/dlsym version above. In the first iteration, for each libvirt API entry point we compute the sum of all optional libraries/functions that are required to execute that entry point, and we generate checks like this:

virFoo ()
{
  // The following checks are generated automatically:
  if (!libaudit)
    return error ("virFoo: you need to install libaudit");
  if (!libaudit_audit_add_watch)
    return error ("virFoo: wrong version of libaudit, "
                  "requires audit_add_watch function");
  //..
  // Here we run the programmer's code:
  //..
}

The first iteration is very conservative. In the second iteration of the project we’d allow the programmer to write fallback code, so that partial API functionality is available even if not all the libraries are. But how to do that and avoid the #ifdef problem?

I think you should be allowed to write alternate functions:

authenticate ()
{
  return polkit_context_is_caller_authorized (pkcontext, ...);
}

authenticate ()
{
  return 1;
}

(Remember this is not C, but some sort of C with transformations applied to it).

Our C transformation chooses the “best” function to call at runtime, where best is simply the one which has the most libraries available. In the above case, the first version of authenticate is chosen if the PolKit library is available, the second version if not.

5 Comments

Filed under Uncategorized

Setting up a serial console in qemu and libvirt

I always forget how to do this, but in fact it’s quite simple.

First ensure your libvirt XML contains a fragment like this (my guest, installed using virt-install, already had this).

<serial type='pty'>
  <target port='0'/>
</serial>
<console type='pty'>
  <target type='serial' port='0'/>
</console>

Second, edit /boot/grub/grub.conf inside the guest, adding the console=ttyS0 element to the kernel command line:

# virt-edit Guestname /boot/grub/grub.conf
...
title Fedora (2.6.38.6-26.rc1.fc15.x86_64)
	root (hd0,0)
	kernel /vmlinuz ro [...] console=ttyS0
...

You don’t need to set the speed. I believe it defaults to 115200 8N1, but I don’t think that qemu serial ports have a “speed” as such, since the hardware is emulated.

Third, start the guest and dump out the running XML:

# virsh start Guestname && virsh dumpxml Guestname
...
    <console type='pty' tty='/dev/pts/8'>

Notice the randomly assigned pty on the host side (/dev/pts/8). Connect to that with Minicom[1], and you should see boot messages and/or a login prompt.

[1] Is there something better than minicom? It’s a horrible program, always has been.

16 Comments

Filed under Uncategorized

New tool: virt-click, send simulated mouse clicks to a VNC server

This new virt tool called “virt-click” lets you send simulated mouse clicks to a VNC server.

$ virt-click --vnc localhost:0 click 200 200 right

The motivation for this tool comes from this posting.

I tested it against QEMU’s internal VNC server, so you can use it on virtual machines already. It probably works on any VNC server, but I have not tested it.

Eventually the plan is to extend this to SPICE, and also allow it to connect transparently to libvirt guests, but that work is not done yet. Also it would be good to add other features, eg:

  • key presses
  • screendumps (like gvnccapture and snappy)
  • better handling of VNC authentication

2 Comments

Filed under Uncategorized

Fun new virt tools: virt-dmesg and virt-uname

About 4 years ago I wrote some code that peeks into the memory of guests, reads out the kernel, and finds out useful stuff: symbols, process tables, kernel messages, network interfaces and so on. Actually I should say at this point you can already do this using the excellent crash tool, and if you have the debuginfo for all your guest kernels, you should use that tool. My code was trying to be more ambitious, because it heuristically examined the kernel memory, guessing the location of various structures.

Well, that code was not so successful. Heuristics only take you so far when you’re looking at something as complex and variable as the process table. One day I hope to make a big breakthrough to make it all possible.

Nevertheless, some things worked quite well: it was actually pretty easy to snoop out the kernel symbols, and once you’ve got those some of the more static kernel structures are easily accessible.

For the past few days I’ve been working on two tools derived from that.

virt-dmesg shows you the kernel messages from a running Linux virtual machine:

# virt-dmesg F14x64 | tail -5
<6>[   11.609206] lo: Disabled Privacy Extensions
<6>[   11.634902] ip6_tables: (C) 2000-2006 Netfilter Core Team
<6>[   15.286998] eth0: link up, 100Mbps, full-duplex, lpa 0x05E1
<7>[   21.898345] mtrr: no MTRR for f0000000,100000 found
<7>[   25.714129] eth0: no IPv6 routers present

virt-uname shows the utsname structure:

# virt-uname Fedora14
Linux f14x64.home.annexia.org 2.6.35.11-83.fc14.x86_64 #1 SMP
Mon Feb 7 07:06:44 UTC 2011 x86_64 (none)

The main page for virt-dmesg and virt-uname is here, and there is also a git repository.

You will need a bunch of patches against libvirt to make it work. I have updated libvirt in Rawhide with all the required patches, and without that version of libvirt it definitely won’t work at all.

It works for many, but not all, guests that I’ve tried. The main problem is with guests that don’t enable the full kallsyms feature. If the guest doesn’t have all of the kernel configuration options below enabled, then you may have to fall back to dumping out the kernel and grepping through it with “strings”.

CONFIG_KALLSYMS=y
CONFIG_KALLSYMS_ALL=y
CONFIG_KALLSYMS_EXTRA_PASS=y

Leave a comment

Filed under Uncategorized