Tip: FUSE-mount a disk image with Windows drive letters

guestmount is the libguestfs tool for taking a disk image and mounting it under the host filesystem. This works great for Linux disk images:

$ virt-builder centos-7.2
$ mkdir /tmp/mnt
$ guestmount -a centos-7.2.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ guestunmount /tmp/mnt

Those files under /tmp/mnt are inside the centos-7.2.img disk image file, and you can read and write them.

guestmount is fine for Windows disk images too, except when Windows has multiple drives, C:, D:, etc., because in that case you’ll only “see” the contents of the C: drive.

But guestmount is nowadays just a wrapper around the “mount-local” API in libguestfs, and you can use that API directly if you want to do anything a bit more complicated … such as exposing Windows drive letters.

Here is a Perl script which uses the mount-local API directly to do this:

#!/usr/bin/perl -w
use strict;
use Sys::Guestfs;
$| = 1;
die "usage: $0 mountpoint disk.img" if @ARGV < 2;
my $mp = shift @ARGV;
my $g = new Sys::Guestfs;
$g->add_drive_opts ($_) foreach @ARGV;
$g->launch;
my @roots = $g->inspect_os;
die "$0: no operating system found" if @roots != 1;
my $root = $roots[0];
die "$0: not Windows" if $g->inspect_get_type ($root) ne "windows";
my %map = $g->inspect_get_drive_mappings ($root);
foreach (keys %map) {
    $g->mkmountpoint ("/$_");
    eval { $g->mount ($map{$_}, "/$_") };
    warn "$@ (ignored)\n" if $@;
}
$g->mount_local ($mp);
print "filesystem mounted on $mp\n";
$g->mount_local_run;
$g->shutdown;

You can use it like this:

$ mkdir /tmp/mnt
$ ./drive-letters.pl /tmp/mnt windows7.img
filesystem ready on /tmp/mnt

in another window:

$ cd /tmp/mnt
$ ls
C  D
$ cd C
$ ls
Documents and Settings
PerfLogs
ProgramData
Program Files
[etc]
$ cd ../..
$ guestunmount /tmp/mnt

(Thanks to Pino Toscano for working out the details)

Tip: guestmount (FUSE mount) every filesystem in a disk image

Maxim asks an interesting question which is if you’ve got a disk image, how do you mount every filesystem onto your host. Like this:

$ ./fs-mount.pl rhel-5.11.img /tmp/fs &
$ cd /tmp/fs
/tmp/fs$ ls
dev
/tmp/fs$ cd dev
/tmp/fs/dev$ ls
sda1  sda2  sda3
/tmp/fs/dev$ cd sda2
/tmp/fs/dev/sda2$ ls
bin   dev  home  lib64       media  mnt  proc  sbin     srv  tmp  var
boot  etc  lib   lost+found  misc   opt  root  selinux  sys  usr
...
$ cd /tmp
$ guestunmount /tmp/fs

The answer is this surprisingly short Perl script.

#!/usr/bin/perl

use warnings;
use strict;

use Sys::Guestfs;

die "usage: $0 disk1 [disk2 ...] mountpoint\n" if @ARGV <= 1;

my $mp = pop;

my $g = Sys::Guestfs->new ();
foreach (@ARGV) {
    $g->add_drive ($_);
}
$g->launch ();

# Examine the filesystems.
my %fses = $g->list_filesystems ();

# Create the mountpoint directories (in the libguestfs namespace)
# and mount the filesystems on them.
foreach my $fs (sort keys %fses) {
    # mkmountpoint is really the same as mkdir.  Unfortunately there
    # is no 'mkdir -p' equivalent, so we have to do this instead:
    my @components = split ("/", $fs);
    for (my $i = 1; $i < @components; ++$i) {
        my $dir = "/" . join ("/", @components[1 .. $i]);
        eval { $g->mkmountpoint ($dir) }
    }

    # Don't fail if the filesystem can't be mounted, eg. it's swap.
    eval { $g->mount ($fs, $fs) }
}

# Export the filesystem on the host.
$g->mount_local ($mp);
$g->mount_local_run ();

# Close nicely since we mounted everything writable.
$g->shutdown ();
$g->close ();

Why has the libguestfs appliance grown by 281 MB?

a.k.a guestmount + filelight are awesome!

Click to see the full image

If you want to reproduce the same diagrams yourself, just do:

$ mkdir /tmp/mp
$ guestmount --ro \
    -a /var/tmp/.guestfs-1000/appliance.d/root \
    -m /dev/sda /tmp/mp
$ filelight /tmp/mp

Tip: Custom guestmount in Perl

Since libguestfs 1.18 guestmount has just been a slim wrapper around the libguestfs guestfs_mount_local API. You can replace guestmount with a small custom script if you want to do tricky/non-standard stuff like setting filesystem mount options, as in the example below.

$ ./mount-local.pl --ro -a /tmp/f17x64.img --mountopts=noatime /tmp/mount
mounting disk on /tmp/mount
to unmount: fusermount -u /tmp/mount

#!/usr/bin/perl -w

use strict;
use Getopt::Long;
use Sys::Guestfs;

my $readonly = 0;
my @drives;
my $mountopts;
my $trace = 0;

GetOptions ("ro|r" => \$readonly,
            "add|a=s" => \@drives,
            "mountopts=s" => \$mountopts,
            "trace|x" => \$trace)
    or die "$0 [--ro] [--add drive] [--mountopts mountopts] mountpoint\n";

die "$0: no drives (-a) were specified\n"
    unless @drives > 0;
die "$0: no mountpoint was specified\n"
    unless @ARGV == 1;

my $g = Sys::Guestfs->new ();
$g->set_trace (1) if $trace;
foreach (@drives) {
    $g->add_drive_opts ($_, readonly => $readonly)
}
$g->launch ();

# Inspect the disk to find OSes.
my @roots = $g->inspect_os ();
unless (@roots == 1) {
    die "$0: no operating systems found\n";
}
my $root = $roots[0];

# Mount up the disks like using the -i option.
my %mps = $g->inspect_get_mountpoints ($root);
my @mps = sort { length $a <=> length $b } (keys %mps);
foreach (@mps) {
    my $options = $readonly ? "ro" : "rw";
    $options .= "," . $mountopts if defined $mountopts;
    eval { $g->mount_options ($options, $mps{$_}, $_) };
    if ($@) {
        print "$@ (ignored)\n"
    }
}

# Export the filesystem using FUSE.
$g->mount_local ($ARGV[0]);

print "mounting disk on $ARGV[0]\n";
print "to unmount: fusermount -u $ARGV[0]\n";
$g->mount_local_run ();
# This returns when the filesystem is unmounted.

$g->shutdown ();
$g->close ();

Scanning offline guests using OpenSCAP and guestmount

OpenSCAP is a project that lets you scan physical machines looking for known vulnerabilities or configuration problems (like public-writable directories).

Obviously it would be good to use this to scan guests, especially in a cloud scenario where you want to help naive users not to deploy guests that are just going to get pwned the minute they go online.

New upstream in OpenSCAP is the ability to scan chroots. You can use this to scan containers, or using guestmount, scan offline guests.

Usage with guestmount is described here or here.

(Thanks Daniel Kopecek and Peter Vrabec)

Use guestfish, virt tools with remote disks

New in libguestfs ≥ 1.21.30 is the ability to use guestfish and some of the virt tools with remote disks.

Currently you can use remote disks over NBD, GlusterFS, Ceph, Sheepdog and (recently upstream) SSH.

For this example I’ll use SSH because it needs no setup, although this requires absolutely the latest qemu and libguestfs (both from git).

Since we don’t have libvirt support for ssh yet, so this only works with the direct backend:

$ export LIBGUESTFS_BACKEND=direct

I can use a ssh:// URI to add disks with guestfish, guestmount and most of the virt tools. For example:

$ virt-rescue -a ssh://localhost/tmp/f17x64.img
[... lots of boot messages ...]
Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> mount /dev/vg_f17x64/lv_root /sysroot
><rescue> cat /sysroot/etc/redhat-release
Fedora release 17 (Beefy Miracle)

Apart from being a tiny bit slower, it just works as if the disk was local:

$ virt-df -a ssh://localhost/tmp/f17x64.img
Filesystem                           1K-blocks       Used  Available  Use%
f17x64.img:/dev/sda1                    487652      63738     398314   14%
f17x64.img:/dev/vg_f17x64/lv_root     28316680    4285576   22586036   16%

$ guestmount -a ssh://localhost/tmp/f17x64.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ cat /tmp/mnt/etc/redhat-release
Fedora release 17 (Beefy Miracle)
$ guestunmount /tmp/mnt

Cool new bash-completions of libguestfs tools

Starting in libguestfs ≥ 1.21.23-2, bash tab completions of guestfish, guestmount and virt-* tools have been rewritten and greatly improved.

Note you will need to install the libguestfs-bash-completion package to enable this feature.

You can now tab complete all long options on most tools:

$ virt-df --[tab]
--add             --domain          --human-readable  --uuid
--connect         --format          --inodes          --verbose
--csv             --help            --one-per-guest   --version
$ virt-resize --[tab]
--align-first          --help                 --no-extra-partition
--alignment            --ignore               --ntfsresize-force
--debug                --lvexpand             --output-format
--debug-gc             --lv-expand            --quiet
--delete               --LVexpand             --resize
--dryrun               --LV-expand            --resize-force
--dry-run              --machine-readable     --shrink
--expand               --no-copy-boot-loader  --version
--format               --no-expand-content    

Where appropriate, the -d option will now expand to the list of libvirt domains:

# virt-df -d [tab]
archlinux20121201x64  f19rawhidex32
f18x64                f19rawhidex64

Finally, guestfish commands are expanded on the command line:

$ guestfish add /tmp/disk : run : list-[tab]
list-9p              list-events          list-md-devices
list-devices         list-filesystems     list-partitions
list-disk-labels     list-ldm-partitions  
list-dm-devices      list-ldm-volumes     

To make this less intrusive, so you can really use it daily, I left the default readline expansions enabled. This means that filenames and so on can continue to be used in every position on the command line, and should mean that bash completions won’t try to be cleverer than the user.

Libguestfs bash completions are also demand-loaded now, so that if you’re not using them, they don’t consume any resources in the shell.

libguestfs on RHEL 5

I’ve spent the past couple of weeks [sic] getting a modern libguestfs into RHEL 5. RHEL 5 was first released in March 2007 and is thus over 5 years old.

It’s hard work.

Firstly RHEL 5 lacks virtio-serial, which is the mechanism used by the library to talk to the appliance. We used to use a TCP connection, but we gave that up since it’s slow, awkward and possibly insecure. To get RHEL 5 going again, I had to forward port the old TCP code into a modern branch of libguestfs.

Secondly RHEL 5 lacks a lot of features that libguestfs presents through the API. I got most of them working by carefully fixing and rewriting code, but some of them are impossible to do (eg. btrfs support).

Altogether I wrote about 30 patches, with about half going upstream, and the other half sitting in an “old Linux”-only side branch.

The release notes are attached below, and the packages should be available in EPEL 5 in a few days.

---

libguestfs for EPEL 5
by Richard W.M. Jones

This package contains modern libguestfs for EPEL 5. Although libguestfs is supplied in RHEL 5, the version supplied is very old and is only intended to be used with virt-v2v from RHEL 5. It is only available in the RHEL-5-V2V channel.

Therefore there exists a need for a modern libguestfs for RHEL 5 users. This package supplies this through the community-supported EPEL repository.

(1) Installation

After enabling the EPEL repository, install libguestfs in the normal way using yum. No post-installation configuration should be needed.

You must have a non-Xen kernel installed on the host. Note that the non-Xen kernel just needs to be installed, it does *not* need to be running (you can run libguestfs under Xen if you want). The non-Xen kernel is used to boot the libguestfs appliance.

You must use this package together with the version of qemu-kvm from RHEL 5 (usually called kvm-83-NNN.el5). Alternate versions of qemu, KVM, using $LIBGUESTFS_QEMU or QEMU wrappers, will probably not work.

Hardware virtualization will greatly improve the performance of libguestfs, but it is not required. libguestfs cannot use hardware virtualization when run in a Xen domain (not even dom0). For more performance tips, read guestfs-performance(1).

(2) Getting help

This package is supported voluntarily by the EPEL community and by the upstream libguestfs maintainers.

It is not supported by Red Hat.

It cannot be used with the virt-v2v package from RHEL 5. If you need to use virt-v2v on RHEL 5, use the libguestfs package from the same RHN channel. However we would advise you to use RHEL 6 or later because that version of virt-v2v is substantially more powerful.

To get help with this package, contact the libguestfs maintainers. See http://libguestfs.org for links to the mailing list, IRC, and how to file bugs.

(3) Features not available

The following features are not compatible with RHEL 5 and have been removed completely:

• PHP bindings
• GObject, GObject introspection, anything that uses GObject Introspection such as the Javascript bindings
• Erlang bindings
• The btrfs filesystem.
• guestfs_fstrim
• guestfs_wipefs
• Various APIs that take UUIDs (eg. guestfs_mkswap_U)
• virtio-scsi
• Internationalization of any non-C programs.

(4) Features that may not work

The following features may not work in this version of libguestfs:

• LUKS (encrypted filesystems within guests)
• MD (RAID devices within guests)
• FUSE, guestmount, the mount-local API. These appear to have some problems because of the older version of FUSE in RHEL 5.
• virt-sysprep --script option (because it requires FUSE).
• The ntfsclone_* APIs only partially work. In particular, the metadataonly flag is not working, and maybe other parts of this API won’t work.
• guestfs_vfs_label cannot read labels from NTFS filesystems.
• guestfs_blkid may return fewer fields than expected on a more modern Linux system.
• guestfs_txz_in and guestfs_txz_out don’t work.
• guestfs_utimens doesn’t let you set the time on a symbolic link (this is a limitation of the RHEL 5 kernel).
• guestfs_mkswap_label followed by guestfs_swapon_label does not work. The reasons are not well understood.
• Adding drives using non-virtio-blk interfaces.

(5) Other notes

An artificial pause of 1 second has been added after unmounting any filesystem.

*Without* the 1 second pause, the following test case will fail sometimes:

  guestfish <<EOF
  sparse test1.img 1G
  run
  part-disk /dev/sda mbr
  mkfs msdos /dev/sda1
  mount /dev/sda1 /
  touch /foo
  umount /
  mkfs ntfs /dev/sda1
  # Next command would fail:
  mount /dev/sda1 /
  vfs-type /dev/sda1 -x
  EOF

With the 1 second pause, this case passes. The reason is not understood.

Tip: Using mount-local API from C

Previously if you wanted to mount a disk image on the host from libguestfs, you had to use a separate program called guestmount.

A couple of months ago, we added FUSE support directly into the libguestfs API, and rewrote guestmount to use it. This also means you can use FUSE from your own libguestfs programs.

Yesterday I wrote a short example of using the mount-local API from C. This program creates a new disk image, formats it etc using libguestfs, then gives you a subshell so you can copy your own files, directories etc. in. When you quit the subshell, the disk image is synced and closed, and you end up with a virtual disk image containing all the files you just added. (Nothing that you couldn’t easily do before, but a nice little demonstration anyway).

Here it is in use:

$ gcc -Wall mount_local.c -o mount_local -lguestfs
$ ./mount_local /tmp/test.img

This is the 'mount-local' demonstration program.  Follow the
instructions on screen.

Creating and formatting the disk image, please wait a moment ...

The _current directory_ is a FUSE filesystem backed by the disk
image which is managed by libguestfs.  Any files or directories
you copy into here (up to 512 MB) will be saved into the disk
image.  You can also delete files, create certain special files
and so on.

When you have finished adding files, hit ^D or exit to exit the
shell and return to the mount-local program.

mount-local-shell> ls
lost+found  PUT_FILES_AND_DIRECTORIES_HERE

From the subshell, I copy in some files:

mount-local-shell> cp -a /usr/share/doc/libguestfs-devel-1.17.40/ .
mount-local-shell> ls
libguestfs-devel-1.17.40  lost+found  PUT_FILES_AND_DIRECTORIES_HERE
mount-local-shell> ls libguestfs-devel-1.17.40/
AUTHORS			       example-ubuntu.xml
BUGS			       example-windows-2003-x64-cd.xml
ChangeLog		       example-windows-2003-x86-cd.xml
copy_over.c		       example-windows.xml
create_disk.c		       example-windows-xp-cd.xml
display_icon.c		       HACKING
example-debian-netinst-cd.xml  inspect_vm.c
example-debian.xml	       README
example-fedora-dvd.xml	       RELEASE-NOTES
example-fedora-netinst-cd.xml  ROADMAP
example-fedora.xml	       TODO
example-rhel-6-dvd.xml	       virt-dhcp-address.c
example-rhel-6-netinst-cd.xml  virt-inspector.rng
example-ubuntu-live-cd.xml

After copying in my files, I exit from the subshell:

mount-local-shell> exit

Any files or directories that you copied in have been saved into
the disk image called '/tmp/test.img'.

Try opening the disk image with guestfish to see those files:

  guestfish -a /tmp/test.img -m /dev/sda1

Here is the disk image that was created (note it is sparse, so it’s not really so large as it appears):

$ ll /tmp/test.img
-rw-r--r--. 1 rjones rjones 536870912 May 14 12:03 /tmp/test.img
$ du -sh /tmp/test.img
18M	/tmp/test.img

We can use guestfish to look inside it:

$ guestfish -a /tmp/test.img -m /dev/sda1

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> ll /
total 18
drwxr-xr-x  4 root root  1024 May 14 12:03 .
drwxr-xr-x 23 1000 1000  4096 May 14 12:18 ..
-rw-r--r--  1 root root     0 May 14 12:03 PUT_FILES_AND_DIRECTORIES_HERE
drwxr-xr-x  2 root root  1024 May 14 08:37 libguestfs-devel-1.17.40
drwx------  2 root root 12288 May 14 12:03 lost+found

Download the test program here: https://github.com/libguestfs/libguestfs/blob/master/examples/mount_local.c

libguestfs: mount local

New in libguestfs ≥ 1.17.22 is the ability to mount the guest filesystem on a local mountpoint. Well, you can already do that using guestmount, but the new thing is that you can do it from the API (from any language).

Here is an example using guestfish:

$ guestfish --ro -a /dev/vg_pin/F16x64 -i
 
Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 16 (Verne)
/dev/vg_f16x64/lv_root mounted on /
/dev/vda2 mounted on /boot

><fs> ! mkdir /tmp/mnt # creates a local directory
><fs> mount-local /tmp/mnt readonly:true
><fs> mount-local-run
# the errors come from a GNOME daemon that
# looks at all new mountpoints
libguestfs: error: lstat: /.Trash: No such file or directory
libguestfs: error: lstat: /.Trash-500: No such file or directory

Over in another terminal, we can see the filesystem mounted on the local directory /tmp/mnt:

$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
$ cat /tmp/mnt/etc/redhat-release
Fedora release 16 (Verne)

Unmounting the filesystem causes the guestfish mount-local-run command to return (since that command was actually serving the FUSE requests from the kernel).

$ fusermount -u /tmp/mnt