FUSE mounting on top of a file

Our tool nbdfuse lets you mount an NBD block device as a file, using Linux FUSE. For example you could create a directory with a single file in it (called nbd) which contains the contents of the NBD export:

$ mkdir /var/tmp/test
$ nbdfuse /var/tmp/test --command nbdkit -s memory 1G &
$ ls -l /var/tmp/test/
total 0
 -rw-rw-rw-. 1 rjones rjones 1073741824 Nov  4 13:25 nbd
$ fusermount -u /var/tmp/test

This is cool, but wouldn’t it be nice to get rid of the directory and create the file anywhere? Recently Max Reitz found out you can mount a FUSE filesystem over a regular file.

It works! (After a few adjustments to the nbdfuse code)

$ touch /var/tmp/disk.img
$ nbdfuse /var/tmp/disk.img --command nbdkit -s memory 1G &
$ ls -l /var/tmp/disk.img
 -rw-rw-rw-. 1 rjones rjones 1073741824 Nov  4 13:29 /var/tmp/disk.img
$ fusermount -u /var/tmp/disk.img 

libnbd + FUSE = nbdfuse

I’ve talked before about libnbd, our NBD client library. New in libnbd 1.2 is a tool called nbdfuse which lets you turn NBD servers into virtual files.

A couple of weeks ago I mentioned you can use libnbd as a C library to edit qcow2 files. Now you can turn qcow2 files into virtual raw files:

$ mkdir dir
$ nbdfuse dir/file.raw \
      --socket-activation qemu-nbd -f qcow2 file.qcow2
$ ls -l dir/
total 0
-rw-rw-rw-. 1 nbd nbd 1073741824 Jan  1 10:10 file.raw

Reads and writes to file.raw are backed by the original qcow2 file which is updated in real time.

Another fun thing to do is to use nbdkit, xz filter and curl to turn xz-compressed remote disk images into uncompressed local files:

$ mkdir dir
$ nbdfuse dir/disk.img \
      --command nbdkit -s curl --filter=xz \
                       http://builder.libguestfs.org/fedora-30.xz
$ ls -l dir/
total 0
-rw-rw-rw-. 1 nbd nbd 6442450944 Jan  1 10:10 disk.img
$ file dir/disk.img
dir/disk.img: DOS/MBR boot sector
$ qemu-system-x86_64 -m 4G \
      -drive file=dir/disk.img,format=raw,if=virtio,snapshot=on

Tip: FUSE-mount a disk image with Windows drive letters

guestmount is the libguestfs tool for taking a disk image and mounting it under the host filesystem. This works great for Linux disk images:

$ virt-builder centos-7.2
$ mkdir /tmp/mnt
$ guestmount -a centos-7.2.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ guestunmount /tmp/mnt

Those files under /tmp/mnt are inside the centos-7.2.img disk image file, and you can read and write them.

guestmount is fine for Windows disk images too, except when Windows has multiple drives, C:, D:, etc., because in that case you’ll only “see” the contents of the C: drive.

But guestmount is nowadays just a wrapper around the “mount-local” API in libguestfs, and you can use that API directly if you want to do anything a bit more complicated … such as exposing Windows drive letters.

Here is a Perl script which uses the mount-local API directly to do this:

#!/usr/bin/perl -w
use strict;
use Sys::Guestfs;
$| = 1;
die "usage: $0 mountpoint disk.img" if @ARGV < 2;
my $mp = shift @ARGV;
my $g = new Sys::Guestfs;
$g->add_drive_opts ($_) foreach @ARGV;
$g->launch;
my @roots = $g->inspect_os;
die "$0: no operating system found" if @roots != 1;
my $root = $roots[0];
die "$0: not Windows" if $g->inspect_get_type ($root) ne "windows";
my %map = $g->inspect_get_drive_mappings ($root);
foreach (keys %map) {
    $g->mkmountpoint ("/$_");
    eval { $g->mount ($map{$_}, "/$_") };
    warn "$@ (ignored)\n" if $@;
}
$g->mount_local ($mp);
print "filesystem mounted on $mp\n";
$g->mount_local_run;
$g->shutdown;

You can use it like this:

$ mkdir /tmp/mnt
$ ./drive-letters.pl /tmp/mnt windows7.img
filesystem ready on /tmp/mnt

in another window:

$ cd /tmp/mnt
$ ls
C  D
$ cd C
$ ls
Documents and Settings
PerfLogs
ProgramData
Program Files
[etc]
$ cd ../..
$ guestunmount /tmp/mnt

(Thanks to Pino Toscano for working out the details)

Tip: guestmount (FUSE mount) every filesystem in a disk image

Maxim asks an interesting question which is if you’ve got a disk image, how do you mount every filesystem onto your host. Like this:

$ ./fs-mount.pl rhel-5.11.img /tmp/fs &
$ cd /tmp/fs
/tmp/fs$ ls
dev
/tmp/fs$ cd dev
/tmp/fs/dev$ ls
sda1  sda2  sda3
/tmp/fs/dev$ cd sda2
/tmp/fs/dev/sda2$ ls
bin   dev  home  lib64       media  mnt  proc  sbin     srv  tmp  var
boot  etc  lib   lost+found  misc   opt  root  selinux  sys  usr
...
$ cd /tmp
$ guestunmount /tmp/fs

The answer is this surprisingly short Perl script.

#!/usr/bin/perl

use warnings;
use strict;

use Sys::Guestfs;

die "usage: $0 disk1 [disk2 ...] mountpoint\n" if @ARGV <= 1;

my $mp = pop;

my $g = Sys::Guestfs->new ();
foreach (@ARGV) {
    $g->add_drive ($_);
}
$g->launch ();

# Examine the filesystems.
my %fses = $g->list_filesystems ();

# Create the mountpoint directories (in the libguestfs namespace)
# and mount the filesystems on them.
foreach my $fs (sort keys %fses) {
    # mkmountpoint is really the same as mkdir.  Unfortunately there
    # is no 'mkdir -p' equivalent, so we have to do this instead:
    my @components = split ("/", $fs);
    for (my $i = 1; $i < @components; ++$i) {
        my $dir = "/" . join ("/", @components[1 .. $i]);
        eval { $g->mkmountpoint ($dir) }
    }

    # Don't fail if the filesystem can't be mounted, eg. it's swap.
    eval { $g->mount ($fs, $fs) }
}

# Export the filesystem on the host.
$g->mount_local ($mp);
$g->mount_local_run ();

# Close nicely since we mounted everything writable.
$g->shutdown ();
$g->close ();

Tip: Custom guestmount in Perl

Since libguestfs 1.18 guestmount has just been a slim wrapper around the libguestfs guestfs_mount_local API. You can replace guestmount with a small custom script if you want to do tricky/non-standard stuff like setting filesystem mount options, as in the example below.

$ ./mount-local.pl --ro -a /tmp/f17x64.img --mountopts=noatime /tmp/mount
mounting disk on /tmp/mount
to unmount: fusermount -u /tmp/mount

#!/usr/bin/perl -w

use strict;
use Getopt::Long;
use Sys::Guestfs;

my $readonly = 0;
my @drives;
my $mountopts;
my $trace = 0;

GetOptions ("ro|r" => \$readonly,
            "add|a=s" => \@drives,
            "mountopts=s" => \$mountopts,
            "trace|x" => \$trace)
    or die "$0 [--ro] [--add drive] [--mountopts mountopts] mountpoint\n";

die "$0: no drives (-a) were specified\n"
    unless @drives > 0;
die "$0: no mountpoint was specified\n"
    unless @ARGV == 1;

my $g = Sys::Guestfs->new ();
$g->set_trace (1) if $trace;
foreach (@drives) {
    $g->add_drive_opts ($_, readonly => $readonly)
}
$g->launch ();

# Inspect the disk to find OSes.
my @roots = $g->inspect_os ();
unless (@roots == 1) {
    die "$0: no operating systems found\n";
}
my $root = $roots[0];

# Mount up the disks like using the -i option.
my %mps = $g->inspect_get_mountpoints ($root);
my @mps = sort { length $a <=> length $b } (keys %mps);
foreach (@mps) {
    my $options = $readonly ? "ro" : "rw";
    $options .= "," . $mountopts if defined $mountopts;
    eval { $g->mount_options ($options, $mps{$_}, $_) };
    if ($@) {
        print "$@ (ignored)\n"
    }
}

# Export the filesystem using FUSE.
$g->mount_local ($ARGV[0]);

print "mounting disk on $ARGV[0]\n";
print "to unmount: fusermount -u $ARGV[0]\n";
$g->mount_local_run ();
# This returns when the filesystem is unmounted.

$g->shutdown ();
$g->close ();

Use guestfish, virt tools with remote disks

New in libguestfs ≥ 1.21.30 is the ability to use guestfish and some of the virt tools with remote disks.

Currently you can use remote disks over NBD, GlusterFS, Ceph, Sheepdog and (recently upstream) SSH.

For this example I’ll use SSH because it needs no setup, although this requires absolutely the latest qemu and libguestfs (both from git).

Since we don’t have libvirt support for ssh yet, so this only works with the direct backend:

$ export LIBGUESTFS_BACKEND=direct

I can use a ssh:// URI to add disks with guestfish, guestmount and most of the virt tools. For example:

$ virt-rescue -a ssh://localhost/tmp/f17x64.img
[... lots of boot messages ...]
Welcome to virt-rescue, the libguestfs rescue shell.

Note: The contents of / are the rescue appliance.
You have to mount the guest's partitions under /sysroot
before you can examine them.

><rescue> mount /dev/vg_f17x64/lv_root /sysroot
><rescue> cat /sysroot/etc/redhat-release
Fedora release 17 (Beefy Miracle)

Apart from being a tiny bit slower, it just works as if the disk was local:

$ virt-df -a ssh://localhost/tmp/f17x64.img
Filesystem                           1K-blocks       Used  Available  Use%
f17x64.img:/dev/sda1                    487652      63738     398314   14%
f17x64.img:/dev/vg_f17x64/lv_root     28316680    4285576   22586036   16%

$ guestmount -a ssh://localhost/tmp/f17x64.img -i /tmp/mnt
$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin  sys  usr
boot  etc  lib   lost+found  mnt    proc  run   srv   tmp  var
$ cat /tmp/mnt/etc/redhat-release
Fedora release 17 (Beefy Miracle)
$ guestunmount /tmp/mnt

Tip: Using mount-local API from C

Previously if you wanted to mount a disk image on the host from libguestfs, you had to use a separate program called guestmount.

A couple of months ago, we added FUSE support directly into the libguestfs API, and rewrote guestmount to use it. This also means you can use FUSE from your own libguestfs programs.

Yesterday I wrote a short example of using the mount-local API from C. This program creates a new disk image, formats it etc using libguestfs, then gives you a subshell so you can copy your own files, directories etc. in. When you quit the subshell, the disk image is synced and closed, and you end up with a virtual disk image containing all the files you just added. (Nothing that you couldn’t easily do before, but a nice little demonstration anyway).

Here it is in use:

$ gcc -Wall mount_local.c -o mount_local -lguestfs
$ ./mount_local /tmp/test.img

This is the 'mount-local' demonstration program.  Follow the
instructions on screen.

Creating and formatting the disk image, please wait a moment ...

The _current directory_ is a FUSE filesystem backed by the disk
image which is managed by libguestfs.  Any files or directories
you copy into here (up to 512 MB) will be saved into the disk
image.  You can also delete files, create certain special files
and so on.

When you have finished adding files, hit ^D or exit to exit the
shell and return to the mount-local program.

mount-local-shell> ls
lost+found  PUT_FILES_AND_DIRECTORIES_HERE

From the subshell, I copy in some files:

mount-local-shell> cp -a /usr/share/doc/libguestfs-devel-1.17.40/ .
mount-local-shell> ls
libguestfs-devel-1.17.40  lost+found  PUT_FILES_AND_DIRECTORIES_HERE
mount-local-shell> ls libguestfs-devel-1.17.40/
AUTHORS			       example-ubuntu.xml
BUGS			       example-windows-2003-x64-cd.xml
ChangeLog		       example-windows-2003-x86-cd.xml
copy_over.c		       example-windows.xml
create_disk.c		       example-windows-xp-cd.xml
display_icon.c		       HACKING
example-debian-netinst-cd.xml  inspect_vm.c
example-debian.xml	       README
example-fedora-dvd.xml	       RELEASE-NOTES
example-fedora-netinst-cd.xml  ROADMAP
example-fedora.xml	       TODO
example-rhel-6-dvd.xml	       virt-dhcp-address.c
example-rhel-6-netinst-cd.xml  virt-inspector.rng
example-ubuntu-live-cd.xml

After copying in my files, I exit from the subshell:

mount-local-shell> exit

Any files or directories that you copied in have been saved into
the disk image called '/tmp/test.img'.

Try opening the disk image with guestfish to see those files:

  guestfish -a /tmp/test.img -m /dev/sda1

Here is the disk image that was created (note it is sparse, so it’s not really so large as it appears):

$ ll /tmp/test.img
-rw-r--r--. 1 rjones rjones 536870912 May 14 12:03 /tmp/test.img
$ du -sh /tmp/test.img
18M	/tmp/test.img

We can use guestfish to look inside it:

$ guestfish -a /tmp/test.img -m /dev/sda1

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

><fs> ll /
total 18
drwxr-xr-x  4 root root  1024 May 14 12:03 .
drwxr-xr-x 23 1000 1000  4096 May 14 12:18 ..
-rw-r--r--  1 root root     0 May 14 12:03 PUT_FILES_AND_DIRECTORIES_HERE
drwxr-xr-x  2 root root  1024 May 14 08:37 libguestfs-devel-1.17.40
drwx------  2 root root 12288 May 14 12:03 lost+found

Download the test program here: https://github.com/libguestfs/libguestfs/blob/master/examples/mount_local.c

libguestfs: mount local

New in libguestfs ≥ 1.17.22 is the ability to mount the guest filesystem on a local mountpoint. Well, you can already do that using guestmount, but the new thing is that you can do it from the API (from any language).

Here is an example using guestfish:

$ guestfish --ro -a /dev/vg_pin/F16x64 -i
 
Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 16 (Verne)
/dev/vg_f16x64/lv_root mounted on /
/dev/vda2 mounted on /boot

><fs> ! mkdir /tmp/mnt # creates a local directory
><fs> mount-local /tmp/mnt readonly:true
><fs> mount-local-run
# the errors come from a GNOME daemon that
# looks at all new mountpoints
libguestfs: error: lstat: /.Trash: No such file or directory
libguestfs: error: lstat: /.Trash-500: No such file or directory

Over in another terminal, we can see the filesystem mounted on the local directory /tmp/mnt:

$ ls /tmp/mnt
bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var
boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr
$ cat /tmp/mnt/etc/redhat-release
Fedora release 16 (Verne)

Unmounting the filesystem causes the guestfish mount-local-run command to return (since that command was actually serving the FUSE requests from the kernel).

$ fusermount -u /tmp/mnt

New tool: virt-list-filesystems

$ virt-list-filesystems Debian5x64.img
/dev/debian5x64/home
/dev/debian5x64/root
/dev/debian5x64/tmp
/dev/debian5x64/usr
/dev/debian5x64/var
/dev/sda1

You can also augment this tool with the -a and -l options. The -a option tells it to list swap partitions too. The -l option tells it to show the filesystem type on each partition that was found:

$ virt-list-filesystems -a -l Fedora12.img
/dev/sda1 ext4
/dev/vg_f12x64/lv_root ext4
/dev/vg_f12x64/lv_swap swap

While this is a fairly simple tool, the use case comes from a user who asked me how I knew what filesystems could be mounted using the guestmount command. The answer is that you don’t know, unless you know something about the guest, or you interactively examine the guest using guestfish, or just use this new tool.

Example: Mount a Debian guest on the host using FUSE and libguestfs

Example — mount my Debian guest on my host Fedora server, using FUSE support which we added to libguestfs today:

$ mkdir /tmp/rich
$ guestmount $(virt-inspector --ro-fish /dev/vg_trick/Debian5x64) /tmp/rich
$ cat /tmp/rich/etc/debian_version
squeeze/sid
$ cat /tmp/rich/etc/hostname
debian5x64
$ ls -l /tmp/rich/etc/apt/
total 19
-rw-r--r-- 1 root root   51 2009-05-14 18:07 apt.conf
drwxr-xr-x 2 root root 1024 2009-08-13 18:10 apt.conf.d
drwxr-xr-x 2 root root 1024 2009-08-06 14:42 preferences.d
-rw------- 1 root root    0 2009-05-14 18:04 secring.gpg
-rw-r--r-- 1 root root  669 2009-05-24 12:02 sources.list
-rw-r--r-- 1 root root    0 2009-05-14 18:04 sources.list~
drwxr-xr-x 2 root root 1024 2009-02-07 21:18 sources.list.d
-rw------- 1 root root 1200 2009-05-14 18:04 trustdb.gpg
-rw------- 1 root root 5801 2009-05-14 18:04 trusted.gpg
-rw------- 1 root root 5801 2009-05-14 18:04 trusted.gpg~
$ nautilus /tmp/rich/home/rjones/d/libguestfs/

As you can see we can browse the Debian guest with nautilus. That’s actually a git checkout of the libguestfs source that we use to verify the build on Debian: