Tag Archives: perl

Tip: guestmount (FUSE mount) every filesystem in a disk image

Maxim asks an interesting question which is if you’ve got a disk image, how do you mount every filesystem onto your host. Like this:

$ ./fs-mount.pl rhel-5.11.img /tmp/fs &
$ cd /tmp/fs
/tmp/fs$ ls
dev
/tmp/fs$ cd dev
/tmp/fs/dev$ ls
sda1  sda2  sda3
/tmp/fs/dev$ cd sda2
/tmp/fs/dev/sda2$ ls
bin   dev  home  lib64       media  mnt  proc  sbin     srv  tmp  var
boot  etc  lib   lost+found  misc   opt  root  selinux  sys  usr
...
$ cd /tmp
$ guestunmount /tmp/fs

The answer is this surprisingly short Perl script.

#!/usr/bin/perl

use warnings;
use strict;

use Sys::Guestfs;

die "usage: $0 disk1 [disk2 ...] mountpoint\n" if @ARGV <= 1;

my $mp = pop;

my $g = Sys::Guestfs->new ();
foreach (@ARGV) {
    $g->add_drive ($_);
}
$g->launch ();

# Examine the filesystems.
my %fses = $g->list_filesystems ();

# Create the mountpoint directories (in the libguestfs namespace)
# and mount the filesystems on them.
foreach my $fs (sort keys %fses) {
    # mkmountpoint is really the same as mkdir.  Unfortunately there
    # is no 'mkdir -p' equivalent, so we have to do this instead:
    my @components = split ("/", $fs);
    for (my $i = 1; $i < @components; ++$i) {
        my $dir = "/" . join ("/", @components[1 .. $i]);
        eval { $g->mkmountpoint ($dir) }
    }

    # Don't fail if the filesystem can't be mounted, eg. it's swap.
    eval { $g->mount ($fs, $fs) }
}

# Export the filesystem on the host.
$g->mount_local ($mp);
$g->mount_local_run ();

# Close nicely since we mounted everything writable.
$g->shutdown ();
$g->close ();

Leave a comment

Filed under Uncategorized

New in nbdkit: Run nbdkit as a captive process

New in nbdkit ≥ 1.1.6, you can run nbdkit as a “captive process” under external programs like qemu or guestfish. This means that nbdkit runs for as long as qemu/guestfish is running, and when they exit it cleans up and exits too.

Here is a rather involved way to boot a Fedora 20 guest:

$ virt-builder fedora-20
$ nbdkit file file=fedora-20.img \
    --run 'qemu-kvm -m 1024 -drive file=$nbd,if=virtio'

The --run parameter is what tells nbdkit to run as a captive under qemu-kvm. $nbd on the qemu command line is substituted automatically with the right nbd: URL for the port or socket that nbdkit listens on. As soon as qemu-kvm exits, nbdkit is killed and cleaned up.

Here is another example using guestfish:

$ nbdkit file file=fedora-20.img \
    --run 'guestfish --format=raw -a $nbd -i'

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Fedora release 20 (Heisenbug)
/dev/sda3 mounted on /
/dev/sda1 mounted on /boot

><fs>

The main use for this is not to run the nbdkit file plugin like this, but in conjunction with perl and python plugins, to let people easily open and edit OpenStack Glance/Cinder and other unconventional disk images.

4 Comments

Filed under Uncategorized

nbdkit: Write plugins in Perl

nbdkit is a liberally licensed NBD server, that lets you serve “unconventional” sources of disk images and make them available to qemu, libguestfs, etc.

In the latest version of nbdkit, you can now write your plugins in Perl [example].

Coming soon: The ability to write plugins in Python too.

1 Comment

Filed under Uncategorized

Tip: Custom guestmount in Perl

Since libguestfs 1.18 guestmount has just been a slim wrapper around the libguestfs guestfs_mount_local API. You can replace guestmount with a small custom script if you want to do tricky/non-standard stuff like setting filesystem mount options, as in the example below.

$ ./mount-local.pl --ro -a /tmp/f17x64.img --mountopts=noatime /tmp/mount
mounting disk on /tmp/mount
to unmount: fusermount -u /tmp/mount
#!/usr/bin/perl -w

use strict;
use Getopt::Long;
use Sys::Guestfs;

my $readonly = 0;
my @drives;
my $mountopts;
my $trace = 0;

GetOptions ("ro|r" => \$readonly,
            "add|a=s" => \@drives,
            "mountopts=s" => \$mountopts,
            "trace|x" => \$trace)
    or die "$0 [--ro] [--add drive] [--mountopts mountopts] mountpoint\n";

die "$0: no drives (-a) were specified\n"
    unless @drives > 0;
die "$0: no mountpoint was specified\n"
    unless @ARGV == 1;

my $g = Sys::Guestfs->new ();
$g->set_trace (1) if $trace;
foreach (@drives) {
    $g->add_drive_opts ($_, readonly => $readonly)
}
$g->launch ();

# Inspect the disk to find OSes.
my @roots = $g->inspect_os ();
unless (@roots == 1) {
    die "$0: no operating systems found\n";
}
my $root = $roots[0];

# Mount up the disks like using the -i option.
my %mps = $g->inspect_get_mountpoints ($root);
my @mps = sort { length $a <=> length $b } (keys %mps);
foreach (@mps) {
    my $options = $readonly ? "ro" : "rw";
    $options .= "," . $mountopts if defined $mountopts;
    eval { $g->mount_options ($options, $mps{$_}, $_) };
    if ($@) {
        print "$@ (ignored)\n"
    }
}

# Export the filesystem using FUSE.
$g->mount_local ($ARGV[0]);

print "mounting disk on $ARGV[0]\n";
print "to unmount: fusermount -u $ARGV[0]\n";
$g->mount_local_run ();
# This returns when the filesystem is unmounted.

$g->shutdown ();
$g->close ();

Leave a comment

Filed under Uncategorized

Setting the root (or other) passwords in a Linux guest

I wrote this example for the CentOS Dojo on Friday (which by the way will be recorded and available on YouTube afterwards).

You can use or modify this script to change the password in /etc/shadow for root or any other user of a guest.

The $5$ causes it to use a SHA-256-encrypted password, but you can change this to $6$ to use SHA-512 (both assume you are using glibc on the host).

#!/usr/bin/perl -w

use strict;
#use Sys::Virt;
use Sys::Guestfs;

my $vm = "dojo";
my $user = "root";
my $newpw = "1234567";

my $salt;
my @chars = ("A".."Z", "a".."z", "0".."9", ".", "/");
$salt .= $chars[rand @chars] for 1..16;
my $crypted = crypt ($newpw, '$5$' . $salt . '$');

my $g = Sys::Guestfs->new ();
$g->set_trace (1);
$g->add_domain ($vm, libvirturi => "qemu:///session");
$g->launch ();
$g->mount ("/dev/fedora/root", "/");

my @shadow = $g->read_lines ("/etc/shadow");
s/^root:.*?:/root:$crypted:/ foreach @shadow;

$g->write ("/etc/shadow", join ("\n", @shadow) . "\n");
$g->chmod (0, "/etc/shadow");

$g->touch ("/.autorelabel");

Leave a comment

Filed under Uncategorized

New in libguestfs: Use SYSLINUX or EXTLINUX to make bootable guests

Although grub support in libguestfs is currently on hold because of an unfortunate situation, the latest libguestfs now supports SYSLINUX and EXTLINUX, which is (let’s be frank about this) a much simpler and more sane bootloader than grub/grub2.

In fact, you can make a bootable Linux guest real easily now. Here’s a script:

#!/usr/bin/perl
# Copyright (C) 2013 Red Hat Inc.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

# This ambitious script creates a complete, bootable guest.

use strict;
use warnings;

use Sys::Guestfs;

my $disk = "syslinux-guest.img";

# Find prerequisites.
my $mbr = "/usr/share/syslinux/mbr.bin";
unless (-f $mbr) {
    $mbr = "/usr/lib/syslinux/mbr.bin";
    unless (-f $mbr) {
        die "$0: mbr.bin (from SYSLINUX) not found\n";
    }
}
print "mbr: $mbr\n";

my $mbr_data;
{
    local $/ = undef;
    open MBR, "$mbr" or die "$mbr: $!";
    $mbr_data = <MBR>;
}
die "invalid mbr.bin" unless length ($mbr_data) == 440;

my $kernel = `ls -1rv /boot/vmlinuz* | head -1`;
chomp $kernel;
unless ($kernel) {
    die "$0: kernel could not be found\n";
}
print "kernel: $kernel\n";

print "writing to: $disk ...\n";

# Create the disk.
unlink "$disk";
open DISK, ">$disk" or die "$disk: $!";
truncate DISK, 100*1024*1024;
close DISK;

my $g = Sys::Guestfs->new ();
$g->add_drive ($disk, format => "raw");
$g->launch ();

unless ($g->feature_available (["syslinux"])) {
    die "$0: 'syslinux' feature not available in this version of libguestfs\n";
}

# Format the disk.
$g->part_disk ("/dev/sda", "mbr");
$g->mkfs ("msdos", "/dev/sda1");
$g->mount ("/dev/sda1", "/");

# Install the kernel.
$g->upload ($kernel, "/vmlinuz");

# Install the SYSLINUX configuration file.
$g->write ("/syslinux.cfg", <<_END);
DEFAULT linux
LABEL linux
  SAY Booting the kernel from /vmlinuz
  KERNEL vmlinuz
  APPEND ro root=/dev/sda1
_END

$g->umount_all ();

# Install the bootloader.
$g->pwrite_device ("/dev/sda", $mbr_data, 0);
$g->syslinux ("/dev/sda1");
$g->part_set_bootable ("/dev/sda", 1, 1);

# Finish off.
$g->shutdown ();

After running the script, you can try booting the minimal “guest” (note it only contains a kernel, not any userspace):

$ qemu-kvm -hda syslinux-guest.img

1 Comment

Filed under Uncategorized

xavierbot lives!

Or at least he now has his own git repository.

1 Comment

Filed under Uncategorized

Which foreign function interface is the best?

I’ve written libguestfs language bindings for Perl, Python, Ruby, Java, OCaml, PHP, Haskell, Erlang and C#. But which of these is the best? Which is the easiest? What makes this hard? Grubbing around in the internals of a language reveals mistakes made by the language designers, but what are the worst mistakes?

Note: There is source that goes with this. Download libguestfs-1.13.13.tar.gz and look in the respective directories.

The best

It’s going to be a controversial choice, but in my opinion: C#. You just add some simple annotations to your functions and structs, and you can call into shared libraries (or “DllImport”s as Microsoft insisted on calling them) directly. It’s just about as easy as directly calling C and that is no simple achievement considering how the underlying runtime of C# is very different from C.

Example: a C struct:

[StructLayout (LayoutKind.Sequential)]
public class _int_bool {
  int i;
  int b;
}

The worst

There are two languages in the doghouse: Haskell and PHP. PHP first because their method of binding is just very broken. For example, 64 bit types aren’t possible on a 32 bit platform. It requires a very complex autoconf setup. And the quality of their implementation is very poor verging on broken — it makes me wonder if the rest of PHP can be this bad.

Haskell: even though I’m an experienced functional programmer and have done a fair bit of Haskell programming in the past, the FFI is deeply strange and very poorly documented. I simply could not work out how to return anything other than integers from my functions. You end up with bindings that look like this:

write_file h path content size = do
  r <- withCString path $ \path -> withCString content $ \content -> withForeignPtr h (\p -> c_write_file p path content (fromIntegral size))
  if (r == -1)
    then do
      err <- last_error h
      fail err
    else return ()

The middle tier

There’s not a lot to choose between OCaml, Ruby, Java and Erlang. For all of them: you write bindings in C, there’s good documentation, it’s a bit tedious but basically mechanical, and in 3 out of 4 you’re dealing with a reasonable garbage collector so you have to be aware of GC issues.

Erlang is slightly peculiar because the method I chose (out of many possible) is to write an external process that talks to the Erlang over stdin/stdout. But I can’t fault their documentation, and the rest of it is sensible.

Example: Here is a function binding in OCaml, but with mechanical changes this could be Ruby, Java or Erlang too:

CAMLprim value
ocaml_guestfs_add_drive_ro (value gv, value filenamev)
{
  CAMLparam2 (gv, filenamev);
  CAMLlocal1 (rv);

  guestfs_h *g = Guestfs_val (gv);
  if (g == NULL)
    ocaml_guestfs_raise_closed ("add_drive_ro");

  char *filename = guestfs_safe_strdup (g, String_val (filenamev));
  int r;

  caml_enter_blocking_section ();
  r = guestfs_add_drive_ro (g, filename);
  caml_leave_blocking_section ();
  free (filename);
  if (r == -1)
    ocaml_guestfs_raise_error (g, "add_drive_ro");

  rv = Val_unit;
  CAMLreturn (rv);
}

The ugly

Perl: Get reading. You’d better start with perlxs because Perl uses its own language — C with bizarre macros on top so your code looks like this:

SV *
is_config (g)
      guestfs_h *g;
PREINIT:
      int r;
   CODE:
      r = guestfs_is_config (g);
      if (r == -1)
        croak ("%s", guestfs_last_error (g));
      RETVAL = newSViv (r);
 OUTPUT:
      RETVAL

After that, get familiar with perlguts. Perl has only 3 structures and you’ll be using them a lot. There are some brilliant things about Perl which shouldn’t be overlooked, including POD which libguestfs uses to make effortless manual pages.

Python: Best described as half arsed. Rather like the language itself.

Python, Ruby, Erlang: If your language depends on “int”, “long”, “long long” without defining what those mean, and differing based on your C compiler and platform, then you’ve made a big mistake that will unfortunately dog you throughout the runtime, FFIs and the language itself. It’s better either to define them precisely (like Java) or to just use int32 and int64 (like OCaml).

And finally, reference counting (Perl, Python). It’s tremendously easy to make mistakes that are fiendishly difficult to track down. It’s a poor way to do GC and it indicates to me that the language designer didn’t know any better.

18 Comments

Filed under Uncategorized

RHEL 6.1 is out, new Virtualization Guide covers libguestfs

Red Hat Enterprise Linux 6.1 is out.

For this release the Virtualization guide was extensively updated. In particular I wrote a section on using libguestfs, guestfish and the virt tools for offline access to disk images.

I also managed to slip in a Perl joke …

5 Comments

Filed under Uncategorized

Decoding the Windows Event Log using guestfish

The Windows Event Log system is Windows’ centralized way of capturing log messages from the operating system and a wide variety of applications.

In all versions of Windows the messages are stored in binary files and normally you can only read these using Microsoft’s proprietary Event Viewer program. In Windows Vista, Microsoft overhauled the entire messaging system and changed the binary format. Luckily a German computer forensics researcher named Andreas Schuster reverse engineered the format [PDF] and wrote a small GPL’d program called EvtxParser [download dir] which can decode it.

We can use guestfish (or libguestfs) along with EvtxParser to easily look at the events in any Windows Vista / 2008 / 7 virtual machine.

Firstly download EvtxParser. You don’t need to install it (indeed, it doesn’t come with any build system so you can’t install it without some effort). You do need to install a few supporting Perl modules though:

# yum install perl-Digest-CRC perl-DateTime \
    perl-Carp-Assert perl-CPAN tidy
# cpan install Data::Hexify

Also the Perl scripts in the EvtxParser zip file aren’t all executable, so chmod them:

$ chmod +x *.pl

Now grab some *.evtx files from your Windows Vista (or later) system. They are stored in the /Windows/System32/winevt/Logs/ directory:

# guestfish --ro -i -d WindowsGuest
><fs> ll win:/Windows/System32/winevt/Logs
total 10540
drwxrwxrwx 1 root root   28672 Oct  1  2010 .
drwxrwxrwx 1 root root       0 Jul 14  2009 ..
-rwxrwxrwx 2 root root 1118208 Dec 23 18:22 Application.evtx
-rwxrwxrwx 2 root root   69632 Sep 19  2010 HardwareEvents.evtx
-rwxrwxrwx 2 root root   69632 Sep 19  2010 Internet Explorer.evtx
-rwxrwxrwx 2 root root   69632 Sep 19  2010 Key Management Service.evtx
-rwxrwxrwx 2 root root   69632 Sep 19  2010 Media Center.evtx
[and many more]
><fs> download win:/Windows/System32/winevt/Logs/System.evtx /tmp/System.evtx
><fs> exit

You can directly dump the files you have downloaded as XML to reveal the events inside them.

$ ./evtxdump.pl /tmp/System.evtx | tidy -xml -indent -quiet | less

This gives me a 40,000 line XML document(!) As a representative sample, the last event is the shutdown event from when I shut the VM off last time:

  <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
    <System>
      <Provider Name="Service Control Manager"
      Guid="{555908d1-a6d7-4695-8e1e-26931d2012f4}"
      EventSourceName="Service Control Manager" />
      <EventID Qualifiers="16384">7036</EventID>
      <Version>0</Version>
      <Level>4</Level>
      <Task>0</Task>
      <Opcode>0</Opcode>
      <Keywords>0x8080000000000000</Keywords>
      <TimeCreated SystemTime="2010-12-23T18:22:58.4980Z" />
      <EventRecordID>1452</EventRecordID>
      <Correlation />
      <Execution ProcessID="456" ThreadID="1748" />
      <Channel>System</Channel>
      <Computer>win7x32</Computer>
      <Security />
    </System>
    <EventData>
      <Data Name="param1">Power</Data>
      <Data Name="param2">stopped</Data>
      <Binary>50006F007700650072002F0031000000</Binary>
    </EventData>
  </Event>

A tip for reading these: the key field is the EventID. For example, EventID 1074 is a user-initiated clean shutdown.

3 Comments

Filed under Uncategorized