Tag Archives: nbdkit

Tip: Run virt-inspector on a compressed disk (with nbdkit)

virt-inspector is a very convenient tool to examine a disk image and find out if it contains an operating system, what applications are installed and so on.

If you have an xz-compressed disk image, you can run virt-inspector on it without uncompressing it, using the magic of captive nbdkit. Here’s how:

nbdkit xz file=win7.img.xz \
    -U - \
    --run 'virt-inspector --format=raw -a nbd://?socket=$unixsocket'

What’s happening here is we run nbdkit with the xz plugin, and tell it to serve NBD over a randomly named Unix domain socket (-U -).

We then run virt-inspector as a sub-process. This is called “captive nbdkit”. (Nbdkit is “captive” here, because it will exit as soon as virt-inspector exits, so there’s no need to clean anything up.)

The $unixsocket variable expands to the name of the randomly generated Unix domain socket, forming a libguestfs NBD URL which allows virt-inspector to examine the raw uncompressed data exported by nbdkit.

The nbdkit xz plugin only uncompresses those blocks of the data which are actually accessed, so this is quite efficient.

2 Comments

Filed under Uncategorized

nbdkit ruby plugin

NBD is a protocol for accessing Block Devices (actual hard disks, and things that look like hard disks). nbdkit is a toolkit for creating NBD servers.

You can now write nbdkit plugins in Ruby.

(So in all that makes: C/C++, Perl, Python, OCaml or Ruby as your choices for nbdkit plugins)

1 Comment

Filed under Uncategorized

New in nbdkit 1.1.10: OCaml plugins

You can now write OCaml plugins for nbdkit – the liberally licensed NBD server. You will, however, need OCaml ≥ 4.02.2+rc1 because of this fix.

Leave a comment

Filed under Uncategorized

Mapping files to disk, part 2

Part 1

Now I’ve written the second tool of virt-bmap which lets you boot a guest and observe what files it is reading from disk. (NB if you want to try this out you will need a patched libguestfs)

The second tool is an nbdkit plugin, so to use the tool you just do:

$ nbdkit -r bmaplogger file=/tmp/win7.img bmap=/tmp/win7.bmap \
  --run ' qemu-kvm -cpu host -m 2048 -hda $nbd '

and watch the output as the guest boots. Note that the bmap file must have been prepared previously by the virt-bmap tool (see part 1).

The results are interesting. Here is Windows 7 booting (edited down for brevity):

read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/tr-TR/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-HK/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/de-DE/bootmgr.exe.mui
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read v /dev/sda
read p /dev/sda2
read d /dev/sda2 /
read f /dev/sda2 /Windows/System32/Msdtc/MSDTC.LOG
read d /dev/sda2 /
read f /dev/sda2 /ProgramData/Microsoft/Search/Data/Applications/Windows/MSSres00001.jrs
read d /dev/sda2 /
read d /dev/sda2 /Users
read p /dev/sda2
read d /dev/sda2 /Windows/assembly/NativeImages_v2.0.50727_64
read d /dev/sda2 /Windows
read p /dev/sda2
read d /dev/sda2 /Windows/servicing
read d /dev/sda2 /Windows
read f /dev/sda2 /Windows/System32/config/SAM.LOG1
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32/en-US/Licenses/_Default
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read d /dev/sda2 /Windows/System32/Tasks/Microsoft/Windows
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/CIRCoInst.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/clb.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cmmon32.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cryptnet.dll
read d /dev/sda2 /Windows/System32
[...]
read f /dev/sda2 /Windows/System32/iscsilog.dll
read f /dev/sda2 /Windows/System32/ksetup.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/ksproxy.ax
read f /dev/sda2 /Windows/System32/NcdProp.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/nci.dll
read f /dev/sda2 /Windows/System32/profsvc.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/propsys.dll
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/winload.exe
[...]

Here is a Windows server that had McAfee (a “virus scanner”) installed:

read v /dev/sda
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /bootmgr
read v /dev/sda
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log0.txt
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/nl-NL/bootmgr.exe.mui
read f /dev/sda1 /Boot/pl-PL/bootmgr.exe.mui
read f /dev/sda1 /Boot/ru-RU/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log4.txt
read f /dev/sda1 /Boot/BCD
read p /dev/sda2
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda2 /Program Files (x86)/Common Files/System/msadc/adcjavas.inc
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/auditmanager.log
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/MSBuild/Microsoft/Windows Workflow Foundation/v3.0/Workflow.Targets
read f /dev/sda2 /Windows/ServerEnterprise.xml
read f /dev/sda2 /Windows/inf/setupapi.dev.log
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/Internet Explorer/en-US/jsprofilerui.dll.mui
read f /dev/sda2 /Users/tempadmin/AppData/Local/Microsoft/Internet Explorer/Recovery/High/Last Active/{7101D2F0-982F-11E0-A584-005056A7000F}.dat
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Plugins/AuEngineUpdater.dll
read f /dev/sda2 /Windows/System32/clusapi.dll
read f /dev/sda2 /Windows/System32/cmcfg32.dll
read f /dev/sda2 /Windows/winsxs/Backup/amd64_microsoft-windows-com-base_31bf3856ad364e35_6.1.7600.16385_none_69e3281e403684ea_comcat.dll_8571d1d1
read f /dev/sda2 /Windows/System32/comdlg32.dll
read f /dev/sda2 /Windows/SysWOW64/comexp.msc
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Schema/linux-definitions-schema.xsd
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/hhctrl.ocx
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log2.txt
read f /dev/sda2 /Windows/System32/KBDA1.DLL
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/Kswdmcap.ax
read f /dev/sda2 /Windows/SysWOW64/NOISE.CHS
read f /dev/sda2 /Windows/System32/NlsData0003.dll
read f /dev/sda2 /Windows/SysWOW64/RacRules.xml
read f /dev/sda2 /Windows/System32/ROUTE.EXE
read f /dev/sda2 /Windows/SysWOW64/en-US/tapimgmt.msc
read f /dev/sda2 /Windows/SysWOW64/en-US/tpm.msc
read f /dev/sda2 /Windows/System32/TpmInit.exe
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/oval.db
read f /dev/sda2 /Windows/Microsoft.NET/Framework64/v4.0.30319/ngen.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Audit.db
read f /dev/sda2 /Windows/System32/winload.exe

I wouldn’t take any of these traces very literally right now. Our method of mapping files to disk blocks is a bit shaky, especially for ntfs-3g. However I did check the major points of the McAfee trace against the raw log and block map and it seems plausible.

Leave a comment

Filed under Uncategorized

Mapping files to disk

Wouldn’t it be cool if you could watch a virtual machine booting, and at the same time see what files it is accessing on disk:

reading /dev/sda1 master boot record
reading /dev/sda1 /grub2/i386-pc/boot.img
reading /dev/sda1 /grub2/i386-pc/ext2.mod
reading /dev/sda1 /vmlinuz
...

You can already observe what disk blocks it is accessing pretty easily. There are several methods, but a quick one would be to use nbdkit’s file plugin with the -f -v flags (foreground and verbose). The problem is how to map disk blocks to the files and other interesting objects that exist in the disk image.

How do you map between files and disk blocks? For simple filesystems like ext4 you can use the FIBMAP ioctl, and perhaps adjust the answer by adding the offset of the start of the partition. However as you get further into the boot process you’ll probably encounter complexities like LVM. There may not even be a 1-1 mapping since RAID means that multiple blocks can store a single file block, and tail packing and deduplication mean that a block can belong to multiple files. And of course there are things other than plain files: directories, swap partitions, master boot records, and boot loaders, that live in and between filesystems.

To solve this I have written a tool called virt-bmap. It takes a disk image and outputs a block map. To do this it uses libguestfs (patched) to control an nbdkit instance, reading each file and recording what blocks in the disk image are accessed. (It sounds complicated, but virt-bmap wraps it up in a simple command line tool.) The beauty of this is that the kernel takes care of the mapping for us, and it works no matter how many layers of filesystem/LVM/RAID are between the file and the underlying device. This doesn’t quite solve the “RAID problem” since the RAID layers in Linux are free to only read a single copy of the file, but is generally accurate for everything else.

$ virt-bmap fedora-20.img
virt-bmap: examining /dev/sda1 ...
virt-bmap: examining /dev/sda2 ...
virt-bmap: examining /dev/sda3 ...
virt-bmap: examining filesystem on /dev/sda1 (ext4) ...
virt-bmap: examining filesystem on /dev/sda3 (ext4) ...
virt-bmap: writing /home/rjones/d/virt-bmap/bmap
virt-bmap: successfully examined 3 partitions, 0 logical volumes,
           2 filesystems, 3346 directories, 20585 files
virt-bmap: output written to /home/rjones/d/virt-bmap/bmap

The output bmap file is a straightforward map from disk byte offset to file / files / object occupying that space:

1 541000 541400 d /dev/sda1 /
1 541400 544400 d /dev/sda1 /lost+found
1 941000 941400 f /dev/sda1 /.vmlinuz-3.11.10-301.fc20.x86_64.hmac
1 941400 961800 f /dev/sda1 /config-3.11.10-301.fc20.x86_64
1 961800 995400 f /dev/sda1 /initrd-plymouth.img
1 b00400 ef1c00 f /dev/sda1 /grub2/themes/system/background.png
1 f00400 12f1c00 f /dev/sda1 /grub2/themes/system/fireworks.png
1 1300400 1590400 f /dev/sda1 /System.map-3.11.10-301.fc20.x86_64

[The 1 that appears in the first column means “first disk”. Unfortunately virt-bmap can only map single disk virtual machines at present.]

The second part of this, which I’m still writing, will be another nbdkit plugin which takes these maps and produces a nice log of accesses as the machine boots.

3 Comments

Filed under Uncategorized

Streaming NBD server

The command:

qemu-img convert input output

does not work if the output is a pipe.

It’d sure be nice if it did though! For one thing, we could use this in virt-v2v to stream images into OpenStack Glance (instead of having to spool them into a temporary file).

I mentioned this to Paolo Bonzini yesterday and he suggested a simple workaround. Just replace the output with:

qemu-img convert -n input nbd:...

and write an NBD server that turns the sequence of writes from qemu-img into a stream that gets written to a pipe. Assuming the output is raw, then qemu-img convert will write, starting at disk offset 0, linearly through to the end of the disk image.

How to write such an NBD server easily? nbdkit is a project I started to make it easy to write NBD servers.

So I wrote a streaming plugin which does exactly that, in 243 lines of code.

Using a feature called captive nbdkit, you can rewrite the above command as:

nbdkit -U - streaming pipe=/tmp/output --run '
  qemu-img convert -n input -O raw $nbd
'

(This command will “hang” when you run it — you have to attach some process to read from the pipe, eg: md5sum < /tmp/output)

Further work

The streaming plugin will a lot more generally useful if it supported a sliding window, allowing limited reverse seeking and reading. So there’s a nice little project for a motivated person. See here

5 Comments

Filed under Uncategorized

nbdkit now supports cURL — HTTP, FTP, and SSH connections

nbdkit is a liberally licensed NBD (Network Block Device) server designed to let you connect all sorts of crazy disk images sources (like Amazon, Glance, VMware VDDK) to the universal network protocol for sharing disk images: NBD.

New in nbdkit 1.1.8: cURL support. This lets you turn any HTTP, FTP, TFTP or SSH server that hosts a disk image into an NBD server.

For example:

$ nbdkit -r curl url=http://onuma/scratch/boot.iso

and then you can read the disk image using guestfish, qemu or any other nbd client:

$ guestfish --ro -a nbd://localhost -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

/dev/sda mounted on /

><fs> _

If you are using a normal SSH server like OpenSSH which supports the SSH File Transfer Protocol (aka SFTP), then you can use SFTP to access images:

$ nbdkit -r curl url=sftp://rjones@localhost/~/fedora-20.img

I’m hoping to enable write support in a future version.

It doesn’t work at the moment because I haven’t worked out how to switch between read (GET) and write (POST) requests in a single cURL handle. Perhaps I need to use two handles? The documentation is confusing.

2 Comments

Filed under Uncategorized