Tag Archives: virtualization

Tip: virt-install Windows with virtio device drivers

You have to unset these variables because of a long-standing bug in SPICE:

# unset http_proxy
# unset https_proxy

You can’t use virt-install’s --cdrom option twice, because virt-install ignores the second use of the option and only adds a single CD-ROM to the guest. Instead, use --disk ...,device=cdrom,bus=ide:

# virt-install --name=w81-virtio --ram=4096 \
    --cpu=host --vcpus=2 \
    --os-type=windows --os-variant=win8.1 \
    --disk /dev/VG/w81-virtio,bus=virtio \
    --disk en-gb_windows_8.1_pro_n_vl_with_update_x64_dvd_6050975.iso,device=cdrom,bus=ide \
    --disk /usr/share/virtio-win/virtio-win.iso,device=cdrom,bus=ide

During the install you’ll have to select the “Load driver” option and load the right viostor driver from the second CD-ROM (E:).

1 Comment

Filed under Uncategorized

Tip: Wake up a guest from screen blank

A few years ago Dan Berrange added a way to send fake keyboard events to libvirt guests. You can use this to inject just a press on the Left Shift key to wake up a guest from screen blank. Very useful if you need to take a screenshot!

$ virsh send-key guest KEY_LEFTSHIFT
$ sleep 1
$ virsh screenshot guest /tmp/screenshot.ppm

Update: A word of warning though. If you try this for Windows guests you’ll hit this message:

win_2003r2_x86_64_no_tools-20150401-130735

The solution is to hit other keys randomly. Grrr.

Leave a comment

Filed under Uncategorized

Edit UEFI varstores

See end of post for an important update

UEFI firmware has a concept of persistent variables. They are used to control the boot order amongst other things. They are stored in non-volatile RAM on the system board, or for virtual machines in a host file.

When a UEFI machine is running you can edit these variables using various tools, such as Peter Jones’s efivar library, or the efibootmgr program.

These programs don’t actually edit the varstore directly. They access the kernel /sys/firmware/efi interface, but even the kernel doesn’t edit the varstore. It just redirects to the UEFI runtime “Variable Services”, so what is really running is UEFI code (possibly proprietary, but more usually from the open source TianoCore project).

So how can you edit varstores offline? The NVRAM file format is peculiar to say the least, and the only real specification is the code that writes it from Tianocore. So somehow you must reuse that code. To make it more complicated, the varstore NVRAM format is tied to the specific firmware that uses it, so varstores used on aarch64 aren’t compatible with those on x86-64, nor are SecureBoot varstores compatible with normal ones.

virt-efivars is an attempt to do that. It’s rather “meta”. You write a small editor program (an example is included), and virt-efivars compiles it into a tiny appliance. You then boot the appliance using qemu + UEFI firmware + varstore combination, the editor program runs and edits the varstore, using the UEFI code.

It works .. at least on aarch64 which is the only convenient machine I have that has virtualized UEFI.

Git repo: http://git.annexia.org/?p=virt-efivars.git;a=summary

Update:

After studying this problem some more, Laszlo Ersek came up with a different and better plan:

  1. Boot qemu with only the OVMF code & varstore attached. No OS or appliance.
  2. This should drop you into a UEFI shell which is accessible over qemu’s serial port.
  3. Send appropriate setvar commands to update the variables. Using expect this should be automatable.

Leave a comment

Filed under Uncategorized

Random numbers in virtual machines

My colleague Amit Shah wrote this interesting article about getting random numbers inside virtual machines.

Leave a comment

Filed under Uncategorized

Half-baked ideas: qemu -M container

For more half-baked ideas, see the ideas tag.

Containers offer a way to do limited virtualization with fewer resources. But a lot of people have belatedly realized that containers aren’t secure, and so there’s a trend for putting containers into real virtual machines.

Unfortunately qemu is not very well suited to just running a single instance of the Linux kernel, as we in the libguestfs community have long known. There are at least a couple of problems:

  1. You have to allocate a fixed amount of RAM to the VM. This is basically a guess. Do you guess too large and have memory wasted in guest kernel structures, or do you guess too small and have the VM fail at random?
  2. There’s a large amount of overhead — firmware, legacy device emulation and other nonsense — which is essentially irrelevant to the special case of running a Linux appliance in a VM.

Here’s the half-baked idea: Let’s make a qemu “container mode/machine” which is better for this one task.

Unlike other proposals in this area, I’m not suggesting that we throw away or rewrite qemu. That’s stupid, as qemu gives us lots of useful abilities.

Instead the right way to do this is to implement a special virtio-ram device where the guest kernel can start off with a very tiny amount of RAM and request more memory on demand. And an empty machine type which is just for running appliances (qemu on ARM already has this: mach-virt).

Libguestfs people and container people, all happy. What’s not to like?

1 Comment

Filed under Uncategorized

Mapping files to disk, part 2

Part 1

Now I’ve written the second tool of virt-bmap which lets you boot a guest and observe what files it is reading from disk. (NB if you want to try this out you will need a patched libguestfs)

The second tool is an nbdkit plugin, so to use the tool you just do:

$ nbdkit -r bmaplogger file=/tmp/win7.img bmap=/tmp/win7.bmap \
  --run ' qemu-kvm -cpu host -m 2048 -hda $nbd '

and watch the output as the guest boots. Note that the bmap file must have been prepared previously by the virt-bmap tool (see part 1).

The results are interesting. Here is Windows 7 booting (edited down for brevity):

read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/tr-TR/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-HK/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/de-DE/bootmgr.exe.mui
read p /dev/sda1
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda1 /Boot/da-DK/bootmgr.exe.mui
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read v /dev/sda
read p /dev/sda2
read d /dev/sda2 /
read f /dev/sda2 /Windows/System32/Msdtc/MSDTC.LOG
read d /dev/sda2 /
read f /dev/sda2 /ProgramData/Microsoft/Search/Data/Applications/Windows/MSSres00001.jrs
read d /dev/sda2 /
read d /dev/sda2 /Users
read p /dev/sda2
read d /dev/sda2 /Windows/assembly/NativeImages_v2.0.50727_64
read d /dev/sda2 /Windows
read p /dev/sda2
read d /dev/sda2 /Windows/servicing
read d /dev/sda2 /Windows
read f /dev/sda2 /Windows/System32/config/SAM.LOG1
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32/en-US/Licenses/_Default
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read d /dev/sda2 /Windows/System32
read d /dev/sda2 /Windows/System32/Tasks/Microsoft/Windows
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/CIRCoInst.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/clb.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cmmon32.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/cryptnet.dll
read d /dev/sda2 /Windows/System32
[...]
read f /dev/sda2 /Windows/System32/iscsilog.dll
read f /dev/sda2 /Windows/System32/ksetup.exe
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/ksproxy.ax
read f /dev/sda2 /Windows/System32/NcdProp.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/nci.dll
read f /dev/sda2 /Windows/System32/profsvc.dll
read d /dev/sda2 /Windows/System32
read f /dev/sda2 /Windows/System32/propsys.dll
read d /dev/sda2 /Windows/System32
read p /dev/sda2
read f /dev/sda2 /Windows/System32/winload.exe
[...]

Here is a Windows server that had McAfee (a “virus scanner”) installed:

read v /dev/sda
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /bootmgr
read v /dev/sda
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log0.txt
read v /dev/sda
read p /dev/sda1
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/nl-NL/bootmgr.exe.mui
read f /dev/sda1 /Boot/pl-PL/bootmgr.exe.mui
read f /dev/sda1 /Boot/ru-RU/bootmgr.exe.mui
read f /dev/sda1 /Boot/zh-TW/bootmgr.exe.mui
read f /dev/sda1 /bootmgr
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/Fonts/kor_boot.ttf
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /BOOTSECT.BAK
read f /dev/sda1 /Boot/BCD
read f /dev/sda1 /Boot/BOOTSTAT.DAT
read f /dev/sda1 /Boot/BCD
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log4.txt
read f /dev/sda1 /Boot/BCD
read p /dev/sda2
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda1 /Boot/cs-CZ/bootmgr.exe.mui
read f /dev/sda2 /Program Files (x86)/Common Files/System/msadc/adcjavas.inc
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/auditmanager.log
read f /dev/sda2 /Program Files (x86)/Common Files/microsoft shared/DAO/dao360.dll
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/MSBuild/Microsoft/Windows Workflow Foundation/v3.0/Workflow.Targets
read f /dev/sda2 /Windows/ServerEnterprise.xml
read f /dev/sda2 /Windows/inf/setupapi.dev.log
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log7.txt
read f /dev/sda2 /Program Files (x86)/Internet Explorer/en-US/jsprofilerui.dll.mui
read f /dev/sda2 /Users/tempadmin/AppData/Local/Microsoft/Internet Explorer/Recovery/High/Last Active/{7101D2F0-982F-11E0-A584-005056A7000F}.dat
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Plugins/AuEngineUpdater.dll
read f /dev/sda2 /Windows/System32/clusapi.dll
read f /dev/sda2 /Windows/System32/cmcfg32.dll
read f /dev/sda2 /Windows/winsxs/Backup/amd64_microsoft-windows-com-base_31bf3856ad364e35_6.1.7600.16385_none_69e3281e403684ea_comcat.dll_8571d1d1
read f /dev/sda2 /Windows/System32/comdlg32.dll
read f /dev/sda2 /Windows/SysWOW64/comexp.msc
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Schema/linux-definitions-schema.xsd
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/SysWOW64/C_10003.NLS
read f /dev/sda2 /Windows/SysWOW64/C_10004.NLS
read f /dev/sda2 /Windows/SysWOW64/C_20005.NLS
read f /dev/sda2 /Windows/SysWOW64/C_21025.NLS
read f /dev/sda2 /Windows/CMAgent/Installer/Providers/ExecutionEngine/providers.catalog
read f /dev/sda2 /Windows/SysWOW64/dfsrHealthReport.xsl
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/hhctrl.ocx
read f /dev/sda2 /Program Files (x86)/McAfee/Real Time/log2.txt
read f /dev/sda2 /Windows/System32/KBDA1.DLL
read f /dev/sda2 /ProgramData/McAfee/Common Framework/Mesh/SvcMgr_WPLCLDWA170.log
read f /dev/sda2 /Windows/System32/Kswdmcap.ax
read f /dev/sda2 /Windows/SysWOW64/NOISE.CHS
read f /dev/sda2 /Windows/System32/NlsData0003.dll
read f /dev/sda2 /Windows/SysWOW64/RacRules.xml
read f /dev/sda2 /Windows/System32/ROUTE.EXE
read f /dev/sda2 /Windows/SysWOW64/en-US/tapimgmt.msc
read f /dev/sda2 /Windows/SysWOW64/en-US/tpm.msc
read f /dev/sda2 /Windows/System32/TpmInit.exe
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/oval.db
read f /dev/sda2 /Windows/Microsoft.NET/Framework64/v4.0.30319/ngen.log
read f /dev/sda2 /Program Files (x86)/McAfee/Policy Auditor Agent/Audit.db
read f /dev/sda2 /Windows/System32/winload.exe

I wouldn’t take any of these traces very literally right now. Our method of mapping files to disk blocks is a bit shaky, especially for ntfs-3g. However I did check the major points of the McAfee trace against the raw log and block map and it seems plausible.

Leave a comment

Filed under Uncategorized

Mapping files to disk

Wouldn’t it be cool if you could watch a virtual machine booting, and at the same time see what files it is accessing on disk:

reading /dev/sda1 master boot record
reading /dev/sda1 /grub2/i386-pc/boot.img
reading /dev/sda1 /grub2/i386-pc/ext2.mod
reading /dev/sda1 /vmlinuz
...

You can already observe what disk blocks it is accessing pretty easily. There are several methods, but a quick one would be to use nbdkit’s file plugin with the -f -v flags (foreground and verbose). The problem is how to map disk blocks to the files and other interesting objects that exist in the disk image.

How do you map between files and disk blocks? For simple filesystems like ext4 you can use the FIBMAP ioctl, and perhaps adjust the answer by adding the offset of the start of the partition. However as you get further into the boot process you’ll probably encounter complexities like LVM. There may not even be a 1-1 mapping since RAID means that multiple blocks can store a single file block, and tail packing and deduplication mean that a block can belong to multiple files. And of course there are things other than plain files: directories, swap partitions, master boot records, and boot loaders, that live in and between filesystems.

To solve this I have written a tool called virt-bmap. It takes a disk image and outputs a block map. To do this it uses libguestfs (patched) to control an nbdkit instance, reading each file and recording what blocks in the disk image are accessed. (It sounds complicated, but virt-bmap wraps it up in a simple command line tool.) The beauty of this is that the kernel takes care of the mapping for us, and it works no matter how many layers of filesystem/LVM/RAID are between the file and the underlying device. This doesn’t quite solve the “RAID problem” since the RAID layers in Linux are free to only read a single copy of the file, but is generally accurate for everything else.

$ virt-bmap fedora-20.img
virt-bmap: examining /dev/sda1 ...
virt-bmap: examining /dev/sda2 ...
virt-bmap: examining /dev/sda3 ...
virt-bmap: examining filesystem on /dev/sda1 (ext4) ...
virt-bmap: examining filesystem on /dev/sda3 (ext4) ...
virt-bmap: writing /home/rjones/d/virt-bmap/bmap
virt-bmap: successfully examined 3 partitions, 0 logical volumes,
           2 filesystems, 3346 directories, 20585 files
virt-bmap: output written to /home/rjones/d/virt-bmap/bmap

The output bmap file is a straightforward map from disk byte offset to file / files / object occupying that space:

1 541000 541400 d /dev/sda1 /
1 541400 544400 d /dev/sda1 /lost+found
1 941000 941400 f /dev/sda1 /.vmlinuz-3.11.10-301.fc20.x86_64.hmac
1 941400 961800 f /dev/sda1 /config-3.11.10-301.fc20.x86_64
1 961800 995400 f /dev/sda1 /initrd-plymouth.img
1 b00400 ef1c00 f /dev/sda1 /grub2/themes/system/background.png
1 f00400 12f1c00 f /dev/sda1 /grub2/themes/system/fireworks.png
1 1300400 1590400 f /dev/sda1 /System.map-3.11.10-301.fc20.x86_64

[The 1 that appears in the first column means “first disk”. Unfortunately virt-bmap can only map single disk virtual machines at present.]

The second part of this, which I’m still writing, will be another nbdkit plugin which takes these maps and produces a nice log of accesses as the machine boots.

3 Comments

Filed under Uncategorized