Tag Archives: esx

Tip: Read guest disks from VMware vCenter using libguestfs

virt-v2v can import guests directly from vCenter. It uses all sorts of tricks to make this fast and efficient, but the basic technique uses plain https range requests.

Making it all work was not so easy and involved a lot of experimentation and bug fixing, and I don’t think it has been documented up to now. So this post describes how we do it. As usual the code is the ultimate repository of our knowledge so you may want to consult that after reading this introduction.

Note this is read-only access. Write access is possible, but you’ll have to use ssh instead.

VMware ESXi hypervisor has a web server but doesn’t support range requests, so although you can download an entire disk image in one go from the ESXi hypervisor, to random-access the image using libguestfs you will need VMware vCenter. You should check that virsh dumpxml works against your vCenter instance by following these instructions. If that doesn’t work, it’s unlikely the rest of the instructions will work.

You will need to know:

  1. The hostname or IP address of your vCenter server,
  2. the username and password for vCenter,
  3. the name of your datacenter (probably Datacenter),
  4. the name of the datastore containing your guest (could be datastore1),
  5. .. and of course the name of your guest.

Tricky step 1 is to construct the vCenter https URL of your guest.

This looks like:

https://root:password@vcenter/folder/guest/guest-flat.vmdk?dcPath=Datacenter&dsName=datastore1

where:

root:password
username and password
vcenter
vCenter hostname or IP address
guest
guest name (repeated twice)
Datacenter
datacenter name
datastore1
datastore

Once you’ve got a URL that looks right, try to fetch the headers using curl. This step is important! not just because it checks the URL is good, but because it allows us to get a cookie which is required else vCenter will break under the load when we start to access it for real.

$ curl --insecure -I https://....
HTTP/1.1 200 OK
Date: Wed, 5 Nov 2014 19:38:32 GMT
Set-Cookie: vmware_soap_session="52a3a513-7fba-ef0e-5b36-c18d88d71b14"; Path=/; HttpOnly; Secure; 
Accept-Ranges: bytes
Connection: Keep-Alive
Content-Type: application/octet-stream
Content-Length: 8589934592

The cookie is the vmware_soap_session=... part including the quotes.

Now let’s make a qcow2 overlay which encodes our https URL and the cookie as the backing file. This requires a reasonably recent qemu, probably 2.1 or above.

$ qemu-img create -f qcow2 /tmp/overlay.qcow2 \
    -b 'json: { "file.driver":"https",
                "file.url":"https://..",
                "file.cookie":"vmware_soap_session=\"...\"",
                "file.sslverify":"off",
                "file.timeout":1000 }'

You don’t need to include the password in the URL here, since the cookie acts as your authentication. You might also want to play with the "file.readahead" parameter. We found it makes a big difference to throughput.

Now you can open the overlay file in guestfish as usual:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish
><fs> add /tmp/overlay.qcow2 copyonread:true
><fs> run
><fs> list-filesystems
/dev/sda1: ext4
><fs> mount /dev/sda1 /

and so on.

4 Comments

Filed under Uncategorized

New in libguestfs 1.27.34 – virt-v2v and virt-p2v

There haven’t been too many updates around here for a while, and that’s for a very good reason: I’ve been “heads down” writing the new versions of virt-v2v and virt-p2v, our tools for converting VMware and Xen virtual machines, or physical machines, to run on KVM.

The new virt-v2v [manual page] can slurp in a guest from a local disk image, local Xen, VMware vCenter, or (soon) an OVA file — convert it to run on KVM — and write it out to RHEV-M, OpenStack Glance, local libvirt or as a plain disk image.

It’s easy to use too. Unlike the old virt-v2v there are no hairy configuration files to edit or complicated preparations. You simply do:

$ virt-v2v -i disk xen_disk.img -o local -os /tmp

That command (which doesn’t need root, naturally) takes the Xen disk image, which could be any supported Windows or Enterprise Linux distro, converts it to run on KVM (eg. installing virtio drivers, adjusting dozens of configuration files), and writes it out to /tmp.

To connect to a VMware vCenter server, change the -i options to:

$ virt-v2v -ic vpx://vcenter/Datacenter/esxi "esx guest name" [-o ...]

To output the converted disk image to OpenStack glance, change the -o options to:

$ virt-v2v [-i ...] -o glance [-on glance_image_name]

Coming up: The new technology we’ve used to make virt-v2v much faster.

14 Comments

Filed under Uncategorized

Notes on getting VMware ESXi to run under KVM

This is mostly adapted from this long thread on the VMware community site.

I got VMware ESXi 5.5.0 running on upstream KVM today.

First I had to disable the “VMware backdoor”. When VMware runs, it detects that qemu underneath is emulating this port and tries to use it to query the machine (instead of using CPUID and so on). Unfortunately qemu’s emulation of the VMware backdoor is very half-assed. There’s no way to disable it except to patch qemu:

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index eaf3e61..ca1c422 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -204,7 +204,7 @@ static void pc_init1(QEMUMachineInitArgs *args,
     pc_vga_init(isa_bus, pci_enabled ? pci_bus : NULL);
 
     /* init basic PC hardware */
-    pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, xen_enabled(),
+    pc_basic_device_init(isa_bus, gsi, &rtc_state, &floppy, 1,
         0x4);
 
     pc_nic_init(isa_bus, pci_bus);

It would be nice if this was configurable in qemu. This is now being fixed upstream.

Secondly I had to turn off MSR emulation. This is, unfortunately, a machine-wide setting:

# echo 1 > /sys/module/kvm/parameters/ignore_msrs
# cat /sys/module/kvm/parameters/ignore_msrs
Y

Thirdly I had to give the ESXi virtual machine an IDE disk and an e1000 vmxnet3 network card. Note also that ESXi requires ≥ 2 vCPUs and at least 2 GB of RAM.

Screenshot - 190514 - 16:04:38

Screenshot - 190514 - 16:21:09

9 Comments

Filed under Uncategorized

Tip: Use libguestfs on VMware ESX guests

You can use libguestfs, guestfish and the virt tools on VMware ESX guests quite easily. However it’s not obvious how to do it, so this post explains that.

You will need:

  • libguestfs tools installed on a Linux machine
  • sshfs installed on the same Linux machine
  • ssh access to the VMware ESX storage (find the root password from the administrator)
  • the name of the guest and the name of the storage volume that the guest is stored on

The guest must be shut down (more on this later).

First of all, make sure you are able to ssh as root to the VMware ESX storage. It will look something like this:

$ ssh root@vmware
root@vmware's password: ****
Last login: Wed May  4 20:47:50 2011 from [...]
[root@vmware ~]# ls -l /vmfs/
total 1
drwxr-xr-x 1 root root 512 May 10 09:22 devices
drwxr-xr-x 1 root root 512 May 10 09:22 volumes

Now you should create a temporary mount point, and mount /vmfs from the VMware ESX storage server using sshfs. The command is quite simple and you don’t need to be root on the Linux side:

$ mkdir /tmp/vmfs
$ sshfs root@vmware:/vmfs /tmp/vmfs
root@vmware's password: ****
$

In another window you can navigate to the guest. For example if the guest was called “test” and it lived on volume “Storage1″ then:

$ cd /tmp/vmfs/volumes/Storage1/test
$ ls -l
total 1718720
-rw------- 1 root root 8589934592 May 10 09:48 test-flat.vmdk
-rw------- 1 root root       8684 May 10 09:37 test.nvram
-rw------- 1 root root        469 Apr  4 08:16 test.vmdk
-rw------- 1 root root          0 May 11  2010 test.vmsd
-rwxr-xr-x 1 root root       2666 May 10 09:37 test.vmx
-rw------- 1 root root        259 May 11  2010 test.vmxf
-rw-r--r-- 1 root root      53966 May 11  2010 vmware-1.log
-rw-r--r-- 1 root root      78771 May 11  2010 vmware-2.log
-rw-r--r-- 1 root root      56483 Apr  4 08:15 vmware-3.log
-rw-r--r-- 1 root root      56305 May 10 09:37 vmware.log

The critical file is guestname-flat.vmdk which is the flat disk image. You can just open this for read or write using guestfish, virt-df, virt-filesystems or other libguestfs tools or programs.

For example:

$ guestfish --rw -i -a test-flat.vmdk

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Red Hat Enterprise Linux Server release 5.5 (Tikanga)
/dev/VolGroup00/LogVol00 mounted on /
/dev/vda1 mounted on /boot

><fs> touch /tmp/hello
><fs> ll /tmp
total 20
drwxrwxrwt.  3 root root 4096 May 10 14:48 .
drwxr-xr-x. 24 root root 4096 May 10 14:36 ..
drwxrwxrwt   2 root root 4096 Apr  4 13:16 .ICE-unix
-rw-r--r--   1 root root    0 May 10 14:48 hello

Notice that guestfish determined the guest operating system and lets you edit the disk.

$ virt-filesystems -a test-flat.vmdk --all --long -h
Name                     Type       VFS  Label Size Parent
/dev/sda1                filesystem ext3 /boot 102M -
/dev/VolGroup00/LogVol00 filesystem ext3 -     7.1G -
/dev/VolGroup00/LogVol01 filesystem swap -     768M -
/dev/VolGroup00/LogVol00 lv         -    -     7.1G /dev/VolGroup00
/dev/VolGroup00/LogVol01 lv         -    -     768M /dev/VolGroup00
/dev/VolGroup00          vg         -    -     7.9G -
/dev/sda2                pv         -    -     7.9G -
/dev/sda1                partition  -    -     102M /dev/sda
/dev/sda2                partition  -    -     7.9G /dev/sda
/dev/sda                 device     -    -     8.0G -
$ virt-df -a test-flat.vmdk -h
Filesystem                                Size       Used  Available  Use%
test-flat.vmdk:/dev/sda1                   99M        12M        81M   13%
test-flat.vmdk:/dev/VolGroup00/LogVol00
                                          6.9G       1.1G       5.5G   16%

With libguestfs we usually allow you to read guests which are running. The results might be inconsistent at times, but it generally works. However VMware itself doesn’t allow running guests to be read. If the guest is running you can see that VMware prevents access:

# file test-flat.vmdk
test-flat.vmdk: writable, regular file, no read permission

Whereas when the same guest is shut down, reads (and writes) are allowed:

# file test-flat.vmdk
test-flat.vmdk: x86 boot sector; partition 1: ID=0x83, active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, starthead 0, startsector 208845, 16563015 sectors, code offset 0x48

This is a limitation of VMware and nothing to do with libguestfs.

A note on performance: I run this from my home to a VMware server which is a third of the way around the planet over plain 2Mbps ADSL. It’s noticeably slower than accessing local disk images, but still very usable. sshfs appears to be very efficiently implemented. It is far faster and more convenient than copying the whole disk image around.

6 Comments

Filed under Uncategorized

Examine VMWare ESX with libguestfs

We worked out yesterday (thanks Matthias Bolte) how to use the libguestfs tools like virt-inspector and guestfish to examine VMWare VMs.

VMWare’s native disk format is VMDK, which is only partially understood by free tools like qemu-img. qemu-img breaks quite badly on the newer variant that ESX 4 uses. Then there’s the issue of how you get the VMDK file from VMWare’s storage. Use their proprietary storage APIs?

Well it turned out both problems could be solved easily. VMWare ESX servers make the storage available over https connections, so you can use a URL like https://root:password@esxserver/folder/ to browse available storage on the server. And VMWare also makes the raw (“flat”) disk images available in the same way.

libvirt has supported ESX management for a while (thanks again to Matthias Bolte), so you can do:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 list --all
Enter username for 192.168.2.121 [root]:
Enter root password for 192.168.2.121:
 Id Name                 State
----------------------------------
  - TestLinux            shut off
  - TestWin              shut off

(Note that the domains must be shut off before VMWare will allow you to access the flat disk images).

Then we can get the storage URL:

$ sudo virsh -c esx://192.168.2.121/?no_verify=1 dumpxml TestLinux > /tmp/xml
$ grep '<source file' /tmp/xml
      <source file='[Storage1] TestLinux/TestLinux.vmdk'/>

And from that storage URL you can grab the disk image directly:

$ wget --no-check-certificate 'https://root:password@192.168.2.121/folder/TestLinux/TestLinux-flat.vmdk?dcPath=ha-datacenter&dsName=Storage1'

The large flat file downloaded is a straight raw disk image that can be examined directly in programs like guestfish and virt-inspector:

$ virt-list-filesystems -al TestLinux-flat.vmdk
/dev/sda1 ext4
/dev/vg_testlinux/lv_root ext4
/dev/vg_testlinux/lv_swap swap

$ guestfish --ro -a TestLinux-flat.vmdk -m /dev/vg_testlinux/lv_root

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ll /
total 116
dr-xr-xr-x.  23 root root  4096 Dec 30 06:27 .
dr-xr-xr-x   29 root root     0 Dec 21 07:59 ..
-rw-r--r--.   1 root root     0 Dec 30 06:27 .autofsck
drwx------.   3 root root  4096 Dec 17 11:58 .dbus
-rw-r--r--.   1 root root     0 Dec 17 12:50 .readahead_collect
dr-xr-xr-x.   2 root root  4096 Dec 17 12:11 bin
[etc]

It’s probably also possible to avoid the download step, since libguestfs is built on qemu which should support http(s) connections directly, but I didn’t try this yet.

3 Comments

Filed under Uncategorized