Tag Archives: virt-resize

New tool for making VMs sparse (aka thin provisioning)

New in libguestfs 1.13.17 is a tool for making virtual machines or disk images sparse, also known as thin provisioning.

This tool restores sparseness lost after you’ve created and deleted files. It works for virtually any guest OS, and supports almost any filesystem.

It can also do some clever things like finding sparseness in the unused parts of LVM physical volumes, or converting between formats (eg. raw to thin-provisioned qcow2).

It’s also (like virt-resize) a temporary tool: hopefully one day KVM and guests will be able to resize and trim themselves. That day isn’t here yet.

virt-sparsify manual page

Advertisements

2 Comments

Filed under Uncategorized

Arrrgghh writing GUI programs is tedious

Last week I thought it would be a good idea to write a small GUI front end to virt-resize.

After two days, I nearly have the first tab (out of four) working.

Granted, maybe the first tab is the hardest one:

The job of the first tab is to ask the user for the source (disk image or libvirt guest). It then fires off a background job to open that guest and inspect it for operating systems. Based on the outcome of that (opened OK, found OSes, no OSes found, etc) it has to update and enable the other tabs.

Also the user can preselect a guest on the command line. We also have to deal with connecting to libvirt asynchronously to read the list of guests (and this might also fail in a number of ways).

So far, 1600 lines of code, and the first tab is by no means complete.

One part of the problem is there’s a certain “impedance mismatch” between functional programming in OCaml and writing Gtk. Gtk is heavily mutable and object based. OCaml prefers (but does not require) immutability, and objects in OCaml are obscure and not widely used, and the Gtk bindings are written in a very hard-core object OCaml style.

Another part is just that it’s tedious. It would be tedious if I was doing this in C or Python too. You’ve got asynchronous actions going off here and there which update bits of state. Every control or input could potentially affect every other control or output, resulting in a kind of O(n2) mess of wiring up signals and callbacks.

Is there an easier way? I don’t know …

13 Comments

Filed under Uncategorized

virt-resize machine readable output

As I planned earlier I have added machine readable output to virt-resize.

One huge frustration we have had with qemu for a long time is that you cannot query a qemu binary for its “capabilities” (eg. does this binary support virtio? does this binary support particular -drive options?).

virt-resize now supports this kind of querying, thus with any virt-resize ≥ 1.12.5 you can do:

$ virt-resize --machine-readable
virt-resize
ntfsresize-force
32bitok
ntfs
btrfs

The simple list of facts is designed to be easy to parse. Does this virt-resize work on 32-bit platforms (old versions written in Perl didn’t)? Answer: yes because of the 32bitok fact. Does this binary support btrfs guests? In this case yes.

Over time as we add more features to virt-resize we will add facts to this list, so that consumers of virt-resize can tell what the binary supports.

This is a simple plan. Qemu developers, take note!

The second thing that this option does is to make the output of virt-resize more regular and easier to consume. At the moment this means that progress bars in virt-resize will revert to a simple format:

$ virt-resize --machine-readable old.img new.img
...
0/100
1/100
2/100
...

so that programs can just match on ^\d+/\d+$ on stdout, and turn that into a progress bar. User messages go to stdout and errors to stderr. The output is designed to be both very lightweight to parse and easy to understand for humans (and in fact not that much different from the regular virt-resize output). We may make small enhancements in future.

Leave a comment

Filed under Uncategorized

Half-baked idea: Standard machine readable output for command line programs

For more half-baked ideas, see the ideas tag.

It’s common, perhaps increasingly common, that one program needs to consume the output of another. This is the Unix philosophy — small, single purpose programs assembled together to carry out a more complex task.

However it’s not necessarily superior to alternative ways of composing programs, like COM or D-Bus.

There are two particular problems: 1. How do you find out if a particular feature is supported by the program? 2. How do you parse the output of the program (eg. to find progress bars or error messages)?

As a concrete example, let’s consider a program I wrote called virt-resize. 1. How do I find out if the version of virt-resize I have supports btrfs? 2. If I want to drive virt-resize from a graphical program, how can I parse the text progress bars that virt-resize prints?

For question 1, typical Unix programs take several approaches:

  1. Ignore the problem completely. Just blindly use the feature, and fail in unpredictable ways if it doesn’t work. This is probably the most popular “strategy”. People who write shell scripts tend to do this all the time. Shell scripts are often either unportable, or end up looking like “configure” because they try to use a very conservative subset of POSIX.
  2. Run the program, if it fails, run the program a bit differently (and if it fails, a bit differently again, …).
  3. Attempt to parse the output of program --help. This depends on help output being stable, when maybe it isn’t, so you end up chasing the upstream project.
  4. Parse program --version and work out if the feature was supported in that particular version. This is not very scalable, and doesn’t work with backports.

Question 2, how to get errors and progress bars, is usually too hard, unless the program offers a special “machine readable” flag (notable examples: rpm, parted).

Here’s my half-baked idea: We should standardize the way that program capabilities, help, progress bars, and error output is done.

An additional option is added to programs that support this:

$ program --machine-readable [...]

Programs that don’t support this, and programs that didn’t support it in earlier versions, ought to give an error if this option is not available.

Firstly, the caller just runs the program with this option on its own, and no other options:

$ program --machine-readable
resize-btrfs
resize-ext3
lv-expand
progress-bars

The program just prints out a list of capabilities, one per line, but with no defined format (that is a contract between the program and the caller). The program then exits with status 0. Using this option should not cause the program to perform any other action.

Secondly, if this was successful, the caller can use this option in combination with other options to produce machine-readable output. At least one other option or command line argument is required for this to work.

I would like to suggest the following standards for version numbers, progress bars and error messages.

Machine-readable version numbers are sent to stdout and have the form “program-name version”, where “program-name” should be one word. This is no different from how most GNU programs work:

$ program --version
program 0.1.2

Machine-readable progress bars are sent to stdout and have the form (example) “[10/100]”:

$ program --machine-readable foo
[0/100]
[1/100]
[2/100]
etc.

Error messages are anything sent to stderr when the status code of the program is non-zero. This is, of course, no change from standard Unix.

$ program --machine-readable foo
foo: File not found

24 Comments

Filed under Uncategorized

virt-resize now works with btrfs

virt-resize now works with guests that use btrfs:

$ virt-resize /dev/vg_pin/F15BTRFSx64 /dev/vg_pin/TmpBTRFS --expand sda3
Examining /dev/vg_pin/F15BTRFSx64 ...
[########################################################################]
**********

Summary of changes:

/dev/sda1: This partition will be left alone.

/dev/sda2: This partition will be left alone.

/dev/sda3: This partition will be resized from 9.0G to 14.0G.  The 
    filesystem btrfs on /dev/sda3 will be expanded using the 
    'btrfs-filesystem-resize' method.

**********
Setting up initial partition table on /dev/vg_pin/TmpBTRFS ...
Copying /dev/sda1 ...
[########################################################################]
Copying /dev/sda2 ...
[########################################################################]
Copying /dev/sda3 ...
[########################################################################]
[########################################################################]
Expanding /dev/sda3 using the 'btrfs-filesystem-resize' method ...

Resize operation completed with no errors.  Before deleting the old 
disk, carefully check that the resized disk boots and works correctly.

Leave a comment

Filed under Uncategorized

Reminder: Summit talk on libguestfs, virt-v2v

Anyone who’s coming to the Red Hat Summit next week (Boston USA, May 3-6), please remember to drop in at our talk on libguestfs, the virt tools, virt-v2v and virt-p2v. It’s on Wednesday morning at 11.30am.

I will be live demonstrating guestfish, virt-resize and more. Matt will be showing us virt-v2v and the new version of virt-p2v that he has written.

To head off the expected question: Will the slides be available after the talk? Mine won’t because I don’t really “do” slides. However there will be a handout available (either on this blog or on the summit web pages or both) which covers all the background material and contains additional worked examples and exercises. We are hoping the talk will also be videoed like last year.

Leave a comment

Filed under Uncategorized

Tip: Expand a Windows virtual machine

(You will need virt-resize ≥ 1.4 for this)

My Windows VM is stored in a logical volume on the host. It is 16GB currently, and for the purposes of this example I will expand that to 18GB. virt-resize doesn’t touch the old disk image, so I have to rename the old one and create a new one like this:

# lvrename /dev/vg_pin/Win7x32 /dev/vg_pin/Win7x32old
  Renamed "Win7x32" to "Win7x32old" in volume group "vg_pin"
# lvcreate -n Win7x32 -L 18G /dev/vg_pin
  Logical volume "Win7x32" created

Let’s have a look at how the Windows VM is organized at the moment. Some versions of Windows put the C: drive on the first partition, but recent versions use a small boot partition and have the C: drive on the second partition. virt-resize is only able to resize the C: drive (but for Linux VMs virt-resize can resize any partition including the boot partition):

# virt-list-filesystems -al /dev/vg_pin/Win7x32old
/dev/sda1 ntfs                <-- boot partition
/dev/sda2 ntfs                <-- C: drive

And now we simply do the resize. In this case I’m going to ask virt-resize to expand the C: drive (/dev/sda2). Other choices would be to ask virt-resize to create an extra partition at the end of the disk or to leave the extra space unpartitioned.

# virt-resize --expand /dev/sda2 /dev/vg_pin/Win7x32old /dev/vg_pin/Win7x32
Summary of changes:
/dev/sda1: partition will be left alone
/dev/sda2: partition will be resized from 15.9G to 17.9G
/dev/sda2: content will be expanded using the 'ntfsresize' method
Copying /dev/sda1 ...
[############################################################################]
Copying /dev/sda2 ...
[############################################################################]
Expanding /dev/sda2 using the 'ntfsresize' method

Finally, check that the new copy of Windows boots! (By the way, Windows will do a chkdsk the first time it boots. This is deliberate. We tell Windows to do this so we can be sure that we haven’t damaged any of the data during the copy or resize).

If it boots, you can delete the old copy of Windows:

# lvchange -an /dev/vg_pin/Win7x32old 
# lvremove /dev/vg_pin/Win7x32old 
  Logical volume "Win7x32old" successfully removed

If it doesn’t boot, please file a bug giving all the details, then revert back to the old copy.

Leave a comment

Filed under Uncategorized