Visualizing reads, writes and alignment

I wrote an out of tree patch to qemu that lets you gather read and write traces when a virtual machine accesses its virtual hard disk.

Using the patched qemu, a qemu wrapper, and some simple visualization tools we can look at how Linux executes individual filesystem operations.

Firstly we use the guestfish prepared disk feature to partition a disk and create an ext2 filesystem on the disk. The command under test is:

$ guestfish -N fs:ext2:10M

and this is what the disk access looks like (click to see the full size image):

The whole box represents the entire disk (10 MB), and each cell represents one sector (512 bytes in this case).

The large number of unaligned writes there points in fact to a mistake in the guestfs part-disk operation which is creating an unaligned whole disk partition. By adding a (non-upstream) patch to make this create an aligned partition, the results look a little better:

The second test is to take this disk image and simply mount it up. The command under test is:

$ guestfish -a test1.img -m /dev/sda1

and the access pattern looks like this:

As expected this is a mostly-read operation, but the act of mounting performs some writes to the ext2 superblock (shown in red).

Finally let’s see what it looks like to create a file on this empty filesystem:

$ guestfish -a test1.img -m /dev/sda1 \
  write /hello "hello, world."

That diagram includes the mount operation, so you have to mentally subtract that to see just the various file and metadata writes.

An obvious area of improvement here is to have libguestfs signal down to qemu to start and stop the trace, so that we can trace single operations (like just creating the file, not mounting the filesystem).

Another area of investigation is to add an LVM layer, and to experiment with the filesystem blocksize and other tunables.

It would also be good to start identifying various filesystem metadata areas by name, such as the inode table, block free bitmap and superblock.

About these ads

5 Comments

Filed under Uncategorized

5 responses to “Visualizing reads, writes and alignment

  1. Pingback: Visualizing #2: watching single operations « Richard WM Jones

  2. Pingback: Visualizing #3: write a file on an ext4 filesystem « Richard WM Jones

  3. frankiii

    These 3 entries are fascinating. I need to read them again. :)

    I’d love to see the performance impact…determining real benefits from playing with alignment is something I’ve struggled myself with.

  4. Rayson Ho

    Hi Richard,

    Interesting… I am going to do a SystemTap demo at a conference, and I am going redo this investigation with SystemTap.

    Without changing the KVM/QEMU source code, the following SystemTap script can give me similar output:

    #! /usr/bin/env stap

    probe process(“/home/rayson/s/qemu-kvm-0.13.0/bld/x86_64-softmmu/qemu-system-x86_64″).function(“bdrv_aio_readv”)
    {
    printf(“bdrv_aio_readv %d %d\n”, $sector_num, $nb_sectors)
    }

    probe process(“/home/rayson/s/qemu-kvm-0.13.0/bld/x86_64-softmmu/qemu-system-x86_64″).function(“bdrv_aio_writev”)
    {
    printf(“bdrv_aio_writev %d %d\n”, $sector_num, $nb_sectors)
    }

    Thanks,
    Rayson

    • rich

      I’ve been meaning to do something like this. If you make it print “R” and “W” there, and add the “S” header, then the changes are my script would already be able to parse it.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s