Benchmarks: uploading files

I spent a few hours working out the fastest way to upload a large file into a disk image (using libguestfs, but the results are applicable to ordinary virtual machines too).

The timings were done on an idle physical machine with plenty of free RAM. The disk image was located in /dev/shm (ie. shared memory) in order to remove the physical effects of spinning magnetic media, so what I hope we are measuring here is pure software overhead. The test script is down at the bottom of this posting in case you want to try to reproduce the results.

In all cases, a 1500M file of zeroes was being uploaded into a 2G disk image. All tests were repeated 5 times, with the last 3 times averaged and shown to the nearest ½ second.

As a baseline I used the libguestfs upload method. This works by copying from a host file, over a fast virtio-serial connection, into the virtual machine. The disk image was raw-format and sparse. The host was synched before and after the upload to ensure that all writes go through to the backing disk (in shared memory).

raw sparse, upload: 8 seconds, 188 MBps

Preallocating the disk made things slower:

raw prealloc, upload: 8.5 s, 176 MBps, -6%

(Preallocation also made no difference when I repeated the test with a real hard disk as backing).

Instead of using the upload command, we can attach the source file as a disk and copy it directly using dd if=/dev/vdb of=/big.file bs=1024k. This was very slightly faster than the baseline:

raw prealloc, dd: 7.5, 200, +6%

However when I tested this again using a real disk for backing, dd made no measurable difference over upload.

Using qcow2 as a backing disk instead of raw made little difference:

qcow2 no prealloc, upload: 7.5, 200, +6%
qcow2 preallocation=metadata, upload: 8, 188, -
qcow2 preallocation=metadata, dd: 8, 188, -

Until very recently, libguestfs defaulted to mounting disks with -o sync, a historical mistake in the API. Although I am doing these tests without this option, adding it shows how much of a penalty this causes:

raw prealloc, dd, sync: 12, 125, -50%
raw prealloc, upload, sync: 138, 11, -1625%

The guest filesystem was ext2 which doesn’t have a journal. What is the penalty for using ext3 (using the default journaling options)?

raw prealloc, dd, ext3: 10.5, 143, -31%

This is surprising because I wouldn’t expect that journal writes while creating a file would be that significant.

Finally, if we compress the input file (down to 1.5MB), surely there will be less data being pushed over the virtio-serial channel and everything will be quicker? In fact, no it’s slower, even if we enable all the virtual CPUs in the guest:

raw prealloc, upload tar.gz: 10.5, 143, -31%
raw prealloc, upload tar.gz, -smp 4: 10.5, 143, -31%

Does it matter that the file I was uploading was all zero bytes? Does qemu optimize this case? I tested this using the fill function to generate files containing other bytes, and as far as I can tell, neither qcow2 nor the kernel (for raw) optimizes the all-zero-byte case.

My conclusion is that intuition is not a very good guide. Measure it!


#!/bin/bash -

set -e

# Configuration.
guest_format=ext2
#disk_image=test1.img
disk_image=/dev/shm/test1.img
disk_image_size=2G
mount_options=
#mount_options=sync

# Create big file to upload.
# Only do this once so the file is cached.

#rm -f big.file big.file.tar.gz
#truncate -s 1G big.file
#tar zcf big.file.tar.gz big.file

test() {
    # Choose a method to allocate the image.
    rm -f times $disk_image

    qemu-img create -f qcow2 $disk_image $disk_image_size >/dev/null
    #qemu-img create -f qcow2 -o preallocation=metadata $disk_image $disk_image_size >/dev/null
    #truncate -s $disk_image_size $disk_image
    #fallocate -l $disk_image_size $disk_image
    # or: dd if=/dev/zero of=$disk_image bs=1024k count=2048 > /dev/null 2>&1

    # Perform the test.

    guestfish -a $disk_image <<EOF

    #smp 4

    # For dd test, add file as /dev/sdb
    #add-ro big.file

    run

    part-disk /dev/sda mbr
    mkfs $guest_format /dev/sda1
    mount-options "$mount_options" /dev/sda1 /

    # Sync before test.
    sync
    !sync

    !date +%s.%N >> times

    #upload big.file /big.file
    #dd /dev/sdb /big.file
    #tgz-in big.file.tar.gz /
    #fill 1 1500M /big.file
    #fill 0 1500M /big.file

EOF

    # Ensure all data is written to disk.
    sync

    date +%s.%N >> times

    # Display time taken in seconds.
    awk '{ if (!start) { start = $1 } else { print $1-start } }' < times
}

test
test
test
test
test

rm -f $disk_image

Leave a comment

Filed under Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s