Tag Archives: curl

nbdkit + xz + curl

I’ve submitted a talk about nbdkit, our flexible, pluggable NBD server, to FOSDEM next year about how you can use nbdkit as a replacement for loopback mounts (or “loop mounts” as I was told off for not calling them last week). In preparation for that talk I ran through it in private to a small Red Hat audience on Monday. If I can I will release that video some time, but I may have to edit out Red Hat “super-secret” stuff first (or most likely not because there aren’t any secrets in it, but I’m still waiting for the internal video to be released).

Anyway this attracted a lot of interest and one question that was asked was why the xz plugin which lets you transparently open and uncompress XZ files on the fly was a plugin at all. Surely it would make more sense for it to be a filter? So it could be used not just to uncompress local files, but also xz-compressed cloud images over HTTPS.

The answer is yes it would! So I fixed it. XZ is now a filter (the plugin is left around but we’ll deprecate it eventually).

You can use it on top of the file plugin, curl plugin or other plugins:

$ nbdkit --filter=xz file file.xz
$ nbdkit --filter=xz curl https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz

This is fun and you can use this to boot the cloud image entirely remotely:

$ qemu-system-x86_64 -machine accel=kvm:tcg \
    -cpu host -m 2048 \
    -drive file=nbd:localhost:10809,if=virtio

However it’s incredibly slow. One problem is that the Fedora mirror sites aren’t very happy about you issuing lots of small HTTP Range requests and I observed that they throttle the connection quite aggressively. The second problem is that the xz block size for these cloud images is too large.

The XZ format (or rather, LZMA format) is divided into streams and blocks. We don’t normally use streams, and many XZ files use a single block. But it’s possible to tell the xz program to use a smaller than default block size, and in that case the output is divided into indexed blocks. Note the block size applies to the uncompressed input, the compressed blocks will have varying sizes, but the index that is created lets us find the block boundaries easily. When a byte is requested we can use a binary search to take us quickly to the compressed block, uncompress it (and cache it), and answer the request. We will only uncompress at most one block instead of the whole file.

For disk images I normally advocate a 16M block size. The current cloud images use (I think) a 192M block size, so both a huge amount of data has to be read over HTTPS to read one uncompressed byte, plus we have to cache very large blocks in RAM.

As an experiment I recompressed the cloud image using xz --block-size=$((16 * 1024 * 1024)) and hosted it locally, and booting is much quicker (albeit still slow because the cloud image contains cloud-init).

But even better we already ship a variety of disk images compressed with a 16M block size for virt-builder here, and these can be booted directly too:

$ nbdkit -U - --filter=xz curl \
        http://builder.libguestfs.org/fedora-29.xz \
        --run \
    'qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=$nbd,if=virtio'

… although you can’t log in because they all have locked root accounts (virt-builder normally customizes them after download).

Advertisements

1 Comment

Filed under Uncategorized

nbdkit now supports cURL — HTTP, FTP, and SSH connections

nbdkit is a liberally licensed NBD (Network Block Device) server designed to let you connect all sorts of crazy disk images sources (like Amazon, Glance, VMware VDDK) to the universal network protocol for sharing disk images: NBD.

New in nbdkit 1.1.8: cURL support. This lets you turn any HTTP, FTP, TFTP or SSH server that hosts a disk image into an NBD server.

For example:

$ nbdkit -r curl url=http://onuma/scratch/boot.iso

and then you can read the disk image using guestfish, qemu or any other nbd client:

$ guestfish --ro -a nbd://localhost -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

/dev/sda mounted on /

><fs> _

If you are using a normal SSH server like OpenSSH which supports the SSH File Transfer Protocol (aka SFTP), then you can use SFTP to access images:

$ nbdkit -r curl url=sftp://rjones@localhost/~/fedora-20.img

I’m hoping to enable write support in a future version.

It doesn’t work at the moment because I haven’t worked out how to switch between read (GET) and write (POST) requests in a single cURL handle. Perhaps I need to use two handles? The documentation is confusing.

2 Comments

Filed under Uncategorized

Using libguestfs to open an ISO on a public website

The new curl support added to libguestfs 1.22 lets you open any ISO remotely from a public web site or FTP server:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -i --format=raw \
    -a http://releases.ubuntu.com/precise/ubuntu-12.04.2-desktop-amd64.iso

Operating system: Ubuntu 12.04.2 LTS "Precise Pangolin" - Release amd64 (20130213)
/dev/sda1 mounted on /

><fs> ll /
total 2506
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 .
drwxr-xr-x 23 1000 1000    4096 May 28 13:55 ..
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 .disk
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 EFI
-r--r--r--  1 root root     236 Feb 13 22:21 README.diskdefines
-r--r--r--  1 root root     134 Feb 13 22:20 autorun.inf
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 boot
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 casper
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 dists
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 install
dr-xr-xr-x  1 root root   18432 Feb 13 22:21 isolinux
-r--r--r--  1 root root   16443 Feb 13 22:21 md5sum.txt
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 pics
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 pool
dr-xr-xr-x  1 root root    2048 Feb 13 22:21 preseed
lr-xr-xr-x  1 root root       1 Feb 13 22:21 ubuntu -> .
-r--r--r--  1 root root 2504624 Feb  8 22:58 wubi.exe

Of course it is slow as hell and not nice on the web host. It makes lots of byte-range requests on the host, downloading a few KB with each request, which is kind of the worst case for webservers to handle.

Note also that Fedora’s curl is broken. I compiled my own from upstream git.

4 Comments

Filed under Uncategorized

Using libguestfs over HTTP (and FTP)

New in libguestfs upstream and 1.21.39 is the ability to access disks over FTP, FTPS, HTTP, HTTPS and TFTP (read-only).

You can use it like this:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -a http://x.x.x.x/scratch/winxp.img -i

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: 'help' for help on commands
      'man' to read the manual
      'quit' to quit the shell

Operating system: Microsoft Windows XP
/dev/sda1 mounted on /

><fs> ll /
total 1573209
drwxrwxrwx  1 root root       4096 Apr 16  2012 .
drwxr-xr-x 23 1000 1000       4096 May 11 18:45 ..
-rwxrwxrwx  1 root root          0 Oct 11  2011 AUTOEXEC.BAT
-rwxrwxrwx  1 root root          0 Oct 11  2011 CONFIG.SYS
drwxrwxrwx  1 root root       4096 Oct 11  2011 Documents and Settings
-rwxrwxrwx  1 root root          0 Oct 11  2011 IO.SYS
-rwxrwxrwx  1 root root          0 Oct 11  2011 MSDOS.SYS
-rwxrwxrwx  1 root root      47564 Apr 14  2008 NTDETECT.COM
drwxrwxrwx  1 root root       4096 Oct 11  2011 Program Files
drwxrwxrwx  1 root root       4096 Oct 11  2011 System Volume Information
drwxrwxrwx  1 root root      28672 Oct 11  2011 WINDOWS
-rwxrwxrwx  1 root root        211 Oct 11  2011 boot.ini
-rwxrwxrwx  1 root root     250048 Apr 14  2008 ntldr
-rwxrwxrwx  1 root root 1610612736 Oct 11  2011 pagefile.sys

Apart from being a tiny bit slower, it just works as if the disk was local.

3 Comments

Filed under Uncategorized