Tag Archives: nbdkit

Video: Take your loop mounts to the next level with nbdkit

Loop mounting is popular, but very limited in what it can do on Linux. I gave a talk at FOSDEM on Saturday entitled Better loop mounts with NBD: Take your loop mounts to the next level with nbdkit, and it’s online already!

Download the WebM format or MP4 format files directly.

Also I did subtitles! Download the subtitles directly here. (The subs only cover the first 30 minutes of the talk, not the Summary and Q&A.)

Photo from presentation
(Thanks to Thomas Huth for the photo)

There are a few small problems and corrections:

  1. There’s a part of the talk where I refer to the light blue trimmed blocks. Unfortunately the video feed didn’t capture that, so the light blue looks like white. If you really want to see that then go look at the video in my earlier post.
  2. During the Q&A I mentioned that we could support writing to xz files. This is true, sort of, but I forgot that there’s a problem: nbdkit doesn’t support file resizing (and I believe that’s even experimental in the NBD protocol), so someone would have to add that first. There are other serious down-sides to implementing writable XZ, I doubt it could ever be fast.

For subtitles I used gaupol which is actually quite nice, although subtitling is inherently slow and tedious. It took me a good 4 hours to subtitle 30 minutes of video.

Advertisements

4 Comments

Filed under Uncategorized

Interview with FOSDEM team about nbdkit loop mounting

I did an email interview which you can read online about my FOSDEM talk about nbdkit loop mounting.

Other FOSDEM 2019 interviews are here.

Leave a comment

Filed under Uncategorized

FOSDEM 2019 – Better loop mounting with NBD

My talk was accepted: https://fosdem.org/2019/schedule/event/nbdkit/

If you’re coming to FOSDEM, please come and say hello. In the meantime if you want to watch a rough early run-through of the talk, see: https://rwmj.wordpress.com/2018/11/26/nbdkit-fosdem-test-presentation/

1 Comment

Filed under Uncategorized

Haiku!

haiku

Looks great, and nbdkit compiles out of the box.

Leave a comment

Filed under Uncategorized

nbdkit inline scripts

I have proposed a patch for nbdkit, our flexible, pluggable Network Block Device server, to make writing Linux block devices into a (long) single command.

Here’s a simple block device with virtual size 1M that reads as zeroes:

nbdkit sh - <<'EOF'
    case "$1" in
        get_size) echo 1M ;;
        pread) dd if=/dev/zero count=$3 iflag=count_bytes ;;
        *) exit 2 ;;
    esac
EOF

Leave a comment

Filed under Uncategorized

nbdkit / FOSDEM test presentation about better loop mounts for Linux

I’ve submitted a talk about nbdkit, our flexible pluggable NBD server, to FOSDEM next February. This is going to be about using NBD as a better way to do loop mounts in Linux.

In preparation I gave a very early version of the talk to a small Red Hat audience.

Video link: http://oirase.annexia.org/rwmj.wp.com/rjones-nbdkit-tech-talk-2018-11-19.mp4

Sorry about the slow start. You may want to skip to 2 mins to get past the intro.

Summary of what’s in the talk:

  1. Demo of regular, plain loop mounting.
  2. Demo of loop mounting an XZ-compressed disk image using NBD + nbdkit.
  3. Slides about how loop device compares to NBD.
  4. Slides about nbdkit plugins and filters.
  5. Using VMware VDDK to access a VMDK file.
  6. Creating a giant disk costing EUR 300 million(!)
  7. Visualizing a single filesystem.
  8. Visualizing RAID 5.
  9. Writing a plugin in shell script (live demo).
  10. Summary.

Screenshot_2018-11-26_17-18-16

2 Comments

Filed under Uncategorized

nbdkit + xz + curl

I’ve submitted a talk about nbdkit, our flexible, pluggable NBD server, to FOSDEM next year about how you can use nbdkit as a replacement for loopback mounts (or “loop mounts” as I was told off for not calling them last week). In preparation for that talk I ran through it in private to a small Red Hat audience on Monday. If I can I will release that video some time, but I may have to edit out Red Hat “super-secret” stuff first (or most likely not because there aren’t any secrets in it, but I’m still waiting for the internal video to be released).

Anyway this attracted a lot of interest and one question that was asked was why the xz plugin which lets you transparently open and uncompress XZ files on the fly was a plugin at all. Surely it would make more sense for it to be a filter? So it could be used not just to uncompress local files, but also xz-compressed cloud images over HTTPS.

The answer is yes it would! So I fixed it. XZ is now a filter (the plugin is left around but we’ll deprecate it eventually).

You can use it on top of the file plugin, curl plugin or other plugins:

$ nbdkit --filter=xz file file.xz
$ nbdkit --filter=xz curl https://download.fedoraproject.org/pub/fedora/linux/releases/29/Cloud/x86_64/images/Fedora-Cloud-Base-29-1.2.x86_64.raw.xz

This is fun and you can use this to boot the cloud image entirely remotely:

$ qemu-system-x86_64 -machine accel=kvm:tcg \
    -cpu host -m 2048 \
    -drive file=nbd:localhost:10809,if=virtio

However it’s incredibly slow. One problem is that the Fedora mirror sites aren’t very happy about you issuing lots of small HTTP Range requests and I observed that they throttle the connection quite aggressively. The second problem is that the xz block size for these cloud images is too large.

The XZ format (or rather, LZMA format) is divided into streams and blocks. We don’t normally use streams, and many XZ files use a single block. But it’s possible to tell the xz program to use a smaller than default block size, and in that case the output is divided into indexed blocks. Note the block size applies to the uncompressed input, the compressed blocks will have varying sizes, but the index that is created lets us find the block boundaries easily. When a byte is requested we can use a binary search to take us quickly to the compressed block, uncompress it (and cache it), and answer the request. We will only uncompress at most one block instead of the whole file.

For disk images I normally advocate a 16M block size. The current cloud images use (I think) a 192M block size, so both a huge amount of data has to be read over HTTPS to read one uncompressed byte, plus we have to cache very large blocks in RAM.

As an experiment I recompressed the cloud image using xz --block-size=$((16 * 1024 * 1024)) and hosted it locally, and booting is much quicker (albeit still slow because the cloud image contains cloud-init).

But even better we already ship a variety of disk images compressed with a 16M block size for virt-builder here, and these can be booted directly too:

$ nbdkit -U - --filter=xz curl \
        http://builder.libguestfs.org/fedora-29.xz \
        --run \
    'qemu-system-x86_64 -machine accel=kvm:tcg -cpu host -m 2048 -drive file=$nbd,if=virtio'

… although you can’t log in because they all have locked root accounts (virt-builder normally customizes them after download).

1 Comment

Filed under Uncategorized