Tag Archives: loop mounting

Loop mount an S3 or Ceph object

This is a fun, small nbdkit Python plugin using the Boto3 AWS SDK:

#!/usr/sbin/nbdkit python

import nbdkit
import boto3
from contextlib import closing

API_VERSION = 2

def thread_model():
    return nbdkit.THREAD_MODEL_PARALLEL

def config(key, value):
    global access_key, secret_key, endpoint_url, bucket_name, key_name

    if key == "access-key" or key == "access_key":
        access_key = value
    elif key == "secret-key" or key == "secret_key":
        secret_key = value
    elif key == "endpoint-url" or key == "endpoint_url":
        endpoint_url = value
    elif key == "bucket":
        bucket_name = value
    elif key == "key":
        key_name = value
    else:
        raise Exception("unknown parameter %s" % key)

def open(readonly):
    global access_key, secret_key, endpoint_url

    s3 = boto3.client("s3",
                      aws_access_key_id = access_key,
                      aws_secret_access_key = secret_key,
                      endpoint_url = endpoint_url)
    if s3 is None:
        raise Exception("could not connect to S3")
    return s3

def get_size(s3):
    global bucket_name, key_name

    resp = s3.get_object(Bucket = bucket_name, Key = key_name)
    size = resp['ResponseMetadata']['HTTPHeaders']['content-length']
    return int(size)

def pread(s3, buf, offset, flags):
    global bucket_name, key_name

    size = len(buf)
    rnge = 'bytes=%d-%d' % (offset, offset+size-1)
    resp = s3.get_object(Bucket = bucket_name, Key = key_name, Range = rnge)
    body = resp['Body']
    with closing(body):
        buf[:] = body.read(size)

This lets you loop mount a single object (file):

$ ./nbdkit-S3-plugin -f -v -U /tmp/sock \
  access_key="XYZ" secret_key="XYZ" \
  bucket="my_files" key="fedora-28.iso"
$ sudo nbd-client -b 2048 -unix /tmp/sock /dev/nbd0
Negotiation: ..size = 583MB
$ ls /dev/nbd0
 nbd0    nbd0p1  nbd0p2  
$ sudo mount -o ro /dev/nbd0p1 /tmp/mnt
$ ls -l /tmp/mnt
 total 11
 dr-xr-xr-x. 3 root root 2048 Apr 25  2018 EFI
 -rw-r--r--. 1 root root 2532 Apr 23  2018 Fedora-Legal-README.txt
 dr-xr-xr-x. 3 root root 2048 Apr 25  2018 images
 drwxrwxr-x. 2 root root 2048 Apr 25  2018 isolinux
 -rw-r--r--. 1 root root 1063 Apr 21  2018 LICENSE
 -r--r--r--. 1 root root  454 Apr 25  2018 TRANS.TBL

I should note this is a bit different from s3fs which is a FUSE driver that mounts all the files in a bucket.

Leave a comment

Filed under Uncategorized

Video: Take your loop mounts to the next level with nbdkit

Loop mounting is popular, but very limited in what it can do on Linux. I gave a talk at FOSDEM on Saturday entitled Better loop mounts with NBD: Take your loop mounts to the next level with nbdkit, and it’s online already!

Download the WebM format or MP4 format files directly.

Also I did subtitles! Download the subtitles directly here. (The subs only cover the first 30 minutes of the talk, not the Summary and Q&A.)

Photo from presentation
(Thanks to Thomas Huth for the photo)

There are a few small problems and corrections:

  1. There’s a part of the talk where I refer to the light blue trimmed blocks. Unfortunately the video feed didn’t capture that, so the light blue looks like white. If you really want to see that then go look at the video in my earlier post.
  2. During the Q&A I mentioned that we could support writing to xz files. This is true, sort of, but I forgot that there’s a problem: nbdkit doesn’t support file resizing (and I believe that’s even experimental in the NBD protocol), so someone would have to add that first. There are other serious down-sides to implementing writable XZ, I doubt it could ever be fast.

For subtitles I used gaupol which is actually quite nice, although subtitling is inherently slow and tedious. It took me a good 4 hours to subtitle 30 minutes of video.

7 Comments

Filed under Uncategorized

FOSDEM 2019 – Better loop mounting with NBD

My talk was accepted: https://fosdem.org/2019/schedule/event/nbdkit/

If you’re coming to FOSDEM, please come and say hello. In the meantime if you want to watch a rough early run-through of the talk, see: https://rwmj.wordpress.com/2018/11/26/nbdkit-fosdem-test-presentation/

1 Comment

Filed under Uncategorized