Good set of videos about Open vSwitch, OpenFlow, SDN

https://www.youtube.com/channel/UCEoaojfEY_6L5TWWjIn9t9Q

Leave a comment

Filed under Uncategorized

AMD Huskyboard (video)

This board has the AMD “Seattle” A1100 64 bit, 8 core ARM.

Leave a comment

Filed under Uncategorized

Half-baked ideas: C strings with implicit length field

For more half-baked ideas, see the ideas tag.

If you prefer just to see the code, then it’s here.

Chris Siebenmann wrote a couple of interesting articles about C’s null terminated strings and how they pre-date C.

Chris notes an alternative is a length + string representation, as used in Pascal. Although there are libraries for this in C, there are several drawbacks and approximately no one uses them.

However it’s possible to have the best of both worlds: Strings using an implicit length field that takes up no extra storage. These strings are backwards compatible with ordinary C strings — you can literally pass them to legacy functions or cast them to char * — yet the equivalent of a strlen operation is O(1).

There are two ideas here: Firstly, when you use the C malloc function, malloc stashes some extra metadata about your allocation, and with most malloc implementations there is a function to obtain the size of the allocation from a pointer. In glibc, the function is called malloc_usable_size. Note that because of alignment concerns, the amount allocated is usually larger than the amount you originally requested.

The second idea comes from OCaml. OCaml stores strings in a clever internal representation which is both backwards compatible with C (a fancy way to say they are null terminated), and it allows you to get the real length of the string even though OCaml — like C — allocates more than requested for alignment reasons.

So here’s how we do it: When allocating an “implicit length string” (ilenstr) we store extra data in the final byte of the “full” malloced space, in the byte marked B in the diagram below:

+-------------------------+----+------------+----+
| the string              | \0 |   ....     | B  |
+-------------------------+----+------------+----+
<----- malloc we requested ---->
<----------- malloc actually allocated ---------->

If malloc allocated exactly the same amount of space as is used by our string + terminating null, then B is simply the terminating \0:

+-------------------------+----+
| the string              | \0 |
+-------------------------+----+

If malloc allocated 1 spare byte, we store B = 1:

+-------------------------+----+----+
| the string              | \0 | 1  |
+-------------------------+----+----+

If malloc allocated 4 spare bytes, we store B = 4:

+-------------------------+----+----+----+----+----+
| the string              | \0 |   ....       | 4  |
+-------------------------+----+----+----+----+----+

Getting the true length of the string is simply a matter of asking malloc for the allocated length (ie. calling malloc_usable_size), finding the last byte (B) and subtracting it. So we can get the true string length in an O(1) operation (usually, although this may depend on your malloc implementation).

ilenstr strings can contain \0 characters within the string.

ilenstr strings are also backwards compatible, in that we can pass one to any “legacy” C function, and assuming the string itself doesn’t contain any \0 inside it, everything just works.

Alright. This is terrible. DO NOT USE IT IN PRODUCTION CODE! It breaks all kinds of standards, is unportable etc. There are security issues with allowing \0-containing strings to be passed to legacy functions. Still, it’s a nice idea. With proper cooperation from libc, standards authorities and so on, it could be made to work.

Here is my git repo:

http://git.annexia.org/?p=ilenstr.git;a=summary

Leave a comment

Filed under Uncategorized

CentOS 7.2, with added virt-v2v

CentOS 7.2 is out. New in this release, you can simply install virt-v2v and all the dependencies (including libguestfs-winsupport!) are included*.

* = except for libguestfs-xfs which you might also need for converting RHEL 7 guests.

# yum install virt-v2v
Dependencies Resolved

================================================================================
 Package                         Arch   Version                   Repository
                                                                           Size
================================================================================
Installing:
 virt-v2v                        x86_64 1:1.28.1-1.55.el7.centos  base     12 M
Installing for dependencies:
 attr                            x86_64 2.4.46-12.el7             base     66 k
 augeas-libs                     x86_64 1.4.0-2.el7               base    355 k
 boost-system                    x86_64 1.53.0-25.el7             base     39 k
 boost-thread                    x86_64 1.53.0-25.el7             base     57 k
 bridge-utils                    x86_64 1.5-9.el7                 base     32 k
 bzip2                           x86_64 1.0.6-13.el7              base     52 k
 celt051                         x86_64 0.5.1.3-8.el7             base     53 k
 cryptsetup                      x86_64 1.6.7-1.el7               base    119 k
 cyrus-sasl                      x86_64 2.1.26-19.2.el7           base     88 k
 cyrus-sasl-md5                  x86_64 2.1.26-19.2.el7           base     56 k
 device-mapper-event             x86_64 7:1.02.107-5.el7          base    167 k
 device-mapper-event-libs        x86_64 7:1.02.107-5.el7          base    169 k
 device-mapper-persistent-data   x86_64 0.5.5-1.el7               base    350 k
 dosfstools                      x86_64 3.0.20-9.el7              base    101 k
 flac-libs                       x86_64 1.3.0-5.el7_1             base    169 k
 fuse                            x86_64 2.9.2-6.el7               base     84 k
 fuse-libs                       x86_64 2.9.2-6.el7               base     93 k
 gdisk                           x86_64 0.8.6-5.el7               base    187 k
 genisoimage                     x86_64 1.1.11-23.el7             base    298 k
 glusterfs                       x86_64 3.7.1-16.el7              base    422 k
 glusterfs-api                   x86_64 3.7.1-16.el7              base     60 k
 glusterfs-client-xlators        x86_64 3.7.1-16.el7              base    732 k
 glusterfs-libs                  x86_64 3.7.1-16.el7              base    321 k
 gnutls-dane                     x86_64 3.3.8-12.el7_1.1          base     32 k
 gnutls-utils                    x86_64 3.3.8-12.el7_1.1          base    228 k
 gperftools-libs                 x86_64 2.4-7.el7                 base    272 k
 gsm                             x86_64 1.0.13-11.el7             base     30 k
 gtk2                            x86_64 2.24.28-8.el7             base    3.4 M
 hexedit                         x86_64 1.2.13-5.el7              base     39 k
 hivex                           x86_64 1.3.10-5.7.el7            base    100 k
 icoutils                        x86_64 0.31.0-3.el7              base     82 k
 ipxe-roms-qemu                  noarch 20130517-7.gitc4bce43.el7 base    559 k
 iscsi-initiator-utils           x86_64 6.2.0.873-32.el7          base    418 k
 iscsi-initiator-utils-iscsiuio  x86_64 6.2.0.873-32.el7          base     81 k
 ldns                            x86_64 1.6.16-7.el7              base    473 k
 libaio                          x86_64 0.3.109-13.el7            base     24 k
 libasyncns                      x86_64 0.8-7.el7                 base     26 k
 libcgroup                       x86_64 0.41-8.el7                base     64 k
 libconfig                       x86_64 1.4.9-5.el7               base     59 k
 libguestfs                      x86_64 1:1.28.1-1.55.el7.centos  base    1.7 M
 libguestfs-tools-c              x86_64 1:1.28.1-1.55.el7.centos  base    2.2 M
 libguestfs-winsupport           x86_64 7.2-1.el7                 base    2.1 M
 libibverbs                      x86_64 1.1.8-8.el7               base     56 k
 libicu                          x86_64 50.1.2-15.el7             base    6.9 M
 libiscsi                        x86_64 1.9.0-6.el7               base     60 k
 libogg                          x86_64 2:1.3.0-7.el7             base     24 k
 libosinfo                       x86_64 0.2.12-3.el7              base    256 k
 librados2                       x86_64 1:0.80.7-3.el7            base    1.5 M
 librbd1                         x86_64 1:0.80.7-3.el7            base    350 k
 librdmacm                       x86_64 1.0.21-1.el7              base     64 k
 libreport-filesystem            x86_64 2.1.11-32.el7.centos      base     37 k
 libseccomp                      x86_64 2.2.1-1.el7               base     49 k
 libsndfile                      x86_64 1.0.25-10.el7             base    149 k
 libunwind                       x86_64 1.1-5.el7                 base     56 k
 libusal                         x86_64 1.1.11-23.el7             base    135 k
 libvirt-client                  x86_64 1.2.17-13.el7_2.2         updates 4.3 M
 libvirt-daemon                  x86_64 1.2.17-13.el7_2.2         updates 584 k
 libvirt-daemon-driver-interface x86_64 1.2.17-13.el7_2.2         updates 161 k
 libvirt-daemon-driver-network   x86_64 1.2.17-13.el7_2.2         updates 301 k
 libvirt-daemon-driver-nodedev   x86_64 1.2.17-13.el7_2.2         updates 160 k
 libvirt-daemon-driver-nwfilter  x86_64 1.2.17-13.el7_2.2         updates 184 k
 libvirt-daemon-driver-qemu      x86_64 1.2.17-13.el7_2.2         updates 569 k
 libvirt-daemon-driver-secret    x86_64 1.2.17-13.el7_2.2         updates 154 k
 libvirt-daemon-driver-storage   x86_64 1.2.17-13.el7_2.2         updates 327 k
 libvirt-daemon-kvm              x86_64 1.2.17-13.el7_2.2         updates 117 k
 libvorbis                       x86_64 1:1.3.3-8.el7             base    204 k
 libxslt                         x86_64 1.1.28-5.el7              base    242 k
 lsof                            x86_64 4.87-4.el7                base    331 k
 lvm2                            x86_64 7:2.02.130-5.el7          base    1.0 M
 lvm2-libs                       x86_64 7:2.02.130-5.el7          base    872 k
 lzop                            x86_64 1.03-10.el7               base     54 k
 mailcap                         noarch 2.1.41-2.el7              base     31 k
 make                            x86_64 1:3.82-21.el7             base    420 k
 mdadm                           x86_64 3.3.2-7.el7               base    393 k
 mtools                          x86_64 4.0.18-5.el7              base    203 k
 netcf-libs                      x86_64 0.2.8-1.el7               base     69 k
 netpbm                          x86_64 10.61.02-9.el7            base    180 k
 netpbm-progs                    x86_64 10.61.02-9.el7            base    1.9 M
 nmap-ncat                       x86_64 2:6.40-7.el7              base    201 k
 numad                           x86_64 0.5-14.20140620git.el7    base     34 k
 openssl                         x86_64 1:1.0.1e-51.el7_2.1       updates 711 k
 perl-Business-ISBN              noarch 2.06-2.el7                base     25 k
 perl-Business-ISBN-Data         noarch 20120719.001-2.el7        base     24 k
 perl-Compress-Raw-Bzip2         x86_64 2.061-3.el7               base     32 k
 perl-Compress-Raw-Zlib          x86_64 1:2.061-4.el7             base     57 k
 perl-Digest                     noarch 1.17-245.el7              base     23 k
 perl-Digest-MD5                 x86_64 2.52-3.el7                base     30 k
 perl-Encode-Locale              noarch 1.03-5.el7                base     16 k
 perl-File-Listing               noarch 6.04-7.el7                base     13 k
 perl-HTML-Parser                x86_64 3.71-4.el7                base    115 k
 perl-HTML-Tagset                noarch 3.20-15.el7               base     18 k
 perl-HTTP-Cookies               noarch 6.01-5.el7                base     26 k
 perl-HTTP-Daemon                noarch 6.01-5.el7                base     20 k
 perl-HTTP-Date                  noarch 6.02-8.el7                base     14 k
 perl-HTTP-Message               noarch 6.06-6.el7                base     82 k
 perl-HTTP-Negotiate             noarch 6.01-5.el7                base     17 k
 perl-IO-Compress                noarch 2.061-2.el7               base    260 k
 perl-IO-HTML                    noarch 1.00-2.el7                base     23 k
 perl-IO-Socket-IP               noarch 0.21-4.el7                base     35 k
 perl-IO-Socket-SSL              noarch 1.94-3.el7                base    113 k
 perl-LWP-MediaTypes             noarch 6.02-2.el7                base     24 k
 perl-Net-HTTP                   noarch 6.06-2.el7                base     29 k
 perl-Net-LibIDN                 x86_64 0.12-15.el7               base     28 k
 perl-Net-SSLeay                 x86_64 1.55-3.el7                base    285 k
 perl-TimeDate                   noarch 1:2.30-2.el7              base     52 k
 perl-URI                        noarch 1.60-9.el7                base    106 k
 perl-WWW-RobotRules             noarch 6.02-5.el7                base     18 k
 perl-hivex                      x86_64 1.3.10-5.7.el7            base     41 k
 perl-libwww-perl                noarch 6.05-2.el7                base    205 k
 pulseaudio-libs                 x86_64 6.0-7.el7                 base    576 k
 python-chardet                  noarch 2.2.1-1.el7_1             base    227 k
 python-kitchen                  noarch 1.1.1-5.el7               base    267 k
 qemu-img                        x86_64 10:1.5.3-105.el7_2.1      updates 657 k
 qemu-kvm                        x86_64 10:1.5.3-105.el7_2.1      updates 1.8 M
 qemu-kvm-common                 x86_64 10:1.5.3-105.el7_2.1      updates 390 k
 radvd                           x86_64 1.9.2-9.el7               base     85 k
 rsyslog-mmjsonparse             x86_64 7.4.7-12.el7              base     29 k
 scrub                           x86_64 2.5.2-5.el7               base     40 k
 seabios-bin                     noarch 1.7.5-11.el7              base     74 k
 seavgabios-bin                  noarch 1.7.5-11.el7              base     32 k
 sgabios-bin                     noarch 1:0.20110622svn-4.el7     base    7.1 k
 spice-server                    x86_64 0.12.4-15.el7             base    380 k
 supermin5                       x86_64 5.1.10-1.2.el7            base    574 k
 syslinux                        x86_64 4.05-12.el7               base    990 k
 syslinux-extlinux               x86_64 4.05-12.el7               base    363 k
 unbound-libs                    x86_64 1.4.20-26.el7             base    296 k
 unzip                           x86_64 6.0-15.el7                base    166 k
 usbredir                        x86_64 0.6-7.el7                 base     44 k
 yajl                            x86_64 2.0.4-4.el7               base     39 k
 yum-utils                       noarch 1.1.31-34.el7             base    113 k

Transaction Summary
================================================================================
Install  1 Package (+130 Dependent packages)

Total download size: 59 M
Installed size: 172 M

Leave a comment

Filed under Uncategorized

Inspection, now with added Prolog

You can give libguestfs an unknown disk image, and it tries to guess what’s on it, in terms of operating systems, Linux distro, Windows drive letter mappings and so on, a process that we call inspection. This is an important part of many of the virt tools, because when you type a command like

$ virt-cat -a linux.img /var/log/messages

how is virt-cat to know that on this particular disk image the sysadmin put /var on a separate partition? Because, inspection.

Given that inspection is such an important part of many tools, and vital for standalone programs like virt-inspector you might wonder how it works.

The answer, right now, is 6000+ lines of hairy, intricate C code, which is difficult to maintain and a source of hard to fix bugs and hard to implement feature requests.

$ wc -l src/inspect*.c
   823 src/inspect-apps.c
   725 src/inspect.c
   777 src/inspect-fs.c
   543 src/inspect-fs-cd.c
  2092 src/inspect-fs-unix.c
   704 src/inspect-fs-windows.c
   600 src/inspect-icon.c
  6264 total

How can we make this better?

Getting back to basics, inspection is really a lot of heuristics. Things like:

  • If this filesystem contains a file /etc/fstab then it could be a Linux root filesystem. And:
  • If this thing we think is a Linux root filesystem contains /etc/debian_version then it could be a Debian root filesystem.

These heuristics can be expressed in a logic language inspired by Prolog:

LinuxRoot(fs) :-
    Filesystem(fs),
    File(fs, "/etc/fstab").
DebianRoot(fs) :-
    LinuxRoot(fs),
    File(fs, "/etc/debian_version").

What we’re doing here is collecting a set of facts (Prolog calls them “compound terms”), like:

Filesystem("/dev/sda1").
File("/dev/sda1", "/etc/fstab").
File("/dev/sda1", "/etc/debian_version").

and deriving new facts using the rules:

LinuxRoot("/dev/sda1").
DebianRoot("/dev/sda1").

(I should say at this point that I’m simplifying things a bit. If you want to get a flavour of what the inspection rules might finally look like, then take a look at this file.)

So far I have written a compiler that compiles inspection rules into fairly efficient C code (and hence to binaries), using a forward chaining strategy. It has some nice features like transparently embedding C code into the rules, allowing you to do more complicated operations directly in C:

Distro(fs, distro) :-
    LinuxRootWithOSRelease(fs), /* has /etc/os-release */
    (distro)?={{
      int r;
      CLEANUP_FREE char *distro = NULL;
      if ((r = get_distro_from_os_release (fs, &distro))
           <= 0)
        return r;
      set_distro (distro);
      return 0;
    }}.

and

BlockDevice(dev) :-
    (dev)*={{
      CLEANUP_FREE_STRING_LIST char **devs =
        get_all_block_devices ();
      if (devs == NULL) return -1;
      for (size_t i = 0; devs[i] != NULL; ++i)
        set_dev (devs[i]);
      return 0;
    }}.

My inspection rules run to < 500 lines of code so far, although it’s hard to compare that to the current code because (a) the inspection rules will likely double or triple in size once they are able to do everything that the current code can do, and (b) there’s a lot of supporting runtime code like get_all_block_devices above.

Nevertheless I hope the new rules system will be faster, more supportable and extensible, and easier to understand than the current code. It will also be 100% backwards compatible with existing libguestfs users (since we never break compatibility).

You can follow development in this branch on github.

Update: Hacker News discussion of this article.

Leave a comment

Filed under Uncategorized

48 core ARM 64 bit server

# cat /proc/cpuinfo
processor	: 0
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

processor	: 1
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

[... very long output truncated here ...]

processor	: 44
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

processor	: 45
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

processor	: 46
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

processor	: 47
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x0
CPU part	: 0x0a1
CPU revision	: 1

It’s a Cavium ThunderX. Thanks Cavium and Jon.

5 Comments

Filed under Uncategorized

Booting Fedora on the Dragonboard

TL;DR: All the crazy bootloader shit that affected 32 bit ARM infects 64 bit ARM too.

I pretty much followed Robert Gadsdon’s instructions here, except I had to compile mkbootimg by hand from the CyanogenMod sources as it doesn’t seem to be available in Fedora.

Luckily I have all the bits you need to attach a USB serial port:

20151112_162617

U-boot was installed as the bootloader as above.

For the disk image, I started with the virt-builder Fedora 23 aarch64 image. I copied this image to the SD card by doing:

virt-builder --arch aarch64 fedora-23 --output /dev/mmcblkX

I didn’t expect this would boot, because the kernel is wrong, and the u-boot I flashed to the Dragonboard is in any case looking for a different file. So I cross-compiled a kernel by following the Linaro instructions here (only up to the make Image step).

I didn’t manage to get the kernel to boot, because it just doesn’t load. Or maybe it does load but doesn’t find the serial port. Whatever.

5 Comments

Filed under Uncategorized