libguestfs tutorial to appear in Linux Format

Chris Brown has written a short tutorial on libguestfs and guestfish which will appear in Linux Format (a UK magazine) #152 in December/January.

Learning Functional Reactive Programming

Functional Reactive Programming (FRP) is a programming technique that I think hasn’t filtered down to the level of workaday programmers (like myself) but is well-regarded in the academic community.

Upside is that it could make writing Gtk and web applications much simpler.

Downside is that no one from the academic world has ever managed to explain FRP in a way that makes any sense to mere mortals. There are, as far as I can see, no good tutorials about FRP on the web, and by “good” I mean to exclude any that mention words like “composition”, “point-free”, “point-wise”, “denotational” or any other jargon like that.

Well, this one isn’t too bad … What I really need though are real, concrete, compilable examples.

The inimitable Daniel Bünzli, author of the OCaml FRP framework React wrote his own intro to FRP today, although I’m still not convinced I understand it.

So I’m going to learn it, and if I can, describe it to you in some forthcoming posts.

But what everyone agrees on are the basics: FRP is about putting the imperative programming concept of a “variable” on a sound theoretical basis. Let me explain …

In C you can declare a variable, like:

#define UP 1
#define DOWN 2
int mouse_button_state = UP;

and you can assign to that C variable over the course of your program:

void button_pressed (void)
{
  mouse_button_state = DOWN;
}

The problem[1] is that mouse_button_state changes as time passes, but the variable itself doesn’t capture the past, present or future of the variable. Assigning a new value to the variable (a) doesn’t record the time that the variable changed, and (b) obliterates all previous states of the variable. The variable has no memory of what happened before.

[1] I’m using the word “problem” advisedly here. You and I might not think this is a problem at all. But FRP is going to persuade us that this is a problem and by putting variables on a sounder theoretical basis, our programs will be shorter and easier to reason about.

C programs that really deal with mouse events are, of course, full of callbacks. When a mouse button changes state (or more often, when it changes state several times) we want to call back some function that takes an action in response to user input. Callbacks are a huge problem in imperative programs, as you’ll know if you’ve ever done anything non-trivial. Amongst the problems are (1) they run asynchronously, (2) they run in some other part of the program and it’s not obvious what runs when and where, and (3) they often have bad interactions when several callbacks run and they might be called in different orders.

FRP promises to fix this, again by modeling events on a sound theoretical basis.

FRP also promises to save us from reasoning about: complicated multithreaded programs, event-driven programs (like operating systems, libvirt, etc.), writing web applications, and writing GUI applications.

I’m yet to be convinced, but I’m going to learn about FRP and try to write the world’s first usable FRP tutorial.

Supermin appliance – now in febootstrap

Q: Can you fit a bootable Fedora distribution into 100 kilobytes?
A: You bet!*
* by cheating … read on.

Take an ordinary Fedora appliance as made by febootstrap or appliance-creator. The appliance image is big because it contains copies of programs (/bin/bash) and libraries (/lib/libc.so), the kernel and kernel modules. It needs to, to make it self-contained.

But a technique we’ve been using for a few months in libguestfs is to say: we’re booting this appliance on a Fedora host. Let’s strip out all those programs and libraries from the appliance, and we’ll add them back from the host just before we launch it.

I called such appliances “supermin appliances”, and now I’ve ported the functionality from libguestfs into febootstrap so everyone can use it.

A supermin appliance is really small:

-rw-rw-r-- 1 rjones rjones 14K 2009-10-22 12:43 hostfiles.txt
-rw-rw-r-- 1 rjones rjones 87K 2009-10-22 12:43 supermin.img

yet it’s fully bootable, given the right sort of host:

It’s really easy to use the new febootstrap to make your own super-small supermin appliances.

We’ll start with this small shell script to make a supermin appliance:

#!/bin/sh -
distro=rawhide
febootstrap -i bash -i coreutils $distro fedora
febootstrap-minimize fedora
cat > init <<EOF
#!/bin/sh
echo Starting /init script ...
PATH=/sbin:/usr/sbin:$PATH
touch /etc/fstab
mount -t proc /proc /proc
mount -t sysfs /sys /sys
exec bash -i
EOF
febootstrap-install fedora init /init 0755 root.root
rm init
# Create the supermin appliance.
febootstrap-to-supermin fedora supermin.img hostfiles.txt
# Create the ordinary appliance just for comparison.
febootstrap-to-initramfs fedora > ordinary.img

You will need to use febootstrap >= 2.5, and set the distro variable so it exactly matches your base Fedora system (eg. set distro=fedora-11).

The script below can be used to boot the appliance, and it’s what I used to get the screenshot above.

#!/bin/sh -
time febootstrap-supermin-helper \
    supermin.img hostfiles.txt kernel initrd
qemu-system-x86_64 -m 1024 -kernel kernel -initrd initrd

A beginners guide to OCaml internals

In this 6 part series, I’m going to introduce the internals of the OCaml programming language (tutorial and other references here). This isn’t going to be very comprehensive or in-depth. It’s more of a digest of the readily available information that I found by reading the manual, header files, and some of the compiler code. It’s definitely pitched at beginners, not experts.

By the end of it, you should be able to look at small pieces of OCaml and work out in your head how they get translated into machine code.

I’m definitely not a great expert on the internals of OCaml. This means a couple of things: (a) I’ve probably made some mistakes (post a comment if you see any). (b) I might “reveal too much” of the specifics of the current implementation: anything could change in a future version of OCaml, although the internals described here have been pretty stable for a long time.

Without further ado …

Part 1: Values

In a running OCaml program, a value is either an “integer-like thing” or a pointer.

What’s an “integer-like thing”? Any OCaml int or char, or some things which are stored like integers for speed, which include the constants true and false, the empty list [], unit (), and some (but not all) variants.

For reasons that I’ll explain later, OCaml stores values which are integer-like and values which are pointers differently.

A value containing a pointer is just stored as the pointer. Because addresses are always word-aligned, the bottom 2 bits of every pointer are always 0 0 (the bottom 3 bits are 0 0 0 for a 64 bit machine). OCaml therefore stores integers by setting the bottom bit to 1 and shifting the integer over 1 place:

+----------------------------------------+---+---+
| pointer                                | 0 | 0 |
+----------------------------------------+---+---+

+--------------------------------------------+---+
| integer (31 or 63 bits)                    | 1 |
+--------------------------------------------+---+

You can quickly tell if a value is an integer-like thing (ie. is the bottom bit 1?).

OCaml gives you some C macros and OCaml functions you can use on values. The C macros are Is_long (is it an integer?) and Is_block (is it a pointer?) in the header file . The OCaml functions are is_int and is_block (for pointers) in the “infamous” Obj module.

# let what_is_it foo =
    let foo_repr = Obj.repr foo in
    if Obj.is_int foo_repr then "integer-like" else "pointer" ;;
val what_is_it : 'a -> string = 
# what_is_it 42 ;;
- : string = "integer-like"
# what_is_it () ;;
- : string = "integer-like"
# what_is_it [1;2;3] ;;
- : string = "pointer"
# type bar = Bar | Baz of int ;;
type bar = Bar | Baz of int
# what_is_it Bar ;;
- : string = "integer-like"
# what_is_it (Baz 5) ;;
- : string = "pointer"

Are ints slow? After all it seems like you have to do a lot of bit shifting to do any arithmetic with them.

It turns out that they are a little bit slower, but not by very much. The OCaml compiler can perform most arithmetic operations in between one and four machine instructions, and usually just one or two, making it the same or fractionally slower than a straight native integer. (Of course, there is a more serious penalty in the cases where you really need the full width of a 32 or 64 bit integer, for example in crypto or compression algorithms).

Increment	addl $2, %eax
Add a constant	addl (constant*2), %eax
Add two values	lea -1(%eax, %ebx), %eax
Subtract	subl %ebx, %eax incl %eax

That these are equivalent isn’t totally obvious, but this post explains why.

That concludes part 1. Tune in tomorrow for part 2, or if you’re reading this in the future, use this handy table of contents:

• Part 2: Strings and other types
• Part 3: The minor heap
• Part 4: The major heap
• Part 5: Garbage collection
• Part 6: Conclusion, further reading

Update

This guide is discussed on reddit here. Also on Hacker News here.

Using autoconf for OCaml projects

Today Zack and I released the first version of ocaml-autoconf, which is a collection of OCaml-related autoconf macros. That package won’t tell you how to use autoconf though, and that is something that people wrongly regard as being incomprehensible or too hard to learn. In this tutorial you’ll see that you only need to create two files to start even a complex autoconf / OCaml project.

The download which goes along with this tutorial is: ocaml-autoconf-example-1.0.tar.gz

configure.ac

The first file controls autoconf. It is called configure.ac and looks like this. Notice the use of the OCaml autoconf macros which are documented in the ocaml.m4 manual page.

AC_INIT(ocaml-autoconf-example,1.0)
AM_INIT_AUTOMAKE

AC_PROG_CC

AC_PROG_OCAML
if test "$OCAMLC" = "no"; then
  AC_MSG_ERROR([You must install the OCaml compiler])
fi

AC_PROG_FINDLIB
if test "$OCAMLFIND" = "no"; then
  AC_MSG_ERROR([You must install OCaml findlib (the ocamlfind command)])
fi

AC_CHECK_OCAML_PKG([lablgtk2])
if test "$OCAML_PKG_lablgtk2" = "no"; then
  AC_MSG_ERROR([Please install OCaml findlib module 'lablgtk2'.])
fi

AC_CONFIG_FILES([Makefile])
AC_OUTPUT

Makefile.am

The second file controls automake. In fact we’re not really using automake’s capabilities, so our Makefile.am looks almost like an ordinary Makefile:

bin_SCRIPTS = gtk_test$(EXEEXT)

PACKAGES = -package lablgtk2
LIBS     = gtkInit.cmo -linkpkg

gtk_test$(EXEEXT): gtk_test.cmo
	$(OCAMLFIND) ocamlc $(PACKAGES) $(LIBS) $^ -o $@

.ml.cmo:
	ocamlfind ocamlc $(PACKAGES) -c $< -o $@

EXTRA_DIST = gtk_test.ml

CLEANFILES = *.cmi *.cmo $(bin_SCRIPTS) *~

m4/ocaml.m4

Those are the only two files you have to write, but you will also need to copy the OCaml autoconf macros (ocaml.m4) into an m4/ subdirectory:

mkdir m4
cp /usr/share/aclocal/ocaml.m4 m4/

autoreconf

Finally run autoreconf to run autoconf and automake and generate the ./configure script. Because automake defaults to assuming your project conforms to all the GNU coding standards, you probably want to add the automake --foreign flag, so:

AUTOMAKE='automake --foreign' autoreconf -i

Finished

Now you’re done. You can copy the gtk_test.ml program into this directory, and try:

./configure
make
make dist
make install

etc.

Download: ocaml-autoconf-example-1.0.tar.gz

Update #1: Zack’s blog posting about ocaml-autoconf

Update #2: See comments.