Tag Archives: ocaml

xavierbot lives!

Or at least he now has his own git repository.

1 Comment

Filed under Uncategorized

OCaml 4.00.0 beta 2 is in Rawhide

OCaml 4.00.0 betas are available and I’ve packaged it up for Fedora Rawhide.

In other OCaml news, the various “retired” code generators (PPC64, MIPS, IA64, HPPA, Alpha) now have their own repository. Who wants to volunteer to write an s/390x code generator?

Leave a comment

Filed under Uncategorized

OCaml Users and Developers conference 2012

The OCaml Users and Developers Conference (14th September 2012, Copenhagen) is looking for speakers!

I’m intending to submit a talk on using OCaml for code generation in libguestfs.

Leave a comment

Filed under Uncategorized

whenjobs 0.7.0 released

I’ve just released whenjobs 0.7.0 to fix a couple of problems that I was having myself.

Firstly, while it’s a nice feature of whenjobs that I can set variables, sometimes I don’t want jobs to be triggered as a result of setting variables. The new whenjobs --whisper command lets me set variables without reevaluating when-clauses:

$ whenjobs --whisper libguestfs_version=1.17.16

Secondly, my jobs file was getting pretty long and unmaintainable (400 lines), since I added various libguestfs branches and hivex to be managed by whenjobs. So I’ve added a feature where you can split the jobs file into multiple files:

$ ls -l .whenjobs/*.ml
-rw-rw-r--. 1 rjones rjones 3284 Mar 13 19:10 .whenjobs/jobs_hivex.ml
-rw-rw-r--. 1 rjones rjones 4039 Mar 13 19:07 .whenjobs/jobs_libguestfs.ml
-rw-rw-r--. 1 rjones rjones 3766 Mar 13 19:09 .whenjobs/jobs_libguestfs_stable.ml
-rw-rw-r--. 1 rjones rjones   45 Mar 13 19:10 .whenjobs/jobs.ml

I think I described whenjobs as a “cron replacement”. It is a cron replacement for me, for my personal use, but it’s not officially a cron replacement and this project has nothing whatsoever to do with Red Hat or Fedora. The reason I say this is I get some pretty idiotic comments like these.

Leave a comment

Filed under Uncategorized

whenjobs — job lists, cancelling, algorithmic cleanup etc

You can now list and cancel jobs:

$ whenjobs --jobs
61 job$1
	running in: /tmp/whenjobs20d88a48f2c4eb0062e1b44ded6d0ae7
	started at: 2012-02-23 22:43:20
62 job$2
	running in: /tmp/whenjobse9e6b93c3ced1967cbf8c5865d6a1ccb
	started at: 2012-02-23 22:43:20
$ whenjobs --cancel 62

You can manually start jobs. Gerd’s ocamlnet makes it almost trivial to add new RPCs between the tool and the daemon, so adding functions like this is simple.

You can put arbitrary OCaml actions into the job script too, so you can run code when a job is cleaned up, and you will (soon) be able to create jobs algorithmically. For example, the standard mailto cleanup lets you send mail containing the output of the job when it finishes.

let from = "me@example.com"
let to_addr = "you@example.com"
let prefix = "hostname "
let script = << # shell script here >>

job (prefix ^ "poll")
cleanup (Whentools.mailto ~from to_addr)
every minute : script

Leave a comment

Filed under Uncategorized

A work in progress: whenjobs — another cron replacement

whenjobs (git repo) is a cron replacement. From the manual page …

Whenjobs is a powerful but simple replacement for cron. It lets you run jobs periodically like cron, but it also lets you trigger jobs to run when user-defined variables are set or change value.

Periodic jobs are written like this:

every 10 minutes :
<<
  # Get the current load average.
  load=`awk '{print $1}' /proc/loadavg`
  whenjobs --set load $load --type float
>>

When-statements let you create jobs that run based on variables set elsewhere:

when load >= 6 :
<<
  mail -s "ALERT: high load average: $load" $LOGNAME < /dev/null
>>

(When statements are "edge-triggered", meaning that this job will only run when the load goes from under 6 to ≥ 6).

The motivation is building things from git automatically. Here is another job script:

Every 10 minutes, get the latest tagged version from the git repository. The variable ‘version’ will be set to something like “v1.2.3″, “v1.2.4″, etc over time as new releases get tagged.

every 10 minutes :
<<
  cd /my/git/repo
  tag=`git-describe --tags`
  whenjobs --set version $tag
>>

When the ‘version’ variable changes (ie. a new release is tagged) try to build it. ‘changes’ is a function that compares the previous value of a variable from when this job last ran with the current value of a variable, and returns true if the previous and current values are different.

when changes version :
<<
  cd /my/git/buildrepo
  git pull
  git reset --hard $version
  ./configure
  make clean all check dist
  whenjobs --set successful_local_build $version
>>

In parallel, build on a remote machine.

when changes version :
<<
  ssh remote ./do_build $version
  whenjobs --set successful_remote_build $version
>>

Only when the new release has been successfully built on local and remote, upload it to the website.

when successful_local_build == version &&
     successful_remote_build == version :
<<
  cd /my/git/buildrepo
  curl -T name-$success.tar.gz ftp://ftp.example.com/upload/
>>

4 Comments

Filed under Uncategorized

Which foreign function interface is the best?

I’ve written libguestfs language bindings for Perl, Python, Ruby, Java, OCaml, PHP, Haskell, Erlang and C#. But which of these is the best? Which is the easiest? What makes this hard? Grubbing around in the internals of a language reveals mistakes made by the language designers, but what are the worst mistakes?

Note: There is source that goes with this. Download libguestfs-1.13.13.tar.gz and look in the respective directories.

The best

It’s going to be a controversial choice, but in my opinion: C#. You just add some simple annotations to your functions and structs, and you can call into shared libraries (or “DllImport”s as Microsoft insisted on calling them) directly. It’s just about as easy as directly calling C and that is no simple achievement considering how the underlying runtime of C# is very different from C.

Example: a C struct:

[StructLayout (LayoutKind.Sequential)]
public class _int_bool {
  int i;
  int b;
}

The worst

There are two languages in the doghouse: Haskell and PHP. PHP first because their method of binding is just very broken. For example, 64 bit types aren’t possible on a 32 bit platform. It requires a very complex autoconf setup. And the quality of their implementation is very poor verging on broken — it makes me wonder if the rest of PHP can be this bad.

Haskell: even though I’m an experienced functional programmer and have done a fair bit of Haskell programming in the past, the FFI is deeply strange and very poorly documented. I simply could not work out how to return anything other than integers from my functions. You end up with bindings that look like this:

write_file h path content size = do
  r <- withCString path $ \path -> withCString content $ \content -> withForeignPtr h (\p -> c_write_file p path content (fromIntegral size))
  if (r == -1)
    then do
      err <- last_error h
      fail err
    else return ()

The middle tier

There’s not a lot to choose between OCaml, Ruby, Java and Erlang. For all of them: you write bindings in C, there’s good documentation, it’s a bit tedious but basically mechanical, and in 3 out of 4 you’re dealing with a reasonable garbage collector so you have to be aware of GC issues.

Erlang is slightly peculiar because the method I chose (out of many possible) is to write an external process that talks to the Erlang over stdin/stdout. But I can’t fault their documentation, and the rest of it is sensible.

Example: Here is a function binding in OCaml, but with mechanical changes this could be Ruby, Java or Erlang too:

CAMLprim value
ocaml_guestfs_add_drive_ro (value gv, value filenamev)
{
  CAMLparam2 (gv, filenamev);
  CAMLlocal1 (rv);

  guestfs_h *g = Guestfs_val (gv);
  if (g == NULL)
    ocaml_guestfs_raise_closed ("add_drive_ro");

  char *filename = guestfs_safe_strdup (g, String_val (filenamev));
  int r;

  caml_enter_blocking_section ();
  r = guestfs_add_drive_ro (g, filename);
  caml_leave_blocking_section ();
  free (filename);
  if (r == -1)
    ocaml_guestfs_raise_error (g, "add_drive_ro");

  rv = Val_unit;
  CAMLreturn (rv);
}

The ugly

Perl: Get reading. You’d better start with perlxs because Perl uses its own language — C with bizarre macros on top so your code looks like this:

SV *
is_config (g)
      guestfs_h *g;
PREINIT:
      int r;
   CODE:
      r = guestfs_is_config (g);
      if (r == -1)
        croak ("%s", guestfs_last_error (g));
      RETVAL = newSViv (r);
 OUTPUT:
      RETVAL

After that, get familiar with perlguts. Perl has only 3 structures and you’ll be using them a lot. There are some brilliant things about Perl which shouldn’t be overlooked, including POD which libguestfs uses to make effortless manual pages.

Python: Best described as half arsed. Rather like the language itself.

Python, Ruby, Erlang: If your language depends on “int”, “long”, “long long” without defining what those mean, and differing based on your C compiler and platform, then you’ve made a big mistake that will unfortunately dog you throughout the runtime, FFIs and the language itself. It’s better either to define them precisely (like Java) or to just use int32 and int64 (like OCaml).

And finally, reference counting (Perl, Python). It’s tremendously easy to make mistakes that are fiendishly difficult to track down. It’s a poor way to do GC and it indicates to me that the language designer didn’t know any better.

16 Comments

Filed under Uncategorized

What I learned about AMQP

I’m playing with AMQP at the moment. I thought I’d start off with RabbitMQ first.

The good:

  • It works.
  • OCaml and Python programs can talk to each other.
  • It works across remote hosts. You need to open port 5672/tcp on the firewall.

The bad:

  • RabbitMQ and Apache Qpid use different versions of AMQP and are not interoperable! Good summary of the mess here. This might be resolved when everyone gets around to supporting AMQP 1-0, but even though that standard has been published, no one is expecting interop to happen for at least a year.
  • You can’t cluster different versions of the RabbitMQ broker together.
  • Even if all your hosts are at the same RabbitMQ version, you have to open more firewall ports and make changes to the start-up scripts. (Dynamic ports? Really? Did we learn nothing from NFS?)
  • Long, obscure Erlang error messages which don’t point to the problem. eg. You’ll get a good 25 lines of error message if another process is already bound to a port.
  • Possibly just a Fedora packaging problem: I managed to get my host into some state where it’s impossible to stop the RabbitMQ server except by kill -9, and after that I can’t start or stop it.

2 Comments

Filed under Uncategorized

Arrrgghh writing GUI programs is tedious

Last week I thought it would be a good idea to write a small GUI front end to virt-resize.

After two days, I nearly have the first tab (out of four) working.

Granted, maybe the first tab is the hardest one:

The job of the first tab is to ask the user for the source (disk image or libvirt guest). It then fires off a background job to open that guest and inspect it for operating systems. Based on the outcome of that (opened OK, found OSes, no OSes found, etc) it has to update and enable the other tabs.

Also the user can preselect a guest on the command line. We also have to deal with connecting to libvirt asynchronously to read the list of guests (and this might also fail in a number of ways).

So far, 1600 lines of code, and the first tab is by no means complete.

One part of the problem is there’s a certain “impedance mismatch” between functional programming in OCaml and writing Gtk. Gtk is heavily mutable and object based. OCaml prefers (but does not require) immutability, and objects in OCaml are obscure and not widely used, and the Gtk bindings are written in a very hard-core object OCaml style.

Another part is just that it’s tedious. It would be tedious if I was doing this in C or Python too. You’ve got asynchronous actions going off here and there which update bits of state. Every control or input could potentially affect every other control or output, resulting in a kind of O(n2) mess of wiring up signals and callbacks.

Is there an easier way? I don’t know …

13 Comments

Filed under Uncategorized

OCaml 1-page “cheat sheets”

OCamlPro, the commercial company set up earlier this year to promote and support the OCaml language, have made some nice 1-page cheat sheets for the language and tools.

Leave a comment

Filed under Uncategorized