Goaljobs, part 3

In part 2 I introduced an example goaljobs script that can rebuild a set of packages in Fedora in the right order.

It’s time to take a closer look at targets — the promise that you make that some condition will be true by the time a goal has run.

In the Fedora rebuild script the goal targets looked like this:

let goal rebuilt pkg =
  target (koji_build_state (fedora_verrel pkg branch)
               == `Complete);
  ...

koji_build_state is a regular function. It’s implemented using the koji buildinfo command line tool for querying the Koji build system. (The koji command line tool is annoyingly hard to automate, but as we’ve got a complete programming language available — not just bash — the implementation of koji_build_state is tedious and long, but doable).

Querying Koji takes a few seconds and we don’t want to do it every time we check a goal. Goaljobs offers a feature called “The Memory” which lets you memoize functions. “The Memory” is just a fancy name for a key/value store which is kept in ~/.goaljobs-memory and persists across goaljobs sessions:

let koji_build_state verrel =
  let key = sprintf "koji_build_complete_%s" verrel in
  if memory_exists key then
    `Complete
  else (
    (* tedious code to query koji *)
    if state == `Complete then
      memory_set key "1";
    state
  )

With strategic use of memoization, evaluating goaljobs goals can be very fast and doesn’t change the fundamental contract of targets.

Finally in this part: a note on how targets are implemented.

A target is a boolean expression which is evaluated once near the beginning of the goal. If it evaluates to true at the beginning of the goal then the rest of the goal can be skipped because the goal has already been achieved / doesn’t need to be repeated.

And since targets are just general expressions, they can be anything at all, from accessing a remote server (as here) to checking the existence of a local file (like make). As long as something can be tested quickly, or can be tested slowly and memoized, it’s suitable to be a target.

Goaljobs, part 2

In part 1 I showed how a simple make rule could be converted to a special “goal” function and I hinted that we were not limited to just the “file is older than” semantics implicit in make.

So let’s have a look at the goals I wrote to automate the recent OCaml rebuild in Fedora.

Recall from part 1: Targets are a contractual promise that you make in goaljobs. They are a promise that some condition will be true after running the goal. Requirements are conditions that must be true before the goal can start running.

For a Fedora package to achieve the goal of being rebuilt, the target is that the Koji build state of the current release must be “Completed”. The requirements are that every dependency of the package has been rebuilt. So:

let goal rebuilt pkg =
  target (koji_build_state (fedora_verrel pkg branch)
               == `Complete);

  (* Require the rebuild to have started: *)
  require (rebuild_started pkg);

  ... some code to wait for the build to finish ...

The above code is not complete (it’s a complex, real-world working example after all).

I split the rebuilt goal into two separate goals for reasons that will become clear later. But the first goal above says that the package rebuild must have been started off, and we’ll wait for the package build to complete.

Note that once the build is complete, the target promise is true.

The subgoal rebuild_started is defined like this:

let goal rebuild_started pkg =
  (* The dependencies of this package: *)
  let deps = List.assoc pkg pkg_deps in

  target (
     match koji_build_state (fedora_verrel pkg branch) with
          | `Building | `Complete -> true
          | _ -> false
    );

  (* All dependent packages must have been fully rebuilt: *)
  List.iter (fun dep -> require (rebuilt dep)) deps;

  (* Rebuild the package in Koji. *)
  koji_build pkg branch

It’s saying that the target (promise) will be that the Koji package will either be building or may even be complete. And that we first of all require that every build dependency of this package has been completely, successfully rebuilt. If those requirements are met, we tell Koji to start building the package (but in this goal we don’t need to wait for it to complete).

Why did I split the goal into two parts?

The reason is that I want to define a make-like all goal:

let goal all () =
  List.iter (fun pkg -> require (rebuild_started pkg))
    source_packages

This iterates over all my source packages and starts rebuilding them.

Note it doesn’t wait for each one to be rebuilt … unless they are required as dependencies of another package, in which case the require (rebuilt dep) will kick in and wait for them before rebuilding the dependent package.

In other words, this code automatically resolves dependencies, waiting where necessary, but otherwise just kicking off builds, which is exactly what I wanted.

---

Finally a bit about how you use a goaljobs script. Unlike make you have to compile the script into a binary. To compile the script, use the convenient wrapper goaljobs (it’s a simple shell script that invokes the OCaml compiler):

goaljobs fedora_ocaml_rebuild.ml

This makes a binary called fedora_ocaml_rebuild which is the program for mass-rebuilding the whole of Fedora’s OCaml subset.

When you run it with no arguments, it searches for a goal called all and “requires” that goal (just like make).

You can also run other goals directly. Any goal which is “published” can be run from the command line. All goals that have no parameters — such as all — are published automatically.

For goals that take parameters, if you want to use them from the command line you have to publish them manually. The reason is that you have to provide a small code snippet to convert the command line parameters to goal parameters, which may involve type conversion or other checks (since OCaml is strongly typed and parameters can be any type, not just strings or filenames).

Goaljobs, part 1

A little more than a year ago I released whenjobs which was an attempt to create a practical language for automating complex “business rules”. The kind of thing I’m talking about is managing the many diverse steps between me tagging a libguestfs commit with a version number and a fully tested tarball appearing on the website. Or the hundreds of steps that go into 100 OCaml packages being updated and rebuilt for Rawhide.

Whenjobs wasn’t the right answer. Goaljobs [very early alpha] might possibly be.

What I need is something which is flexible, can deal with failures (both hard and intermittent), and can be killed and restarted at any point.

The first observation is that make is nearly the right tool. It’s goal-based, meaning that you set down a target that you want to have happen, and some rules to make that happen, and this lets you break down a problem from the larger goal (“build my program!”) to smaller subgoals (“compile this source file”).

program: main.o utils.o
  cc $^ -o $@

The goal is “program is built”. There are some requirements (main.o, utils.o), and there’s a recipe (run cc). You can also kill make in the middle and restart it, and it’ll usually continue from where it left off.

Make also lets you parameterize goals, although only in very simple ways:

%.o: %.c
  cc -c $< -o $@

Implicit in the “:” (colon) character is make’s one simple rule, which is roughly this: “if the target file doesn’t exist, or the prerequisite files are newer than the target, run the recipe below”.

In fact you could translate the first make rule into an ordinary function which would look something like this:

function build_program ()
{
  if (!file_exists ("program") ||
      file_older ("program", "main.o") ||
      file_older ("program", "utils.o")) {
    shell ("cc -c %s -o %s", "main.o utils.o",
           "program");
  }
}

Some points arise here:

• Why can’t we change the target test to something other than “file exists or is newer”?
  How about “remote URL exists” (and if not, we need to upload a file)?
  How about “Koji build completed successfully” (and if not we need to do a Fedora build)?
• What could happen if we could add parameters to build_program?

Goaljobs attempts to answer these questions by turning make-style rules into “goals”, where goals are specialized functions similar to the one above that have a target, requirement(s), a recipe to implement them, and any number of parameters.

For example, a “compile *.c to *.o” goal looks like this:

let goal compiled c_file =
  (* convert c_file "foo.c" -> "foo.o": *)
  let o_file = change_file_extension "o" c_file in

  target (more_recent [o_file] [c_file]);

  sh "
    cd $builddir
    cc -c %s -o %s
  " c_file o_file

The goal is called compiled and it has exactly one parameter, the name of the C source file that must be compiled.

The target is a promise that after the recipe has been run the *.o file will be more recent than the *.c file. The target is both a check used to skip the rule if it’s already true, but also a contractual promise that the developer makes (and which is checked by goaljobs) that some condition holds true at the end of the goal.

sh is a lightweight way to run a shell script fragment, with printf-like semantics.

And the whole thing is wrapped in a proper programming language (preprocessed OCaml) so you can do things which are more complicated than are easily done in shell.

whenjobs 0.7.0 released

I’ve just released whenjobs 0.7.0 to fix a couple of problems that I was having myself.

Firstly, while it’s a nice feature of whenjobs that I can set variables, sometimes I don’t want jobs to be triggered as a result of setting variables. The new whenjobs --whisper command lets me set variables without reevaluating when-clauses:

$ whenjobs --whisper libguestfs_version=1.17.16

Secondly, my jobs file was getting pretty long and unmaintainable (400 lines), since I added various libguestfs branches and hivex to be managed by whenjobs. So I’ve added a feature where you can split the jobs file into multiple files:

$ ls -l .whenjobs/*.ml
-rw-rw-r--. 1 rjones rjones 3284 Mar 13 19:10 .whenjobs/jobs_hivex.ml
-rw-rw-r--. 1 rjones rjones 4039 Mar 13 19:07 .whenjobs/jobs_libguestfs.ml
-rw-rw-r--. 1 rjones rjones 3766 Mar 13 19:09 .whenjobs/jobs_libguestfs_stable.ml
-rw-rw-r--. 1 rjones rjones   45 Mar 13 19:10 .whenjobs/jobs.ml

I think I described whenjobs as a “cron replacement”. It is a cron replacement for me, for my personal use, but it’s not officially a cron replacement and this project has nothing whatsoever to do with Red Hat or Fedora. The reason I say this is I get some pretty idiotic comments like these.

whenjobs 0.6 released

whenjobs 0.6 fixes two out of three bugs mentioned previously, and fixes the bug Joe found in whenjobs --daemon-start (et al).

Changes:

• You can set multiple variables atomically, using:

  whenjobs --set cat=sushi food=fish
  

  The effect is that when-clauses are reevaluated only after all the variables have changed, rather than after each change as before.

• You can test the effect of setting variables, without setting them (or running jobs). The new --test option displays which jobs would run:

  $ whenjobs --test food=meat
  job$1
  job$5
  

• whenjobs --daemon-{start,restart,status} now all work as expected
• whenjobs --job-names lists job names of all loaded jobs, whether running or not.

whenjobs — job lists, cancelling, algorithmic cleanup etc

You can now list and cancel jobs:

$ whenjobs --jobs
61 job$1
	running in: /tmp/whenjobs20d88a48f2c4eb0062e1b44ded6d0ae7
	started at: 2012-02-23 22:43:20
62 job$2
	running in: /tmp/whenjobse9e6b93c3ced1967cbf8c5865d6a1ccb
	started at: 2012-02-23 22:43:20
$ whenjobs --cancel 62

You can manually start jobs. Gerd’s ocamlnet makes it almost trivial to add new RPCs between the tool and the daemon, so adding functions like this is simple.

You can put arbitrary OCaml actions into the job script too, so you can run code when a job is cleaned up, and you will (soon) be able to create jobs algorithmically. For example, the standard mailto cleanup lets you send mail containing the output of the job when it finishes.

let from = "me@example.com"
let to_addr = "you@example.com"
let prefix = "hostname "
let script = << # shell script here >>

job (prefix ^ "poll")
cleanup (Whentools.mailto ~from to_addr)
every minute : script

A work in progress: whenjobs — another cron replacement

whenjobs (git repo) is a cron replacement. From the manual page …

Whenjobs is a powerful but simple replacement for cron. It lets you run jobs periodically like cron, but it also lets you trigger jobs to run when user-defined variables are set or change value.

Periodic jobs are written like this:

every 10 minutes :
<<
  # Get the current load average.
  load=`awk '{print $1}' /proc/loadavg`
  whenjobs --set load $load --type float
>>

When-statements let you create jobs that run based on variables set elsewhere:

when load >= 6 :
<<
  mail -s "ALERT: high load average: $load" $LOGNAME < /dev/null
>>

(When statements are "edge-triggered", meaning that this job will only run when the load goes from under 6 to ≥ 6).

The motivation is building things from git automatically. Here is another job script:

Every 10 minutes, get the latest tagged version from the git repository. The variable ‘version’ will be set to something like “v1.2.3”, “v1.2.4”, etc over time as new releases get tagged.

every 10 minutes :
<<
  cd /my/git/repo
  tag=`git-describe --tags`
  whenjobs --set version $tag
>>

When the ‘version’ variable changes (ie. a new release is tagged) try to build it. ‘changes’ is a function that compares the previous value of a variable from when this job last ran with the current value of a variable, and returns true if the previous and current values are different.

when changes version :
<<
  cd /my/git/buildrepo
  git pull
  git reset --hard $version
  ./configure
  make clean all check dist
  whenjobs --set successful_local_build $version
>>

In parallel, build on a remote machine.

when changes version :
<<
  ssh remote ./do_build $version
  whenjobs --set successful_remote_build $version
>>

Only when the new release has been successfully built on local and remote, upload it to the website.

when successful_local_build == version &&
     successful_remote_build == version :
<<
  cd /my/git/buildrepo
  curl -T name-$success.tar.gz ftp://ftp.example.com/upload/
>>

Tip: daily Bugzilla reports

This is my inbox, and it sucks:

Bugzilla is like a black hole for bugs. The search tools fail so badly it’s often impossible to find a bug that you were working on the day before. It’s slow, clumsy, and disorganized.

But one bright point is it has a reasonable command line reporting tool also available as a Fedora package. So I decided yesterday to write a Bugzilla report script and have it email me daily from cron.

The starting point is to identify “bugs of interest”. The bugs of interest to me are:

• bugs I reported
• bugs I am assigned to fix
• bugs I’m CC’d on

I put some thought into this set of criteria for bugs:

1. I should be able to register an interest in any bug: Yes, by adding myself to the CC field.
2. I should be able to unregister an interest: Yes, by removing myself from the CC, or closing or reassigning those bugs where I’m reporter or assignee.
3. It shouldn’t tell me about bugs I’m not interested in: Yes, I would be reporter, assignee or in the CC list for any bug related to me or a package I maintain.
4. It shouldn’t miss out any bugs I might be interested in: Yes, if I’ve ever added a comment, I will be in the CC field at least, and I can always unregister myself from any I no longer care about.

With the command line tool, getting the raw list of bug IDs is very simple.

for flag in -r -c -a; do
  bugzilla query $flag $email -i
done | sort -n -u

(replacing $email with your email address, or even mine if you so care).

That pulls out 781 unique bug IDs, the oldest being one I don’t remember reporting and the most recent being a virt-v2v bug. (Note this includes CLOSED bugs which in the script are ignored).

Now I take the list of bug IDs and pull out the other fields I want from the Bugzilla database. The command below is just an illustrative example of what the script does:

$ bugzilla query -b "86308,601092" \
  --outputformat="%{bug_id}|%{bug_status}|%{product}|%{component}|%{short_desc}" |
  sort -n -t"|"
86308|CLOSED|Red Hat Web Site|Other|Red Hat Command Centre site is down
601092|NEW|Red Hat Enterprise Linux 5|libguestfs|[RFE]Incorrect error msg popped up when missing "-f" in v2v command

As far as I know you have to guess what the %-fields are called. I also had to choose a separator character which wouldn’t appear in any field except the short_desc (which is always the last field), since fields like the product name can and do contain spaces.

The rest of the script is “merely” formatting this whole lot into a nice looking email report:

Bugs in OPEN states:

*** coccinelle ***

  In Fedora:
    502185 coccinelle segfaults on ppc64
    579457 coccinelle-0.2.3rc1 is available

*** e2fsprogs ***

  In Red Hat Enterprise Linux 5:
    518190 mke2fs -t option in RHEL5 differs vs upstream, leading to confusion

[etc]

You can download the script here: http://oirase.annexia.org/tmp/bugs-report.ml.txt

Tip: find out when filesystems get full with virt-df (working version!)

Thanks go to “S” for pointing out the mistake in my last posting. It’s easy to fix though.

When virt-df is run with the –csv option, it outputs a Comma-Separated Values format spreadsheet. CSV is not totally straightforward to parse and so you should leave it up to a specialized tool. Of course I recommend my own tool csvtool which is available in Fedora, Debian and Ubuntu.

The basic plan is to use csvtool to drop the first row (the column headings row), and pull out columns 1, 2 and 6 from the rest of the output. Those columns are respectively the virtual machine name, the filesystem, and the %usage:

$ virt-df --csv | csvtool drop 1 - | csvtool col 1-2,6 -
Debian5x64,/dev/debian5x64/home,35.6%
Debian5x64,/dev/debian5x64/root,60.9%
Debian5x64,/dev/debian5x64/tmp,3.3%
Debian5x64,/dev/debian5x64/usr,40.5%
Debian5x64,/dev/debian5x64/var,15.2%
Debian5x64,/dev/vda1,20.3%
RHEL620100329n0x64,/dev/vda1,7.4%
RHEL620100329n0x64,/dev/vg_rhel620100329n0/lv_root,16.7%
Ubuntu910x64,/dev/vda1,42.5%
[...snipped...]

Adding the previous awk test gives the desired output, all filesystems with ≥ 60% usage:

$ virt-df --csv |
    csvtool drop 1 - |
    csvtool col 1-2,6 - |
    csvtool readable - |
    awk 'strtonum(substr($3,0,length($3)-1))>=60 {print}'
Debian5x64         /dev/debian5x64/root            60.9% 
F13Rawhidex64      /dev/vg_f13rawhide/lv_root      66.0% 
Windows7x32        /dev/vda2                       74.6% 
Windows7x64        /dev/vda2                       90.7% 

This can be placed in a script or cron job.

Tip: find out when filesystems get full with virt-df

Another easy tip for the day: Want to receive a warning when one of your virtual machines’ filesystems is filling up? Use virt-df, awk, and cron like this.

Note: I’ve set this up so I can do it as non-root, but that meant I had to add myself to the ‘disk’ group first, so that virt-df can read out the contents of the disks of the virtual machines. There’s one extra thing needed which is to set the LIBVIRT_DEFAULT_URI environment variable to access the global libvirtd socket.

$ export LIBVIRT_DEFAULT_URI='qemu+unix:///system?socket=/var/run/libvirt/libvirt-sock-ro'
$ virt-df | awk 'strtonum(substr($5,0,length($5)-1))>=60 {print}'
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%

You can set the threshold to whatever you want. In the above awk code it’s set to 60%, you’d probably want it higher.

Now put those two commands into a script file somewhere, and create a cronjob. The following one runs daily:

MAILTO=you@example.com
0 1 * * * /usr/local/etc/diskcheck.sh

What do you do when a VM is running out of space? Why, you run virt-resize of course.

---

Ob-geekiness: The full virt-df output from one of my test hosts. This one is used for a lot of interop testing:

$ virt-df
Filesystem                           1K-blocks       Used  Available  Use%
RHEL6200910210x32:/dev/vda1             198337      36582     151515   19%
RHEL6200910210x32:/dev/vg_rhel6200910210x32/lv_root
                                       9147776    5305876    3377212   59%
RHEL6201002033x64:/dev/vda1             495844      32396     437848    7%
RHEL6201002033x64:/dev/vg_rhel6201002033x64/lv_root
                                      18102140    4097800   13084788   23%
Debian5x64:/dev/debian5x64/home        3555936    1264800    2110504   36%
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Debian5x64:/dev/debian5x64/tmp          309401      10294     283133    4%
Debian5x64:/dev/debian5x64/usr         3539776    1434740    1925224   41%
Debian5x64:/dev/debian5x64/var         1741648     264632    1388544   16%
Debian5x64:/dev/vda1                    233335      47272     173615   21%
Ubuntu910x64:/dev/vda1                 9827520    4180524    5147780   43%
RHEL6Alpha3x64:/dev/vda1                198337      22879     165218   12%
RHEL6Alpha3x64:/dev/vg_rhel6alpha3x64/lv_root
                                       8180152    4174904    3589712   52%
RHEL54Betax64:/dev/VolGroup00/LogVol00
                                      15109112    8695548    5633688   58%
RHEL54Betax64:/dev/vda1                 101086      12449      83418   13%
F10x32:/dev/VolGroup00/LogVol00        9191640    3083532    5642856   34%
F10x32:/dev/vda1                        194442      20706     163697   11%
F12x64:/dev/vda1                        198337      22782     165315   12%
F12x64:/dev/vg_f12x64/lv_root          9115576    4396520    4256004   49%
F13Rawhidex64:/dev/vda1                 198337      60031     128066   31%
F13Rawhidex64:/dev/vg_f13rawhide/lv_root
                                       9115576    6018664    2633860   67%
F12x64preview:/dev/vda1                 198337      22782     165315   12%
F12x64preview:/dev/vg_f12x64/lv_root   9115576    4781384    3871140   53%
Win2003x32:/dev/vda1                  20956760    3053092   17903668   15%
VSphere:/dev/vda1                     31447204    8159028   23288176   26%
Windows7x32:/dev/vda1                   102396      24712      77684   25%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda1                   102396      24704      77692   25%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%
CentOS5x32:/dev/VolGroup00/LogVol00    9014656    4069840    4479512   46%
CentOS5x32:/dev/vda1                    101086      36210      59657   36%

Update There is a small but important bug in this code. Please see the fixed version.