Goaljobs, part 1

A little more than a year ago I released whenjobs which was an attempt to create a practical language for automating complex “business rules”. The kind of thing I’m talking about is managing the many diverse steps between me tagging a libguestfs commit with a version number and a fully tested tarball appearing on the website. Or the hundreds of steps that go into 100 OCaml packages being updated and rebuilt for Rawhide.

Whenjobs wasn’t the right answer. Goaljobs [very early alpha] might possibly be.

What I need is something which is flexible, can deal with failures (both hard and intermittent), and can be killed and restarted at any point.

The first observation is that make is nearly the right tool. It’s goal-based, meaning that you set down a target that you want to have happen, and some rules to make that happen, and this lets you break down a problem from the larger goal (“build my program!”) to smaller subgoals (“compile this source file”).

program: main.o utils.o
  cc $^ -o $@

The goal is “program is built”. There are some requirements (main.o, utils.o), and there’s a recipe (run cc). You can also kill make in the middle and restart it, and it’ll usually continue from where it left off.

Make also lets you parameterize goals, although only in very simple ways:

%.o: %.c
  cc -c $< -o $@

Implicit in the “:” (colon) character is make’s one simple rule, which is roughly this: “if the target file doesn’t exist, or the prerequisite files are newer than the target, run the recipe below”.

In fact you could translate the first make rule into an ordinary function which would look something like this:

function build_program ()
{
  if (!file_exists ("program") ||
      file_older ("program", "main.o") ||
      file_older ("program", "utils.o")) {
    shell ("cc -c %s -o %s", "main.o utils.o",
           "program");
  }
}

Some points arise here:

  • Why can’t we change the target test to something other than “file exists or is newer”?
    How about “remote URL exists” (and if not, we need to upload a file)?
    How about “Koji build completed successfully” (and if not we need to do a Fedora build)?
  • What could happen if we could add parameters to build_program?

Goaljobs attempts to answer these questions by turning make-style rules into “goals”, where goals are specialized functions similar to the one above that have a target, requirement(s), a recipe to implement them, and any number of parameters.

For example, a “compile *.c to *.o” goal looks like this:

let goal compiled c_file =
  (* convert c_file "foo.c" -> "foo.o": *)
  let o_file = change_file_extension "o" c_file in

  target (more_recent [o_file] [c_file]);

  sh "
    cd $builddir
    cc -c %s -o %s
  " c_file o_file

The goal is called compiled and it has exactly one parameter, the name of the C source file that must be compiled.

The target is a promise that after the recipe has been run the *.o file will be more recent than the *.c file. The target is both a check used to skip the rule if it’s already true, but also a contractual promise that the developer makes (and which is checked by goaljobs) that some condition holds true at the end of the goal.

sh is a lightweight way to run a shell script fragment, with printf-like semantics.

And the whole thing is wrapped in a proper programming language (preprocessed OCaml) so you can do things which are more complicated than are easily done in shell.

About these ads

6 Comments

Filed under Uncategorized

6 responses to “Goaljobs, part 1

  1. Pingback: Goaljobs, part 2 | Richard WM Jones

  2. jamesjustjames

    The thing you’re looking for is called puppet. At least that might solve your problem. I don’t know all the specifics of what you want to do.

    Cheers
    James

    • rich

      I’m pretty sure that Puppet is not relevant here, but please enlighten me if it is.

      • jamesjustjames

        Well we’re in a pickle, because I don’t know about your jobs project, and you don’t know about puppet. The reason it jumped out at me as a possibility was when I read the part about:

        automating complex “business rules”.

        and

        managing the many diverse steps.

        Those two things can be done in puppet. You can create dependency relationships, and exec commands “onlyif” some command is true, and / or unless a command is true, and so on…

      • rich

        I think I’m pretty familiar with Puppet (Edit: although definitely not as familiar as you judging by your blog …). I’m still pretty sure that Puppet managing machines isn’t relevant to this. It did set me looking to see if Puppet had some sort of business rules mode, but I can’t find anything.

        Edit: In the interests of giving Puppet a fair hearing, how would you monitor a git repository, and when a new commit appears, have it run “./configure && make && make check” in that repository?

        What is similar to this are the many business rules engines out there, including Red Hat’s very own Drools (aka JBoss BRMS). BREs are also much smarter than this. Thing is, those are massively overengineered for what I’m trying to do, where I really need something much closer to “make” and quite far away from something that needs an entire dedicated system and its own manager to run (although I’d like to find a business rules engine that was small enough to use from the command line).

      • jamesjustjames

        Cool! Well I hope you find what you need. I’m a bit biased in that I try to push puppet to it’s limits. You can even make it do weird things like recursion: http://ttboj.wordpress.com/2012/11/20/recursion-in-puppet-for-no-particular-reason/

        Cheers
        James

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s