New project: wrappi: an API for Linux

Over the holidays I started a new project called “wrappi”.

The name is a play on “wrapper” and “API”. And a play on what has been discussed for a very long time inside Red Hat — an API for everything that Red Hat Enterprise Linux does. A “rh-api”, if you will.

Wrappi aims to wrap up everything you can do with a Linux box:

  • many POSIX functions for creating files, mkdir, reading and writing files, etc
  • many command line tools, mkfs, fsck, parted, pvcreate, cp etc
  • block devices, firewall rules, services, processes etc
  • reboot, shutdown
  • a way to read and modify all configuration files
  • access to all the performance stats

We take all of that and present it back to you as an API:

  • available from C and a dozen other programming languages
  • available in a more powerful shell than bash, something like Microsoft’s PowerShell
  • accessible remotely over ssh, HTTP, XML-RPC, dbus, and a dozen other protocols

The aim is that from a PowerShell-like interface (or your own program) you could control and maintain 100s of Linux machines remotely. Like puppet/chef, but at a lower level.

Here is what a C program using the API might look like.

So this is ambitious.

An API that did all of the above might run to 10,000 different calls. To make this feasible to maintain and implement, we have to be able to generate the code for just about everything.

Each API call starts with a description. Because we need to write this description for 10,000 calls, it’s best to keep this short. In the best case, just a single line would be required, but in some cases it’ll be more complicated:

entry_point void mkdir (pathname path, fileperm perm)
system_object block_device
  dir_list "/sys/block/[hsv]d[a-z]*"
entry_point
struct timeval gettimeofday ()
<<
  int r;
  struct timeval tv;

  r = gettimeofday (&tv, NULL);
  if (r == -1) {
    set_error_errno ("gettimeofday");
    return NULL;
  }
  ret->tv_sec = tv.tv_sec;
  ret->tv_usec = tv.tv_usec;
  return ret;
>>
includes ["sys/time.h"]

From that metadata we can generate automatically everything we need, all the language bindings, all the remote access code, the implementation.

When a new shiny-thing comes along (“JSON-powered Enterprise GObject”) we can simply add a new generator backend, and we’ll support the whole API through that.

If you’ve followed my blog, you’ll know that this looks a lot like the libguestfs generator, and in many ways this is the generator done right.

There’s lots of code in the git repo. If you want to find out more, follow the blog, or ask questions below!

9 Comments

Filed under Uncategorized

9 responses to “New project: wrappi: an API for Linux

  1. Frank Ch. Eigler

    It’s not obvious why this sort of thing would be useful, or how it would deal with nontrivialities (e.g., security, the effort required to maintain 10000 wrapper metadata bits). Could you say a little more about the motivation?

    • rich

      Good point — I edited the post to add the motivation for this.

      • Frank Ch. Eigler

        Have you thought about the necessity to designate or design certain particular levels of abstraction for the various services to be exposed this way?

        For example, is it a good idea to expose e.g. both remote file operations, and firewall-manipulation operations (which map to file operations too). Wouldn’t you have to declare somehow the possible interactions/conflicts between the various APIs?

        For another example, what about a management client that needs to support more than a solitary version of the OS installed on that cluster? If these wrappers provide no meaningful abstractions, how would the client avoid having to have hard-coded logic for each RHELX.Y, FedoraZ etc. variant?

  2. I’d be interested if you could write more about the target audience and the sort the programs that you envision being written in this API. To me wrappi seems like a “high level”, synchronous API targeted at sysadmins. (“Like puppet/chef, but at a lower level”) Hopefully the end result will be easier to use then the Windows Management Interface stuff.

    An automated testsuite for the the functions you listed being run regularly on the Fedora virt stack will likely trigger some interesting bugs.

    For firewall rules do you envision more work being done at a lower level so that NetworkManager and libvirt can manage their firewall rules in a modular fashion etc?

  3. Looking at the example C program, it looks like you envision wrapping the client side of remote transports. Is that the case? Would you see wrapping D-bus, QMF, whatever, all in this same API?

    I can see this approach making a lot of sense on the server side of the connection (generate a D-Bus agent, a QMF agent, etc), but the benefits aren’t quite as clear on the client side.

    • rich

      It doesn’t make a lot of sense to call “wrap_reboot” if you can just call “reboot” (ie. in the local case). But as you say it makes more sense if you wanted to write an agent and had to have that agent called over (insert protocol here) that instead of writing the agent and all that code, you simply assemble the right bits of wrappi and the right bits of the generator to build an agent for you. I don’t want to get stuck on particular protocols, but definitely dbus, QMF, XML-RPC are on the cards. A simple XDR-based protocol that works over ssh is very nearly working already (end of today hopefully).

  4. Further to Russell’s comment, the example C program gives the impression that the protocol abstraction will limit remote communication to the lowest common denominator – i.e. synchronous and serial. Do you have a plan for scaling this up to “100s of Linux machines”?

    A number of projects, such as MCollective, exist specifically to avoid calling ssh in a for loop (see ). Protocols like QMF are able to dispatch RPC in parallel (and, if necessary, asynchronously), but it seems like these capabilities would be neutralised by this abstraction?

    • rich

      You’d have a different handle for each of your hundred Linux machines, and it’d be up to your program to call them in serial or in parallel as required. Retries and failures would have to be handled by the underlying protocol (QMF or whatever). Since the API is fully generated we can also add the ability to dispatch partial calls and have an event loop.

  5. Tibco Hawk is a similar api. Commerical product, runs on a messaging system called Rendezvous.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s