Tag Archives: unix

whenjobs — job lists, cancelling, algorithmic cleanup etc

You can now list and cancel jobs:

$ whenjobs --jobs
61 job$1
	running in: /tmp/whenjobs20d88a48f2c4eb0062e1b44ded6d0ae7
	started at: 2012-02-23 22:43:20
62 job$2
	running in: /tmp/whenjobse9e6b93c3ced1967cbf8c5865d6a1ccb
	started at: 2012-02-23 22:43:20
$ whenjobs --cancel 62

You can manually start jobs. Gerd’s ocamlnet makes it almost trivial to add new RPCs between the tool and the daemon, so adding functions like this is simple.

You can put arbitrary OCaml actions into the job script too, so you can run code when a job is cleaned up, and you will (soon) be able to create jobs algorithmically. For example, the standard mailto cleanup lets you send mail containing the output of the job when it finishes.

let from = "me@example.com"
let to_addr = "you@example.com"
let prefix = "hostname "
let script = << # shell script here >>

job (prefix ^ "poll")
cleanup (Whentools.mailto ~from to_addr)
every minute : script

Leave a comment

Filed under Uncategorized

A work in progress: whenjobs — another cron replacement

whenjobs (git repo) is a cron replacement. From the manual page …

Whenjobs is a powerful but simple replacement for cron. It lets you run jobs periodically like cron, but it also lets you trigger jobs to run when user-defined variables are set or change value.

Periodic jobs are written like this:

every 10 minutes :
<<
  # Get the current load average.
  load=`awk '{print $1}' /proc/loadavg`
  whenjobs --set load $load --type float
>>

When-statements let you create jobs that run based on variables set elsewhere:

when load >= 6 :
<<
  mail -s "ALERT: high load average: $load" $LOGNAME < /dev/null
>>

(When statements are "edge-triggered", meaning that this job will only run when the load goes from under 6 to ≥ 6).

The motivation is building things from git automatically. Here is another job script:

Every 10 minutes, get the latest tagged version from the git repository. The variable ‘version’ will be set to something like “v1.2.3”, “v1.2.4”, etc over time as new releases get tagged.

every 10 minutes :
<<
  cd /my/git/repo
  tag=`git-describe --tags`
  whenjobs --set version $tag
>>

When the ‘version’ variable changes (ie. a new release is tagged) try to build it. ‘changes’ is a function that compares the previous value of a variable from when this job last ran with the current value of a variable, and returns true if the previous and current values are different.

when changes version :
<<
  cd /my/git/buildrepo
  git pull
  git reset --hard $version
  ./configure
  make clean all check dist
  whenjobs --set successful_local_build $version
>>

In parallel, build on a remote machine.

when changes version :
<<
  ssh remote ./do_build $version
  whenjobs --set successful_remote_build $version
>>

Only when the new release has been successfully built on local and remote, upload it to the website.

when successful_local_build == version &&
     successful_remote_build == version :
<<
  cd /my/git/buildrepo
  curl -T name-$success.tar.gz ftp://ftp.example.com/upload/
>>

4 Comments

Filed under Uncategorized

What can affect a process?

I’ve been following an interesting thread on fedora-devel which set me thinking. What is the complete list of different things that can affect the way your process runs?

Here’s my list below. If you have any other ideas, post a comment and I’ll keep this list updated.

  1. Environment variables. Obviously there are direct effects, like if $PATH is different then you may end up running different sub-processes. But there are more subtle differences like what happens if the environment is too large or completely empty? Also $LD_* variables can make a big difference to what is inside your process.
  2. ulimits. Too small and your program could fail to allocate memory or fail to open a file.
  3. Signal masks. Often overlooked, but I’ve hit this one: If a signal is masked, your program can behave quite differently. There is a famous bug where SIGPIPE was masked in the whole of Fedora, because some early program (login) was using dbus which promiscuously masked the signal, then login was forking every other program with this signal masked.
  4. Program arguments. You could put this in the “too obvious” class if you want, but consider also argv[0] which might affect your program but not be an immediately visible change.
  5. The PID. I have actually seen this: a program (sshd) was trying to create some lock file, something like /var/lock/sshd.<pid> at boot time in order to ensure only one instance was running. It was consistently failing to start sshd at boot. It took me some time to work out that because the boot was exactly predictable (and thus the PID was always the same), it was falling over its own lock file left from last time the machine was shut down.
  6. The file descriptors. Does the program change behaviour if fds 0, 1, 2 (stdin, stdout, stderr) are not open? How about if other open fds are leaked from the parent process?
  7. Current working directory. Affects what files are opened by relative paths. I guess you could include the chroot here too, but that is quite an obvious change.
  8. Number of other processes. This is like an “unofficial” ulimit, since as normally configured Linux will only allow 32766(?) processes (less PID 0 which is reserved and PID 1 for init).
  9. UIDs and GIDs. If these are very large, Bad Things can happen. External utilities like cpio and tar will fail.
  10. SELinux context. (Suggested by David Malcolm) One thing to note is that the SELinux context of a root login can be different from the context of, say, a daemon started at boot.
  11. Wallclock time or other timers. (Alexander E. Patrakov)
  12. Filesystem journalling mode, filesystem type. (see Bruno Wolff’s comment)

That’s all I can think of for now. Post a comment if you can think of any more.

10 Comments

Filed under Uncategorized

Tip: find out when filesystems get full with virt-df

Another easy tip for the day: Want to receive a warning when one of your virtual machines’ filesystems is filling up? Use virt-df, awk, and cron like this.

Note: I’ve set this up so I can do it as non-root, but that meant I had to add myself to the ‘disk’ group first, so that virt-df can read out the contents of the disks of the virtual machines. There’s one extra thing needed which is to set the LIBVIRT_DEFAULT_URI environment variable to access the global libvirtd socket.

$ export LIBVIRT_DEFAULT_URI='qemu+unix:///system?socket=/var/run/libvirt/libvirt-sock-ro'
$ virt-df | awk 'strtonum(substr($5,0,length($5)-1))>=60 {print}'
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%

You can set the threshold to whatever you want. In the above awk code it’s set to 60%, you’d probably want it higher.

Now put those two commands into a script file somewhere, and create a cronjob. The following one runs daily:

MAILTO=you@example.com
0 1 * * * /usr/local/etc/diskcheck.sh

What do you do when a VM is running out of space? Why, you run virt-resize of course.


Ob-geekiness: The full virt-df output from one of my test hosts. This one is used for a lot of interop testing:

$ virt-df
Filesystem                           1K-blocks       Used  Available  Use%
RHEL6200910210x32:/dev/vda1             198337      36582     151515   19%
RHEL6200910210x32:/dev/vg_rhel6200910210x32/lv_root
                                       9147776    5305876    3377212   59%
RHEL6201002033x64:/dev/vda1             495844      32396     437848    7%
RHEL6201002033x64:/dev/vg_rhel6201002033x64/lv_root
                                      18102140    4097800   13084788   23%
Debian5x64:/dev/debian5x64/home        3555936    1264800    2110504   36%
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Debian5x64:/dev/debian5x64/tmp          309401      10294     283133    4%
Debian5x64:/dev/debian5x64/usr         3539776    1434740    1925224   41%
Debian5x64:/dev/debian5x64/var         1741648     264632    1388544   16%
Debian5x64:/dev/vda1                    233335      47272     173615   21%
Ubuntu910x64:/dev/vda1                 9827520    4180524    5147780   43%
RHEL6Alpha3x64:/dev/vda1                198337      22879     165218   12%
RHEL6Alpha3x64:/dev/vg_rhel6alpha3x64/lv_root
                                       8180152    4174904    3589712   52%
RHEL54Betax64:/dev/VolGroup00/LogVol00
                                      15109112    8695548    5633688   58%
RHEL54Betax64:/dev/vda1                 101086      12449      83418   13%
F10x32:/dev/VolGroup00/LogVol00        9191640    3083532    5642856   34%
F10x32:/dev/vda1                        194442      20706     163697   11%
F12x64:/dev/vda1                        198337      22782     165315   12%
F12x64:/dev/vg_f12x64/lv_root          9115576    4396520    4256004   49%
F13Rawhidex64:/dev/vda1                 198337      60031     128066   31%
F13Rawhidex64:/dev/vg_f13rawhide/lv_root
                                       9115576    6018664    2633860   67%
F12x64preview:/dev/vda1                 198337      22782     165315   12%
F12x64preview:/dev/vg_f12x64/lv_root   9115576    4781384    3871140   53%
Win2003x32:/dev/vda1                  20956760    3053092   17903668   15%
VSphere:/dev/vda1                     31447204    8159028   23288176   26%
Windows7x32:/dev/vda1                   102396      24712      77684   25%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda1                   102396      24704      77692   25%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%
CentOS5x32:/dev/VolGroup00/LogVol00    9014656    4069840    4479512   46%
CentOS5x32:/dev/vda1                    101086      36210      59657   36%

Update There is a small but important bug in this code. Please see the fixed version.

3 Comments

Filed under Uncategorized

pod2man, a great way to write Unix man pages

I write quite a bit of code in Perl, and more in C, OCaml and other strange languages, but for writing man pages there’s one thing I always use: POD.

Although POD comes from a Perl background, you can really use it to document any program. All you need is a “foo.pod” file starting with this template:

=encoding utf8

=head1 NAME

foo - Foobar your bazfiles

=head1 SYNOPSIS

 foo [bazfile]

=head1 DESCRIPTION

Write your description here.
Use B<for bold>, I<for italic> and C<for inline code>,
and a blank line to separate paragraphs.

=head1 SEE ALSO

L<bar(1)>, L<baz(5)>

See perlpod for the full markup, but the advantage of POD is the markup is deliberately limited and simple.

As well as turning your POD file into a manpage using pod2man, you can also turn them into web pages. The default look for POD-generated web pages is pretty sucky. Nevertheless with only a tiny bit of styling you can make them look great, which is how I make all the libguestfs web pages. Examples: guestfish(1), guestfs(3).

The only thing I haven’t worked out yet with the web pages is how to get inter-manpage links to work properly.

Nevertheless … POD is great and simple. Use it!

4 Comments

Filed under Uncategorized

Tip: ^ and ! in the shell

I’m astounded. So don’t people know about using ^ and ! in the shell?!? Last week I watched an experienced Linux user carefully hit the ↑ cursor key to get a previous line of history, then ←&→ just to make a simple edit!

Here’s my 30 second guide:

!! Repeat the previous command. Example:

$ ls
bin  d  Desktop  rpmbuild  tmp
$ !!
ls
bin  d  Desktop  rpmbuild  tmp
!-2 (etc) Repeat the command 2 previously (so !! = !-1), or for any number previously. This is the most useful I think.
^foo^bar Replace foo with bar in the previous command. eg:

$ ls -l /etx/httpd/conf.d/local.conf
ls: cannot access /etx/httpd/conf.d/local.conf: No
such file or directory
$ ^etx^etc
ls -l /etc/httpd/conf.d/local.conf 
-rw-r--r-- 1 root root 76 2009-07-16 14:59 /etc/httpd/conf.d/local.conf
!foo Run the most recent command that started foo, eg:

$ !ls
ls -l /etc/httpd/conf.d/local.conf 
-rw-r--r-- 1 root root 76 2009-07-16 14:59 /etc/httpd/conf.d/local.conf

Today I discovered that these are called event designators. When I first saw someone using them it was on an Apollo workstation circa 1991, and I pretty quickly picked these up myself.

18 Comments

Filed under Uncategorized