Tag Archives: awk

Quick tip: Timing things in a shell script

We all know about the ‘time’ command. That’s great if you have one command that you want to time.

But how about if you want to time several actions together in a shell script, eg:

#!/bin/sh -
start_the_clock
A
B
C
how_long_so_far
D
E
stop_the_clock

The ‘time’ command isn’t so useful for this, unless you want to stick ‘time’ in front of each action A, B, C, D & E and add them up in your head.

But there’s a nice little date/awk trick you can use for this:

#!/bin/sh -
date +%s.%N > /tmp/times
A
B
C
date +%s.%N >> /tmp/times
D
E
date +%s.%N >> /tmp/times
awk '{ if (!start) { start = $1 } else { print $1-start } }' < /tmp/times

which would print something out like this:

25.2957   # the time that actions A-C took
128.529   # the time that actions A-E took

(Thanks Jim Meyering for suggesting date +%s.%N)

Advertisements

1 Comment

Filed under Uncategorized

Tip: Use ‘du’ to see which subdirectories use most space

In the previous tip I showed how to use the du command to display the space used by a single directory. Often you want to find out which subdirectory is using the most space, and using the glob command and some awk scripting we can do that:

$ echo glob du '/usr/*' |
  guestfish --ro -i Fedora12 |
  awk '{if (and (NR, 1)) dir = $2; else print $1,"\t",dir; }' |
  sort -nr
1824432  /usr/share/
856092 	 /usr/lib64/
362864 	 /usr/bin/
244908 	 /usr/lib/
140824 	 /usr/include/
67948 	 /usr/libexec/
44072 	 /usr/src/
29720 	 /usr/sbin/
568 	 /usr/kerberos/
136 	 /usr/local/
4 	 /usr/tmp/
4 	 /usr/games/
4 	 /usr/etc/

Replace /usr/* with /home/* to see whose home directory is hogging all the space.

5 Comments

Filed under Uncategorized

Tip: find out when filesystems get full with virt-df (working version!)

Thanks go to “S” for pointing out the mistake in my last posting. It’s easy to fix though.

When virt-df is run with the –csv option, it outputs a Comma-Separated Values format spreadsheet. CSV is not totally straightforward to parse and so you should leave it up to a specialized tool. Of course I recommend my own tool csvtool which is available in Fedora, Debian and Ubuntu.

The basic plan is to use csvtool to drop the first row (the column headings row), and pull out columns 1, 2 and 6 from the rest of the output. Those columns are respectively the virtual machine name, the filesystem, and the %usage:

$ virt-df --csv | csvtool drop 1 - | csvtool col 1-2,6 -
Debian5x64,/dev/debian5x64/home,35.6%
Debian5x64,/dev/debian5x64/root,60.9%
Debian5x64,/dev/debian5x64/tmp,3.3%
Debian5x64,/dev/debian5x64/usr,40.5%
Debian5x64,/dev/debian5x64/var,15.2%
Debian5x64,/dev/vda1,20.3%
RHEL620100329n0x64,/dev/vda1,7.4%
RHEL620100329n0x64,/dev/vg_rhel620100329n0/lv_root,16.7%
Ubuntu910x64,/dev/vda1,42.5%
[...snipped...]

Adding the previous awk test gives the desired output, all filesystems with ≥ 60% usage:

$ virt-df --csv |
    csvtool drop 1 - |
    csvtool col 1-2,6 - |
    csvtool readable - |
    awk 'strtonum(substr($3,0,length($3)-1))>=60 {print}'
Debian5x64         /dev/debian5x64/root            60.9% 
F13Rawhidex64      /dev/vg_f13rawhide/lv_root      66.0% 
Windows7x32        /dev/vda2                       74.6% 
Windows7x64        /dev/vda2                       90.7% 

This can be placed in a script or cron job.

1 Comment

Filed under Uncategorized

Tip: find out when filesystems get full with virt-df

Another easy tip for the day: Want to receive a warning when one of your virtual machines’ filesystems is filling up? Use virt-df, awk, and cron like this.

Note: I’ve set this up so I can do it as non-root, but that meant I had to add myself to the ‘disk’ group first, so that virt-df can read out the contents of the disks of the virtual machines. There’s one extra thing needed which is to set the LIBVIRT_DEFAULT_URI environment variable to access the global libvirtd socket.

$ export LIBVIRT_DEFAULT_URI='qemu+unix:///system?socket=/var/run/libvirt/libvirt-sock-ro'
$ virt-df | awk 'strtonum(substr($5,0,length($5)-1))>=60 {print}'
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%

You can set the threshold to whatever you want. In the above awk code it’s set to 60%, you’d probably want it higher.

Now put those two commands into a script file somewhere, and create a cronjob. The following one runs daily:

MAILTO=you@example.com
0 1 * * * /usr/local/etc/diskcheck.sh

What do you do when a VM is running out of space? Why, you run virt-resize of course.


Ob-geekiness: The full virt-df output from one of my test hosts. This one is used for a lot of interop testing:

$ virt-df
Filesystem                           1K-blocks       Used  Available  Use%
RHEL6200910210x32:/dev/vda1             198337      36582     151515   19%
RHEL6200910210x32:/dev/vg_rhel6200910210x32/lv_root
                                       9147776    5305876    3377212   59%
RHEL6201002033x64:/dev/vda1             495844      32396     437848    7%
RHEL6201002033x64:/dev/vg_rhel6201002033x64/lv_root
                                      18102140    4097800   13084788   23%
Debian5x64:/dev/debian5x64/home        3555936    1264800    2110504   36%
Debian5x64:/dev/debian5x64/root         329233     200395     111840   61%
Debian5x64:/dev/debian5x64/tmp          309401      10294     283133    4%
Debian5x64:/dev/debian5x64/usr         3539776    1434740    1925224   41%
Debian5x64:/dev/debian5x64/var         1741648     264632    1388544   16%
Debian5x64:/dev/vda1                    233335      47272     173615   21%
Ubuntu910x64:/dev/vda1                 9827520    4180524    5147780   43%
RHEL6Alpha3x64:/dev/vda1                198337      22879     165218   12%
RHEL6Alpha3x64:/dev/vg_rhel6alpha3x64/lv_root
                                       8180152    4174904    3589712   52%
RHEL54Betax64:/dev/VolGroup00/LogVol00
                                      15109112    8695548    5633688   58%
RHEL54Betax64:/dev/vda1                 101086      12449      83418   13%
F10x32:/dev/VolGroup00/LogVol00        9191640    3083532    5642856   34%
F10x32:/dev/vda1                        194442      20706     163697   11%
F12x64:/dev/vda1                        198337      22782     165315   12%
F12x64:/dev/vg_f12x64/lv_root          9115576    4396520    4256004   49%
F13Rawhidex64:/dev/vda1                 198337      60031     128066   31%
F13Rawhidex64:/dev/vg_f13rawhide/lv_root
                                       9115576    6018664    2633860   67%
F12x64preview:/dev/vda1                 198337      22782     165315   12%
F12x64preview:/dev/vg_f12x64/lv_root   9115576    4781384    3871140   53%
Win2003x32:/dev/vda1                  20956760    3053092   17903668   15%
VSphere:/dev/vda1                     31447204    8159028   23288176   26%
Windows7x32:/dev/vda1                   102396      24712      77684   25%
Windows7x32:/dev/vda2                 10381308    7740852    2640456   75%
Windows7x64:/dev/vda1                   102396      24704      77692   25%
Windows7x64:/dev/vda2                 10381308    9417676     963632   91%
CentOS5x32:/dev/VolGroup00/LogVol00    9014656    4069840    4479512   46%
CentOS5x32:/dev/vda1                    101086      36210      59657   36%

Update There is a small but important bug in this code. Please see the fixed version.

3 Comments

Filed under Uncategorized