For more half-baked ideas, see the ideas tag.
These needs a better name, but the idea is simple. Take a screenshot of your guest’s graphical console, and OCR it in order to diagnose the VM’s state, any errors, etc.
Case 1: VM error on boot
Case 2: VM has reached graphical login screen
OCR would analyze the screenshot, find everything that looks like text and convert it to text, allowing for simple regular expressions to be used to identify VM state.
Notes:
- Strictly speaking in the first case you could read the screen text directly out of the text framebuffer, but OCRing is more general and handles the second case.
- virt-dmesg could also solve the first case, assuming virt-dmesg wasn’t broken.
Richard WM Jones 
I’ve also thought about this recently. Initial testing using gocr and tesseract didn’t lead to any promising results.
But definetly using OCR might be a better alternative (because it’s text and regex based) then using computer vision where you need to maintain templates to match anything specififc on the screen.
We (oVirt Node) are using the first approach for functional test of our TUI.
Well, I learn at least that there exists open source OCR, so that’s a good thing …
Yeah, this sort of thing would rock for Oz. If there is a guest VM failure, Oz already takes a screenshot[1] and puts it on disk for the user, but doing an OCR on the resulting image would make the error messages much better.
[1] Using the libvirt screenshot mechanism.
We are doing this sort of automated ‘did it boot ?’, ‘did login work ?’ testing in ostree (http://build.gnome.org/#gnome-ostree) by looking for expected messages in the journal. Using message ids lets us do this pretty reliably.
How do you get to the journal in the guest? (Or does ostree use containers and chrooted filesystems?)
We use libguestfs to pop open the guest before booting it and add a custom service which exports the journal over a virtio-serial channel:
https://git.gnome.org/browse/gnome-ostree/tree/src/js/libqa.js#n186
https://git.gnome.org/browse/gnome-ostree/tree/src/tests/gnome-ostree-export-journal-to-serialdev
https://git.gnome.org/browse/gnome-ostree/tree/src/tests/gnome-ostree-export-journal-to-serialdev.service
Cool, I didn’t know that.