Tag Archives: hivex

Tip: Another way to get the IP address of a virtual machine

In this earlier post several ways were discussed of getting the IP address of a virtual machine (specifically if the VM is acquiring a different address each time from DHCP).

Another way is to use virt-cat or virt-win-reg to grab the information out of the virtual machine itself.

When a VM acquires an IP address from DHCP then it writes the address to a log file, or in the Windows case updates the Windows Registry. The idea is simply that we’ll read the information from the VM’s log files or the Registry.

The details vary depending on the precise guest type, and probably we should wrap this up in a nice virt tool. But here’s how you would do it for Fedora/RHEL and Windows guests:

# virt-cat RHEL60x64 /var/log/messages | grep 'dhclient.*bound to'
Mar 30 19:56:22 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1527 seconds.
Mar 30 20:21:49 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1375 seconds.
Mar 30 20:44:44 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1287 seconds.
Mar 30 21:06:11 rhel60x64 dhclient: bound to 192.168.122.220 -- renewal in 1461 seconds.
# virt-win-reg Win7x32 \
  'HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\Tcpip\Parameters' | \
  grep DhcpIPAddress
"DhcpIPAddress"=hex(1):31,00,39,00,32,00,2e,00,31,00,36,00,38,00,2e,00,31,00,32,00,32,00,2e,00,31,00,37,00,38,00,00,00

In the Windows case you need to note that ControlSet001 isn’t always the right control set — consult this note in the virt-win-reg man page to find the correct way to do this. Furthermore virt-win-reg prints out the hex-encoded UTF16-LE string, which requires a little bit of conversion to produce a printable string (192.168.122.178 in this instance).

Update: To add to all the other methods from the previous post, and the method described above, Eric Blake also points out that nwfilter can sniff IP addresses used by VMs.

2 Comments

Filed under Uncategorized

Tip: List services in a Windows guest

The lengthy program after the fold uses libguestfs and hivex to list out the services from a Windows guest. You point it to a Windows guest and it will produce a rather long list, like this:

# ./services.pl WindowsGuest
[...]
SysMain:
  Path: %systemroot%\system32\svchost.exe -k LocalSystemNetworkRestricted
  Start flag: autoload
  Service type: Win32 service using svchost
  Error control: ignore
TabletInputService:
  Path: %SystemRoot%\System32\svchost.exe -k LocalSystemNetworkRestricted
  Group: PlugPlay
  Start flag: load on demand
  Service type: Win32 service using svchost
  Error control: normal
TapiSrv:
  Path: %SystemRoot%\System32\svchost.exe -k NetworkService
  Start flag: load on demand
  Service type: Win32 service using svchost
  Error control: normal
TBS:
  Path: %SystemRoot%\System32\svchost.exe -k LocalServiceAndNoImpersonation
  Start flag: load on demand
  Service type: Win32 service using svchost
  Error control: normal
Tcpip:
  Path: System32\drivers\tcpip.sys
  Group: PNP_TDI
  Start flag: boot loader
  Service type: kernel device driver
  Error control: normal
TCPIP6 (Microsoft IPv6 Protocol Driver):
  Microsoft IPv6 Protocol Driver
  Path: system32\DRIVERS\tcpip.sys
  Start flag: load on demand
  Service type: kernel device driver
  Error control: normal
tcpipreg (TCP/IP Registry Compatibility):
  Provides compatibility for legacy applications which interact with TCP/IP through the registry. If this service is stopped, certain applications may have impaired functionality.
  Path: System32\drivers\tcpipreg.sys
  Start flag: autoload
  Service type: kernel device driver
  Error control: normal
[...]

Continue reading

2 Comments

Filed under Uncategorized

Use hivex from Python to read and write Windows Registry “hive” files

I added Python bindings to hivex today.

Here is an example using Python, libguestfs and hivex to download the user preferences registry from a Windows virtual machine and print out the Internet Explorer start page for a particular user. When you run it, it should print out something like:

User rjones's IE home page is http://go.microsoft.com/fwlink/?LinkId=69157

This example shows downloading and printing values, but libguestfs and hivex can also be used to make changes (but not to live guests).

#!/usr/bin/python

import guestfs
import hivex

# The name of a Windows virtual machine on this host.  This
# example script makes some assumptions about the registry
# location and contents which only apply on Windows Vista
# and later versions.
windows_domain = "Win7x32"

# Username on the Windows VM.
username = "rjones"

# Use libguestfs to download the HKEY_CURRENT_USER hive.
g = guestfs.GuestFS ()
g.add_domain (windows_domain, readonly=1)
g.launch ()

roots = g.inspect_os ()
root = roots[0]
g.mount_ro (root, "/")

path = "/users/%s/ntuser.dat" % username
path = g.case_sensitive_path (path)
g.download (path, "/tmp/ntuser.dat")

# Use hivex to pull out a registry key.
h = hivex.Hivex ("/tmp/ntuser.dat")

key = h.root ()
key = h.node_get_child (key, "Software")
key = h.node_get_child (key, "Microsoft")
key = h.node_get_child (key, "Internet Explorer")
key = h.node_get_child (key, "Main")

val = h.node_get_value (key, "Start Page")
start_page = h.value_value (val)
#print start_page

# The registry key is encoded as UTF-16LE, so reencode it.
start_page = start_page[1].decode ('utf-16le').encode ('utf-8')

print "User %s's IE home page is %s" % (username, start_page)

9 Comments

Filed under Uncategorized

Arch Linux PKGBUILD for hivex

# Contributor: Richard W.M. Jones
pkgname=hivex
pkgver=1.2.3
pkgrel=1
pkgdesc="Windows Registry hive extraction library"
arch=('i686' 'x86_64')
license=('LGPL')
url="http://libguestfs.org/"
depends=('perl' 'readline' 'libxml2' 'ocaml' 'perl-io-stringy')
makedepends=('gcc' 'readline' 'libxml2' 'ocaml' 'perl-io-stringy')
# missing: ocaml-findlib
options=(!libtool)
source=(http://libguestfs.org/download/hivex/$pkgname-$pkgver.tar.gz)
md5sums=('606dd17876be527bed68f93c6468b453')

build() {
  cd "${srcdir}/${pkgname}-${pkgver}"
  ./configure --prefix=/usr
  make
  make DESTDIR=${pkgdir} install
}

4 Comments

Filed under Uncategorized

Windows SAM and hivex

On Windows, the file C:\windows\system32\config\SAM contains the users and passwords known to the local machine. hivex can process this file to reveal the usernames and password (hashes):

$ virt-win-reg WinGuest HKLM\\SAM > sam.reg

For each local user you’ll see a key like this:

[HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users\Names\rjones]
@=hex(3e9):

With typical technical brilliance Microsoft developers have written a zero-length key with the type field (0x3e9) overloaded as a key to use in another part of the registry:

[HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users\000003E9]
"F"=hex(3):omitted...
"UserPasswordHint"=str(3):"usual"
"V"=hex(3):omitted...

(Apparently the number 0x3e9 is called the “RID” in Microsoft parlance).

My password hint is the “usual”. The “F” key is a dumped C structure containing the last login date amongst other things. The “V” key is another C structure containing my full name, home directory, the password hash and a bunch of other stuff.

With a bit of effort it looks like you could read and even modify these entries.

1 Comment

Filed under Uncategorized

Tip: Install a service in a Windows VM

Previously I discussed how to get a script to run the first time a user logs in. This tip goes further and demonstrates how to install a service into a Windows VM using guestfish, virt-win-reg and a new open source program written by my colleague Yuval Kashtan called RHSrvAny1.

First, compile RHSrvAny from source. You can do this using our completely free Fedora Windows cross-compiler stack. Just:

# yum install mingw32-gcc

Clone the RHSrvAny git repo and compile it:

$ mingw32-configure
$ make

Second we’ll copy the files we need into the Windows guest. Note: The Windows VM must be shut off.

# guestfish -i Windows7x64
Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for a list of commands
      'man' to read the manual
      'quit' to quit the shell

><fs> upload RHSrvAny/rhsrvany.exe /rhsrvany.exe
><fs> upload test.exe /test.exe
><fs> exit

“test.exe” is a little program I wrote which writes the date into C:\TEST.LOG but you can also use the batch file from the last tip or any JScript or VBScript you happen to have (via cscript.exe).

Third we need to add some Windows Registry keys to tell Windows about the new service:

# cat service.reg
[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\RHSrvAny]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"ImagePath"="c:\\rhsrvany.exe"
"DisplayName"="RHSrvAny"
"ObjectName"="LocalSystem"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services\RHSrvAny\Parameters]
"CommandLine"="c:\\test.exe"
"PWD"="c:\\Temp"
# virt-win-reg --merge Windows7x64 service.reg

The magic numbers in the registry entries let you do things like boot with the service disabled. See this MSDN article.

Edit: See Yuval’s comment about alternatives to using "ObjectName"="LocalSystem".

Now boot your Windows guest, and observe the log file to prove that test.exe was run, and/or look at the list of services in the control panel.

><fs> cat /TEST.LOG
Hello
Thu Apr 29 18:39:13 2010

1 Actually you could install any service, but I’m using RHSrvAny because it can turn ordinary Windows programs and scripts into services. It takes care of the Windows “Service Control Protocol” for us.

10 Comments

Filed under Uncategorized

Tip: Get a Windows VM to run a batch file at boot

With the virt-win-reg tool built on top of libguestfs and hivex it’s now relatively straightforward to modify a Windows virtual machine so that it runs a batch file, script or program at next boot.

Note: The Windows VM must be shut down before you attempt this.

The plan is that we upload the batch script to some place in the VM, and then add a “RunOnce” key in the Windows Registry (explained in this MSDN article and this article). First let’s just take a look at what’s in the key. In most cases it will be empty:

# virt-win-reg Windows7x64 \
    'HKLM\Software\Microsoft\Windows\CurrentVersion\RunOnce'
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\RunOnce]

Now we’ll prepare our batch file and upload it:

# cat test.bat
ECHO HELLO > C:\TEST.LOG
TIME /T >> C:\TEST.LOG
# guestfish -i Windows7x64

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for a list of commands
      'man' to read the manual
      'quit' to quit the shell

><fs> upload test.bat /test.bat
><fs> ^D

And finally we modify the RunOnce registry key:

# virt-win-reg --merge Windows7x64
[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce]
"Test"="c:\\test.bat"
^D

One potential gotcha: You must be running hivex ≥ 1.2.2.

Now you can boot your Windows guest and check that the script runs after the user logs in. Look for the file C:\TEST.LOG:

><fs> cat /TEST.LOG
HELLO 
09:21

Because we’re using the RunOnce key, the script will run just one time. If you want it to run every time, use the Run key.

Now, how do we make the script work without the user needing to log in? (Clue: The answer is not RunServicesOnce — this does not work in Windows 7). What’s surprising (coming from a Linux background) is the huge amount of incomplete, contradictory and simply false information contained in MSDN about this topic.

9 Comments

Filed under Uncategorized

RHEL 6 public beta 1

Old Red Hat Linux 6.2 boxWe released the first public beta of Red Hat Enterprise Linux 6 today. You might be able to get it from the official Red Hat FTP site, but it was giving “421 There are too many connected users” errors just now. You can grab this unofficial mirror of the 64 bit DVD ISO instead. The official press release is here.

There are probably a fair few bugs — although not as many as the internal alphas I was playing with last year!

The highlights from my point of view: supported stable libguestfs, libguestfs tools and V2V, OCaml compiler (not supported), virt-top, and hivex.

Edit: Excellent interview with Tim Burke in The Register which also covers the main new features.

6 Comments

Filed under Uncategorized

Use hivex to unpack a Windows Boot Configuration Data (BCD) hive

Thanks to “TJ” for tipping me off about another use of the Registry “hive” format in recent versions of Windows.

There are scant details available, but if you have a version of Windows Vista or later, then the boot loader is no longer configured through a plain text file (“BOOT.INI”) but via a binary blob. Microsoft provides a tool called “BCDEDIT.EXE” that you are supposed to use to edit this, but the blob is a hive so you can use hivex to display or modify it.

We first use guestfish to download the blob:

$ guestfish --ro -a /dev/vg_trick/Windows7x64 -m /dev/sda1

Welcome to guestfish, the libguestfs filesystem interactive shell for
editing virtual machine filesystems.

Type: 'help' for help with commands
      'quit' to quit the shell

><fs> ll /
total 392
drwxrwxrwx  1 root root   4096 Dec 15 04:48 .
dr-xr-xr-x 20 root root      0 Mar 30 13:30 ..
-rwxrwxrwx  1 root root   8192 Dec 15 12:47 BOOTSECT.BAK
drwxrwxrwx  1 root root   4096 Dec 15 12:47 Boot
drwxrwxrwx  1 root root      0 Dec 15 04:48 System Volume Information
-rwxrwxrwx  1 root root 383562 Jul 13  2009 bootmgr
><fs> ll /Boot/
total 596
drwxrwxrwx 1 root root   4096 Dec 15 12:47 .
drwxrwxrwx 1 root root   4096 Dec 15 04:48 ..
-rwxrwxrwx 1 root root  24576 Mar 25 12:25 BCD
-rwxrwxrwx 1 root root  21504 Mar 25 12:25 BCD.LOG
-rwxrwxrwx 2 root root      0 Dec 15 12:47 BCD.LOG1
-rwxrwxrwx 2 root root      0 Dec 15 12:47 BCD.LOG2
-rwxrwxrwx 1 root root  65536 Dec 15 12:47 BOOTSTAT.DAT
[snipped]
><fs> download /Boot/BCD /tmp/BCD
><fs> ^D

Then we can dump the contents out using hivexregedit. (We could also browse the contents with hivexsh).

$ hivexregedit --export /tmp/BCD '\' > /tmp/BCD.reg

In typical Microsoft style, the contents themselves are obscure, consisting of plenty of subkeys that look like this:

[\Objects\{1afa9c49-16ab-4a5c-901b-212802da9460}\Elements\14000006]
"Element"=hex(7):7b,00,37,00,65,00,61,00,32,00,65,00,31,00,61,00,63,\
  00,2d,00,32,00,65,00,36,00,31,00,2d,00,34,00,37,00,32,00,38,00,2d,00,\
  61,00,61,00,61,00,33,00,2d,00,38,00,39,00,36,00,64,00,39,00,64,00,30,\
  00,61,00,39,00,66,00,30,00,65,00,7d,00,00,00,00,00

(Note that “type 7” is a list of strings, and the whole thing is encoded in UTF-16LE, so this requires some further work to parse).

There’s scope here to extend virt-inspector to understand this stuff, or even to write a BCDEDIT-style tool to modify the way Window VMs boot. Apparently the current BCDEDIT tool is half-arsed, so here’s another opportunity to beat Microsoft’s own tooling.

4 Comments

Filed under Uncategorized

Why the Windows Registry sucks … technically

It’s quite popular to bash the Windows Registry in non-technical or lightly technical terms. I’ve just spent a couple of weeks reverse engineering the binary format completely for our hivex library and shell which now supports both reading and writing to the registry. So now I can tell you why the Registry sucks from a technical point of view too.

1. It’s a half-arsed implementation of a filesystem

It’s often said that the Registry is a “monolithic file”, compared to storing configuration in lots of discrete files like, say, Unix does under /etc. This misses the point: the Registry is a filesystem. Sure it’s stored in a file, but so is ext3 if you choose to store it in a loopback mount. The Registry binary format has all the aspects of a filesystem: things corresponding to directories, inodes, extended attributes etc.

The major difference is that this Registry filesystem format is half-arsed. The format is badly constructed, fragile, endian-specific, underspecified and slow. The format changes from release to release of Windows. Parts are undocumented, seemingly to the Windows developers themselves (judging by the NT debug symbols that one paper has reproduced). Parts of the format waste space, while in other parts silly “optimizations” are made to save a handful of bytes (at the cost of making access much more complex).

2. Hello Microsoft programmers, a memory dump is not a file format

The format is essentially a dump of 32 bit C structures in a C memory heap. This was probably done originally for speed, but it opens the format to all sorts of issues:

  1. You can hide stuff away in unused blocks.
  2. You can create registries containing unreachable blocks or loops or pointers outside the heap, and cause Windows to fail or hang (see point 3).
  3. It’s endian and wordsize specific.
  4. It depends on the structure packing of the original compiler circa 1992.

3. The implementation of reading/writing the Registry in Windows NT is poor

You might expect, given how critical the Registry is to Window’s integrity, that the people who wrote the code that loads it would have spent a bit of time thinking about checking the file for consistency, but apparently this is not done.

  1. All versions of Windows tested will simply ignore blocks which are not aligned correctly.
  2. Ditto, will ignore directory entries which are not in alphabetical order (it just stops reading at the first place it finds a subdirectory named B > next entry A).
  3. Ditto, will ignore file entries which contain various sorts of invalid field.

The upshot of this is you can easily hide stuff in the Registry binary which is completely invisible to Windows, but will be apparent in other tools. From the point of view of other tools (like our hivex tool) we have to write exactly the same bits that Windows would write, to be sure that Windows will be able to read it. Any mistakes we make, even apparently innocuous ones, are silently punished.

Compare this to using an established filesystem format, where everyone knows the rules, and consistency (eg. fsck/chkdsk) matters.

Writing sucks too, because the programmers don’t correctly zero out fields, so you’ll find parts (particularly the Registry header) which contain random bits of memory, presumably kernel memory, dumped into the file. I didn’t find anything interesting there yet …

I also found Registries containing unreachable blocks (and not, I might add, ones which I’d tried modifying). I find it very strange that relatively newly created Windows 7 VMs which don’t have any sort of virus infection, have visible Registry corruption.

4. Types are not well specified

Each registry field superficially is typed, so REG_SZ is a string, and REG_EXPAND_SZ is, erm, also a string. Good, right? No, because what counts as a “string” is not well-defined. A string might be encoded in 7 bit ASCII, or UTF-16-LE. The only way to know is to know what versions of Windows will use the registry.

Strings are also stored in REG_BINARY fields (in various encodings), but also raw binary data is stored in these fields.

Count yourself lucky if you only access official Microsoft fields though because some applications don’t confine themselves to the published types at all, and just use the type field for whatever they feel like.

And what’s up with having REG_DWORD (little-endian of course) and REG_DWORD_BIG_ENDIAN, and REG_QWORD, but no REG_QWORD_BIG_ENDIAN?

5. Interchange formats are not well specified

A critical part of installing many drivers is making registry edits, and for this a text format (.REG) is used along with the REGEDIT program. The thing is though that the .REG format is not well-specified in terms of backslash escaping. You can find examples of .REG files that have both:

"Name"="\Value"

and

"Name"="\\Value"

In addition the encoding of strings is again not specified. It seems to depend on the encoding of the actual .REG file, as far as anyone can tell. eg. If your .REG file itself is UTF-16-LE, then REGEDIT will encode all strings you define this way. Presumably if you transfer the .REG file to a system that changes the encoding, then you’ll get different results when you load the registry.

6. The Registry arrangement is a mess

Take a look at this forensic view of interesting Registry keys (PDF). List of mounted drives? HKEY_LOCAL_MACHINE\SYSTEM\MountedDevices. But what the user sees is stored in HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Explorer\MountPoints2\CPC\Volume\. Unless you mean USB devices which might be in the above list, or in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\USBSTOR. And the entries in those lists are by no means obvious — containing impenetrable binary fields and strange Windows paths.

If you browse through the Registry some time you’ll see it’s a giant accreted mess of non-standardized, overlapping information stored in random places. Some of it is configuration, much of it is runtime data. This is a far cry from /etc/progname.conf in Linux.

7. The Registry is a filesystem

Back to point 1, the Registry is a half-assed, poor quality implementation of a filesystem. Importantly, it’s not a database. It should be a database! It could benefit from indices to allow quick lookups, but instead we have to manually and linearly traverse it.

This leads to really strange Registry keys like:

\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00000000

which are crying out to be implemented as indexed columns in a real database.

8. Security, ha ha, let’s pretend

Despite the fact that the Registry is just a plain file that you can modify using all sorts of external tools (eg. our hivex shell), you can create “unreadable” and “unwritable” keys. These are “secure” from the point of view of Windows, unless you just modify the Registry binary file directly.

Windows also uses an unhealthy dose of security-through-obscurity. It hides password salts in the obscure “ClassName” field of the Registry key. The “security” here relies entirely on the fact that the default Windows REGEDIT program cannot view or edit the ClassName of a key. Anyone with a binary editor can get around this restriction trivially.

9. The Registry is obsolete, sorta

Well the good news is the Registry is obsolete. The bad news is that Vista has introduced another, incompatible way to store application data, in AppData/Local and AppData/LocalLow directories, but that Windows Vista and Windows 7 continue to rely on the Registry for all sorts of critical data, and it doesn’t look like this mess is going to go away any time soon.

* * *

libguestfs on Fedora now provides the tools you need to manage the Registry in Windows virtual machines. For more details, see hivexsh and virt-win-reg documentation.

Update

Thanks to all who commented. There is further discussion here on Reddit and here on Hacker News (including discussion of inaccuracies in what I wrote). If you want to look at our analysis code, it’s all here in our source repository. For further references on the Registry binary format, follow the links in the hivex README file.

68 Comments

Filed under Uncategorized