A while ago I wrote a web page on xenserver.org describing how to debug Windows VM that crash/BSOD. I managed to publish it in a pretty obscure location, here, and I’m not at all sure anyone has actually read it! I was inspired by tweet from Reinhard Teischl (@rteischl) on twitter, containing a universal truth:

  • “even virtual desktops have BSODs (bad apps and so on)”

Actually it’s still very rare to get BSOD or crashes with recent versions of Windows but if it does happen it’s rarely caused by the virtualisation or hypervisor. I’ve only saw a couple of cases in the two years I worked on Citrix XenServer. One involved corrupt memory arising from an out of date BIOS, this was actually identified by uploading the server status reports to TaaS which identified the known problematic BIOS, so it always pays to do this – instructions on how to collect server status reports and upload them for scrutiny are available here.

Anyway if you are a Microsoft administrator and interested in debugging virtualised Windows VM’s here is my original article, which does assume some knowledge of administration/development debug:

Debugging Windows Guests on XenServer

WinDbg

WinDbg is one of a number of tools available from Microsoft that can be used for debugging Windows guests in XenServer environments.

The Official Method

You can get QEMU to passive-open a TCP port on dom0 for serial output and wait for a connection, this method will work if you’re running on a machine with a dynamically assigned IP address:

  • on XenServer 6.2 or later
xe vm-param-add uuid=<uuid> param-name=platform hvm_serial=tcp::<port>,server,nodelay
  • on earlier versions
xe vm-param-add uuid=<uuid> param-name=other-config hvm_serial=tcp::<port>,server,nodelay

Then:

  1. Grab a copy of HW Virtual Serial Port from http://www.hw-group.com/products/hw_vsp/hw_vsp2_en.html and install it.
  2. Fire up the application and configure a COM port to point at your dom0 IP address and the <port> you specified to QEMU above). Make sure that you uncheck ‘NVT Enable’ in the Settings tab.
  3. Start the COM port.
  4. Configure debugging inside the guest by editing boot.ini in the usual way.
  5. Reboot the VM.
  6. Start WinDbg and point it at the COM port you created.

An Unofficial Method

To the best of my knowledge we have never released sockpipe.exe but this is a method some of our own developers use, it isn’t supported but may be useful to developers in a debug scenario.

If you’re running WinDbg on a machine with a statically assigned IP address then you can use the following setting to get QEMU to active-open a TCP port on your machine for serial output:

  • on XenServer 6.2 or later
xe vm-param-add uuid=&lt;uuid&gt; param-name=platform hvm_serial=tcp:&lt;machine&gt;:&lt;port&gt;
  • on earlier versions
xe vm-param-add uuid=&lt;uuid&gt; param-name=other-config hvm_serial=tcp:&lt;machine&gt;:&lt;port&gt;

Then:

  1. Run sockpipe.exe (which seems to be available here) on the machine where you want to run WinDbg. Run it without arguments to get help; it should be fairly obvious what’s going on. For example, you could use ‘sockpipe <name> <port>’ (where <port> is the name number as specified to QEMU above).
  2. Fire up windbg and go “File->Kernel Debugging”, make sure that “Pipe” is ticked, and enter \\.\pipe\<name> in the Port box, where <name> is the pipe name specified to sockpipe. Hit OK.
  3. Configure debugging inside the guest by editing boot.ini in the usual way.
  4. Reboot the VM.
  5. Debug as normal.

Note: It may be a bit slow, but it mostly works although it has limitations; it will not work for SMP 64 bit or SMP Vista

Further Information

Some users have recommended this external blog detailing WinDbg in XenServer environments and in conjunction with other Citrix products, the advice contained has not been reviewed by Citrix: http://vikashkumarroy.blogspot.ru/2012_08_01_archive.html