In an effort to support the Citrix Desktop Transformation booth at Synergy 2011 in San Francisco we built a XenDesktop 5 and Hyper-V (R2) infrastructure able of hosting 1.000 desktops. While building this environment we encountered a couple of challenges which might be seen in other similar infrastructure as well. Therefore we’d like to share our lessons learned as part of this blog, which is a joint effort between Michel Lüscher (Consultant Datacenter – Windows Server and Virtualization) from Microsoft Consulting in Switzerland and myself.

A Hyper-V Cluster requires a Domain Controller to become functional. For transferring the infrastructure into the exhibition hall we had to power down everything. While preparing this power down / up cycle, we discovered that a Hyper-V Cluster requires a connection to a domain controller to become functional. Without a connection the Failover Cluster will not work and it would not be possible to start any virtual machine which is part of this Hyper-V cluster. Therefore we moved one of our domain controllers onto a stand-alone Hyper-V host and powered on this system (as well as the domain controller VM) first.

There’re a few things to consider when you host Active Directory domain controllers in a virtual environment. Microsoft recommends maintaining physical domain controllers for each of the domains. This mitigates the risk of a virtualization platform malfunction that affects all host systems that use that platform.

You can find all the details on Microsoft TechNet and Support KB:

SCVMM Tuning for VDI environments. Handling hundreds or even thousands of virtual machines is challenging for every virtual infrastructure, including Hyper-V and SCVMM. In order to ensure good response times for XenDesktop triggered VM actions (i.e. power-on / power-down) and to lower the CPU utilization of the SCVMM servers we were required to implement the SCVMM tuning configurations as outlined in this article written by Vishwa from the SCVMM product group:

Creating a template in SCVMM and using it for the XenDesktop Setup Wizard is not as easy as it seems, if you are new to SCVMM. The individual steps are actually fairly easy, but it is important to follow the steps in the right order. The following Microsoft TechNet article and Citrix blog outlines the process in detail:

Note: The current version of XenDesktop and Provisioning Server do not support nor work with SCVMM 2012 which is available as a beta today. Citrix will have to / has planned to release an update after RTM is available to make XenDesktop compatible with the new management product.

In case you’re not familiar with the features coming in SCVMM 2012, Michel has written a few articles about this topic. Check them out on www.server-talk.eu.

XenDesktop Setup Wizard machine creation performance. To ensure best performance we were required to equip every virtual desktop with a NTFS formatted 4GB virtual disk for storing the Provisioning Services Write Cache. Therefore we created a virtual machine template which contains an appropriate virtual disk and stored it within the SCVMM Library. When running the XenDesktop Setup Wizard we saw a machine creation time of about 4 minutes per new VM, whereof the vast majority of the time was consumed for copying the virtual disk. In order to minimize the time required for copying the VHD we took a look at how the VM provisioning process works. It turned out that for every new VM the template was copied over the network using BITS from the SCVMM Library to the Hyper-V server selected within the XD Setup Wizard. Furthermore we figured out that only the owner of the Cluster Shared Volume (i.e. HYPER01 for Cluster Disk 1 in screenshot below) has full read / write access to the CSV. In this design this owner is called the “Coordinator node”. Only this node has metadata access and the permission to create / edit / delete files. Meaning certain parts of the VHD were transferred from the selected Hyper-V server to the CSV owner which then performs the actual write to the CSV.

Cluster Shared Volume - Admin Console

To streamline the process and to minimize the servers involved when creating new VMs we created a new SCVMM Library on the CSV owner system and moved our VM template there. Furthermore we selected the CSV owner within the XenDesktop Setup Wizard for performing the VM creation. By doing so we were able to decrease the time per new VM to a little less the one minute (on enterprise class storage) and 2:30 (on SMB class storage).

The nice and new feature called “Dynamic Memory”. When using the initial version of the new XenDesktop Wizard for creating new virtual desktops, advanced settings such as the dynamic memory configuration specified within the VM template got lost. Therefore we were required to upgrade to the latest version of the Wizard, which has been published here: http://support.citrix.com/article/CTX129381.

XenDesktop 5 Setup Wizard does not list any hosts. When we ran the XenDesktop Setup Wizard the first time it did not list any host. This was related to the way we configured the host connections to SCVMM/Hyper-V within XenDesktop. The issue and the resolution is discussed within the knowledge base article CTX128809 (http://support.citrix.com/article/CTX128809). When trying to modify the host connection accordingly the next challenge we faced was that we were unable to select a storage repository within the host connection wizard. The solution is simple and discussed within eDocs (http://support.citrix.com/proddocs/topic/xendesktop-rho/cds-scvmm-rho.html)

“For a multiple Hyper-V host deployment, ensure your Hyper-V hosts are set up in a Hyper-V Failover Cluster with Cluster Shared Volume storage. On one of your Hyper-V servers, create a Windows network share, that is writeable by the System Center Virtual Machine Manager administrator account for the Cluster Shared Volume mount point, typically C:\ClusterStorage. For more information about setting up a Hyper-V Failover Cluster with Cluster Shared Volume storage, see your Microsoft documentation.”

After this final modification we were able to run the host connection / XenDesktop setup wizard successfully.

Optimize Windows for being used as virtual desktop. The default installation settings of Windows 7 might be perfectly suited for traditional desktop PCs or notebooks, but for running virtual desktops some optimizations are required. In fact some of the default settings of Windows 7 (such as the scheduled hard disk defragmentation) have the potential to actually break a virtual desktop infrastructure. Therefore it was important to evaluate / apply the recommended settings discussed within the various optimization guides. Good starting points are:

Optimize 3rd party software for being used within virtual desktops. Similar to Windows 7 most of the software available on the market was not build for virtual desktop infrastructures and need some optimization. In our case we saw a couple of applications and agents which wrote IDs for authenticating a device into the registry of our Windows 7 master device, which caused some issues for cloned machines based on this master. The only workaround was to apply computer startup scripts (similar to application compatibility scripts know from the Server Based Computing era). Of great help for this task was the personality.ini file which resides within the root of the C: drive and contains information about the current status of the vDisk. So for example the “_DiskMode” property allows determining the mode of the vDisk (shared / private). This enabled us running certain scripts for private mode vDisks only.

Optimize Anti Virus for virtual desktops. This topic has been stressed a couple of times already, but within our Desktop Transformation VDI infrastructure we could verify once more that optimizing the anti virus agents is vital. In fact we could decrease the amount of data read from the vDisk during the startup phase from around 1.4GB per virtual desktop to approximately 350MB by optimizing the scanning settings of the anti virus agent. On a scale of 1.000 desktops this saving sums up to more than 1TB for every startup phase. This minimizes the load on the network as well as the Provisioning Servers tremendously. The Citrix Best Practices we applied have been documented here: http://support.citrix.com/article/CTX127030

Note: Optimizing the anti virus agents is important for the Hyper-V servers as well, in case an agent has been installed within the parent partition of the hosts. In this case it is necessary to configure scanning exclusions as discussed within the Microsoft KB article http://support.microsoft.com/kb/961804. Otherwise you may see problems when starting virtual machines after the installation of an anti virus software in the management operating system (parent partition). The root cause of this problem is that a number of these programs monitor file access in a way that interferes with Hyper-V’s attempts to open virtual machine files.

In general a very good resource for information in regards to VDI is the Citrix Consulting XenDesktop Design Handbook, which is available here: http://bit.ly/xdhandbook.