Hello Community,

I have just finished working on a XenDesktop 5 error for one of my customers, and as usual, wanted to share the troubleshooting process with you.

Customer Environment

A high-level view of the customer’s environment is:

  • XenDesktop 5 SP1 as the broker
  • 2 DDCs in the XD5 site
  • VMware ESX 4 hosting the DDCs and virtual desktops.
  • Approximately 180 Windows 7 virtual desktops in the primary production desktop catalog
  • Two small catalogs for non-production use.
  • MCS is being used to rapidly deploy virtual desktops from a single Master image.

 

Problem

The customer had a requirement to add a small number of machines to their production catalog.  This is a simple process as the catalog already exists; the customer simply needed to right-click on the catalog and choosing “Add machines”.  With the settings for catalog already defined the customer only had to enter how many additional machines to create and XenDesktop will do the rest.

Unfortunately when the customer completed the “Add Machines” wizard, Desktop Studio returned the error below (I have removed a few of the customer-specific details for anonymity purposes):

Desktop Studio Provisioning Task error

The customer then contacted me to open a new support case as they needed this resolved urgently.  I was unsure how to troubleshoot this initially as I did not had too much experience of troubleshooting XenDesktop 5 so received some excellent assistance from some of the XenDesktop developers.

 

What is a Provisioning Task?

When an administrator decides to use Machine Creation Services to create virtual desktops, Desktop Studio will create a “provisioning task” which is an administrative reference and a group of settings for the work being done to clone virtual machines from the master image.  You can see an example of how Provisioning Tasks are used to create catalogs here and the PowerShell commands are documented here.

 

Finding the running task

A desktop catalog can only have a single provisioning task running at a time, and although Desktop Studio did not show any tasks in progress, when the customer tried to add more virtual desktops to the catalog we saw an error indicating exactly that.

So Desktop Studio could not help us here; we could not see any running tasks.  Therefore we had to drop to the command line and use PowerShell to understand the problem, and then to resolve it.

The first command we ran was:

<strong>Get-ProvTask</strong>

The output of this command was a full list of all the tasks that have been run by Machine Creation Services since the customer created the XenDesktop site; about 70 in total.  Of course filters could have been passed to this command to reduce the output, but at this stage I was just trying to get an understanding of the environment (and also the PowerShell commands!).  Each task output looks something like this (some information anonymised):
TaskId                             : 4da36947-2d25-4213-860a-35bf37207d00
Active                             : False
Host                               : XD5DDC1
DateStarted                        : 04/01/2012 14:41:09
Type                               : NewVirtualMachine
Metadata                           : {Citrix_DesktopStudio_DesktopCatalogId = 13, Citrix_DesktopStudio_StartTime = 6346
                                     12848661173317, Citrix_DesktopStudio_TaskGroupId = b3670ee2-2fa9-4814-acb1-04f5f70
                                     858b0, Citrix_DesktopStudio_TimeTaken = 754970396}
WorkflowStatus                     : Completed
MasterImage                        : XDHyp:\HostingUnits\XENDESKTOP\win764GOLD.vm\Citrix_XD_Win7  64 Bit.snapshot\Citri
                                     x_XD_Win 7 64 Bit.snapshot\Citrix_XD_Windows 7 64.snapshot\Citrix_XD_Windows 7 64.
                                     snapshot
ProvisioningSchemeName             : Windows 7 Desktop
ProvisioningSchemeUid              : 76b65abd-ed08-4e0f-91ef-397e6d9dbe83
TaskState                          : Finished
TaskStateInformation               :
HostingUnitUid                     : 45142a5f-7828-4be7-a223-0693934d5905
HostingUnitName                    : XD5
IdentityPoolUid                    : 2950cdb0-423a-4044-814c-a99cdc97afe1
IdentityPoolName                   : Windows 7
VirtualMachinesToCreateCount       : 1
VirtualMachinesCreatedCount        : 1
VirtualMachinesCreationFailedCount : 0
CreatedVirtualMachines             : {VDI-0000123}
FailedVirtualMachines              : {}
ProvisioningJob                    : dcc8c19a-a5b6-46d6-9d4b-63cbb8c51333
ProvisioningStatus                 : Completed

Immediately the “Task State” property looked interesting.  I looked through the list of tasks and found a task that had been running for over a week:
TaskId                       : 3071f618-c786-444e-bf3b-8a1e7b05baf5
Active                       : True
Host                         : XD5DDC1
DateStarted                  : 29/12/2011 11:35:06
Type                         : RemoveVirtualMachine
Metadata                     : {Citrix_DesktopStudio_TaskGroupId = 6175c94a-e068-49e0-aba8-e39503230c55}
WorkflowStatus               : Running
ProvisioningSchemeUid        : 5dd053a0-4dad-4618-a497-f04ea8e4228c
ProvisioningSchemeName       : Windows 7 Desktop
TaskState                    : Removing
TaskStateInformation         :
VirtualMachinesToRemoveCount : 6
RemovedVirtualMachines       : {VDI-0000119, VDI-0000111, VDI-0000120, VDI-0000123...}
FailedVirtualMachines        : {}
VirtualMachinesRemovedCount  : 6
VirtualMachinesFailedCount   : 0
ProgressEstimator            : 5

What we see here is a Provisioning Task that is removing 6 desktops from a catalog.  In fact if we compare the “VirtualMachinesToRemoveCount” with the “VirtualMachinesRemovedCount” both settings have a value of 6.  This suggested that this task to remove 6 virtual machines had in fact completed, but the task had become stuck or timed out or had not been marked as completed in the database.

 

Stopping the Task

Because the task showed as completed (all virtual machines had been removed) and because 8 days had elapsed since the task was started, we decided it would be safe to manually stop this task with the following command:

<strong>Stop-ProvTask –TaskId 3071f618-c786-444e-bf3b-8a1e7b05baf5 –AdminAddress &lt;your controller address&gt;</strong>

(I should caution you not to stop tasks that have not completed.  Some tasks will create or delete accounts from Active Directory, and if you manually stop the task you may leave accounts in place when in fact you want them deleted.  If you are unsure how to proceed please contact Citrix Technical Support.)

We ran get-provtask again and could see that this task was now stopped.  We could see another task running with the details shown below, but this task should NOT be stopped.  It is an admin task that cleans up unused Master Images:

TaskId         : 00de4ac5-1bff-43ca-ba0d-86f638fe9fed
Active         : True
Host           : XD5DDC1
DateStarted    : 15/08/2011 16:35:20
Type           : DisusedImageCleanUp
Metadata       : {}
WorkflowStatus : Running
PendingDeletes : 1

 

Did this fix it?

Not quite.  Desktop Studio would still not allow the customer to create more VMs, again showing an error indicating that a Provisioning Task was running.  At this point again I contacted our engineering team who helped us onward.

Provisioning Schemes

As well as Provisioning Tasks which are the “units of work” for Machine Creation Services, there are also items called “Provisioning Schemes”.  A Provisioning Scheme is the collection of settings that will be applied to the virtual desktops created under the scheme.  You can view your provisioning schemes by running the following command:

<strong>get-provscheme</strong>

I asked the customer to run this and we saw the following output:
ProvisioningSchemeUid  : 5dd053a0-4dad-4618-a497-f04ea8e4228c
ProvisioningSchemeName : Windows 7 Desktop
CpuCount               : 1
MemoryMB               : 4096
DiskSize               : 30
MasterImageVM          : XDHyp:\HostingUnits\XENDESKTOP\IT6MCSGOLD.vm\Citrix_XD_IT6 Catalog.snapshot
MasterImageVMDate      : 17/08/2011 12:18:39
IdentityPoolUid        : a71ad46d-bfbc-4f5f-bd22-21d2f5b8872f
IdentityPoolName       : Windows 7
HostingUnitUid         : 45142a5f-7828-4be7-a223-0693934d5905
HostingUnitName        : XD5
CleanOnBoot            : False
TaskId                 : 3071f618-c786-444e-bf3b-8a1e7b05baf5
Metadata               : {}
MachineCount           : 73
ControllerAddress      : {XD5DDC1.domain.local, XD5DDC2.domain.local}
VMMetadata             : {H, 4, s, I...}

If you look at the TaskId field we see a reference to the task we manully stopped.  This lingering reference means the provisioning scheme is locked so we cannot initiate any more tasks under this scheme.

To clear this reference we ran the following PowerShell:

<strong>Unlock-ProvScheme -ProvisioningSchemeUid "5dd053a0-4dad-4618-a497-f04ea8e4228c”</strong>

<strong> </strong>

This unlocked the provisioning scheme and allowed the customer to create virtual machines again.  The customer was pleased to get this error resolved, and for me it was my first toe-dip into XenDesktop 5 PowerShell.  The following PowerShell commands were used to fix this issue, with their documentation links:
<strong>Get-ProvTask - <a href="http://support.citrix.com/static/kc/CTX127254/help/Get-ProvTask.htm">http://support.citrix.com/static/kc/CTX127254/help/Get-ProvTask.htm</a></strong>

<strong><strong>Stop-ProvTask - <a href="http://support.citrix.com/static/kc/CTX127254/help/Stop-ProvTask.htm">http://support.citrix.com/static/kc/CTX127254/help/Stop-ProvTask.htm</a></strong></strong>

<strong><strong><strong>Get-ProvScheme - <a href="http://support.citrix.com/static/kc/CTX127254/help/Get-ProvScheme.htm">http://support.citrix.com/static/kc/CTX127254/help/Get-ProvScheme.htm</a></strong></strong></strong>

<strong><strong><strong><strong>Unlock-ProvScheme - <a href="http://support.citrix.com/static/kc/CTX127254/help/Unlock-ProvScheme.htm">http://support.citrix.com/static/kc/CTX127254/help/Unlock-ProvScheme.htm</a></strong>
</strong></strong></strong>