Since CloudPortal Services Manager v10 was released last December, the support forum has been a key resource for CSPs in getting installation issues resolved. In this post I’ll highlight solutions to the most frequently raised issues in the forum hopefully give some context that will help others get issues resolved more quickly.

Creating the First Location

Creating the first location has been the trickiest part of the deployment process. Here are some common issues in this area and ways to go about resolving them.

Login fails for the CSP administrator account

After completing the all deployment steps, you attempt to login with the CSP administrator and receive the error “Your credentials are invalid. Try again or contact your service provider.”

Background

Front-end authentication is handled by Directory Web Service. If that service is not running, or is not reachable (e.g. due to firewall rules), you will see login failures in Web.

Provisioning requests are recorded in the OLM database, and then sent to the Provisioning Engine for processing.  The Configuration Tool returns successfully if provisioning requests are able to be recorded in OLM and sent to the Provisioning Engine.  However, provisioning requests could potentially not be processed if the Cortex Queue Monitor service is not running or an error occurs with one of the requests.

Resolution

Verify that the Directory Web Service is running and is accessible.  See the topic Directory Web Service-related Issues below for details.  If those checks pass, then verify that the administrator account exists in Active Directory. If it doesn’t, then the provisioning requests to create the first location, customer and user were either not processed, or not executed successfully.

To see if the provisioning requests were processed or not:

  1. Log on to the Provisioning Engine server and check that the “Cortex Queue Monitor” service is running.
  2. Open Provisioning Manager from the Start Menu, drill down to Message Queues > My Computer > Private Queues, and check the “cortexrequest” and “cortexbulkrequest” queues for any unprocessed messages.
  3. If either of the above steps reveals an issue, stop the Cortex Queue Monitor service (the CortexQueueMonitor.exe process may take a minute or two to actually terminate) and start it again.

To see if provisioning errors occurred:

  1. Connect to the OLM database and “SELECT * FROM RequestLog WHERE Severity=2”.

To enable tracing in the Queue Monitor:

  1. Log on to the Provisioning Engine server and open “%ProgramFiles(x86)%\Citrix\Cortex\Provisioning Engine\CortexQueueMonitor.exe.config” in a text editor.
  2. Find the element “<add name=”Trace” value=”Off” />” and change the value attribute to “All”.
  3. Find the element “<add name=”FileListener” />” and uncomment it. Note the log file specified in the initializeData attribute and set it accordingly if needed.
  4. Save the configuration file, and restart Queue Monitor as described above.

An error related to Directory Web Service was returned

In the Primary Location wizard, an error occurs at one of the following steps:

  • Register Location’s Directory Web Service
  • Register Domain Controllers

Or an error message in the log file references the Directory Web Service.

Background

During Primary Location wizard, a number of operations on the Directory Web Service are invoked in order to provision the OU structure & user, as well as to enumerate servers in the environment. If the service is not running and reachable, this can cause the Primary Location wizard to fail.

Resolution

Verify that the Directory Web Service is running:

  1. Log on to the server where Directory Web Service is installed
  2. At the start prompt, run “inetmgr”, drill down to the “CortexServices” site and find the “Directory” application.
  3. Right-click and select “Switch to Content View”.
  4. Right-click “Directory.asmx” and select “Browse”. This should launch the available service operations in IE.

If you receive an HTTP 503 “Service Unavailable” error or an HTTP 401 “Unauthorized” error at step #4, the most common reason is that the credentials for the application pool identity are incorrect. To resolve this:

  1. Find the DirectoryWSAppPool in IIS and note the user account it is using.
  2. Reset this user’s password in AD.
  3. Run the configuration wizard for Directory Web Service again, passing in the user’s new password.

If you receive an error mentioning ISAPI/CGI restrictions, it is possible that the ISAPI filters for ASP.NET v4 are not enabled.  Try the following:

  1. From a command prompt, run “\Windows\Microsoft.NET\Framework64\v4.0.30319\aspnet_regiis.exe –ir –enable”.

Verify that the Directory Web Service can be accessed from the server where the Web server role was installed, and from the server where you are running the Primary Location wizard. Check for any firewall rules that might be blocking needed ports.  A summary of the firewall requirements can be found here on eDocs: http://support.citrix.com/proddocs/topic/ccp-10/ccp-10-system-requirements-roles.html

Recovering from a primary location provisioning error

The documentation recommends that you back up CPSM databases before configuring the first location so that if an error occurs you can address the issue (e.g. using the tips highlighted above), roll back to a clean snapshot of the databases, and run the first location wizard again.  Without database backups the official recommendation is to reinstall from scratch.  However it may be possible to recover the deployment before resorting to drastic measures.  Here are two approaches to doing this.

Fix the database

The first approach is to try to manually bring the database back into a state that allows the first location & customer to be provisioned again.  To do this, you need to remove records of the customer and location specified earlier by running the following SQL commands:

USE OLM
EXECUTE sp_CustomersDelete @CustomerID = 1
DBCC CHECKIDENT ('CUSTOMERS',RESEED,0)
DBCC CHECKIDENT ('USERS',RESEED,0)
EXECUTE sp_LocationsDelete @LocationID = 1
DBCC CHECKIDENT ('LOCATIONS',RESEED,0)

Locations are given IDs starting at 1. If you’ve run the first location wizard multiple times, there may be more than location in the database that needs to be cleaned up. “SELECT * FROM Locations” to verify this.

Redeploy the database

Another approach is to redeploy the system databases, manually update the deployment configuration file, and run all the configuration wizards again.

  1. Redeploy the database
    1. If using the same SQL server instance, delete the databases (OLM, OLMReports & ExchangeLogs) and logins (OLMUser, OLMReportsUser, CortexProp & ExchangeLogsUser) created by the Configuration Tool
  1. Run the System Databases wizard again. Specify a different path for the deployment configuration file so that the original isn’t overwritten.
  2. Create backups of the new databases
  3. Update the original deployment configuration file
    1. Replace all properties in the original configuration file with the values set in the newly created one.
    2. Reconfigure all server roles
      1. For each server role (Directory Web Service, Provisioning Engine, Web), run the configuration wizard again, specified the original configuration file that was hand-edited earlier.
      2. Run the first location wizard again

Hopefully this helps you get going with CPSM deployments more quickly.