This is the 3rd blog of a three part blog series on architecting an on-demand VM provisioning system in the cloud. In the first blog, I discussed the conceptual architecture for how you could design the system using a Cloud Provider (such as SoftLayer or Amazon EC2) or using your own virtual machine infrastructure (such as XenServer or Hyper-V). In the second blog, I discussed the Virtual Computing Demo Center project that allows our sales teams to provision Citrix demo environments on-demand utilizing SoftLayer cloud infrastructure.

This blog will be dedicated to discussing a second real-world example of cloud computing that we use at Citrix – the Solution Center Cloud. Before I get into the architecture of the Solution Center Cloud, it’s important to first understand what the Solution Center is and what this project was about at a high-level.

What is the Solution Center and Solution Center Cloud?
The Solution Center is a small team within Worldwide Consulting Solutions that is dedicated for managing and running the high-profile projects requested by the Geo teams. These projects have included scalability testing, POC guides, best practice guides, 3rd-party integration how-to guides, and much more. You may have seen some of this material posted by Carisa Stringer – who leads the Solution Center team. This team is also dedicated for managing the Citrix Consulting lab – which is a test laboratory based in Ft. Lauderdale that the consultants can use for testing products and solutions for supporting their projects in the field.

Since Citrix has consultants spread out all over the globe and not everyone can make it to Ft. Lauderdale in person, there was a huge need to build an on-demand provisioning system that could be accessible from anywhere, was simple to use, allowed testing of products in a low-risk environment, and provided turn-around of lab resources within minutes. The Solution Center Cloud was created to fulfill this need. The Solution Center Cloud is a web-based, lab resource provisioning system that allows Citrix Consultants to spin up resources on-demand (virtual machines, VPX appliances, etc.) in a lab testing capacity. The Solution Center Cloud leverages XenServer as the underlying hypervisor for running the virtual machines.

The user experience of the Solution Center Cloud is very similar to the Virtual Computing Demo Center mentioned in the previous blog in this series. Users navigate to the Solution Center Cloud portal to request one or more resources to be provisioned. The resources are provisioned by a backend Windows service that calls the XenServer API’s directly to spin-up new virtual machines. When the virtual machines are ready, an email is sent to the user and the user can connect to the machines to begin their work. After a period of time, the virtual machines are cancelled automatically by the system or users can cancel their resources manually after use.

A screen shot of the Solution Center Cloud portal is shown below. This screen shows how users can request lab resources on-demand – a NetScaler VPX appliance and two Windows Server 2008 virtual machines!

What does the Solution Center Cloud provision?
The Solution Center Cloud allows consultants to provision three types of resources as outlined in the table below:

Demo Environment Description
XenServer Virtual Machines This is the most common type of resource that users provision within the system. Several base OS templates have been created to represent various Windows operating systems (W2K3, W2K8, Win7, WinXP, etc). A NetScaler VPX template is also provided to allow a user to provision a NetScaler virtual appliance.
Physical NetScaler environment A physical NetScaler can also be requested by the system. You might be thinking a few questions here – “If I’m allowed to provision a NetScaler VPX, when would I need to provision a physical NetScaler and what does this ‘provision’ entail?” Excellent questions – I’m glad you asked
The truth is that the Solution Center Cloud project was started before the NetScaler VPX appliance was available. A physical NetScaler was procured and placed within the Citrix Consulting lab to allow users to play with the physical device. When users provision this type of resource, all the system really does is change the “nsroot” password and swap out the ns.conf file to the base un-configured state. This allows each user to start fresh each time.
Physical Branch Repeater environment A physical Branch Repeater environment can also be requested by the system. If you read the NetScaler environment section above, you probably have a few questions about this type of resource as well – “Could we just provide a Branch Repeater VPX and what does this type of ‘provision’ entail?
The Branch Repeater VPX was just recently released by Citrix and there are plans on adding this type of resource to the Solution Center Cloud in the near future. In the meantime, a few physical Branch Repeaters have been procured and placed within the Citrix Consulting lab to allow users to request this type of lab resource. When a user provisions a physical Branch Repeater environment, they are really just provisioning a client machine on the XenServer that talks to the pre-configured Branch Repeater environment on the backend. Since most users just provision virtual machines on XenServer, I’ll leave a more detailed description of this resource type for a future blog.

Solution Center Cloud Physical Architecture
The physical architecture of the Solution Center Cloud is shown in the picture below. The key item to note about this architecture is that the Portal Server and Backend Resources (XenServer, NetScalers, Branch Repeaters) all reside locally in the Citrix Consulting lab in Ft. Lauderdale. The Windows Services that run on the Portal Server all make native API calls to XenServer, NetScaler, and Branch Repeater. This is different than the Virtual Computing Demo Center solution that resides on the SoftLayer cloud. In that architecture, the solution leveraged the SoftLayer APIs for provisioning resources on the SoftLayer CloudLayer. It had no direct communication with the underlying XenServers.

The components that make up the architecture are discussed in the following table:

Component Description
Web Portal The Web Portal is the user interface that consultants go to request resources, cancel resources, and check on status. The Web Portal also has admin functions built into the user interface that only administrators can view. Access to the web portal is all handled via Active Directory group permissions. If you are a member of the domain, you have access to the user-based functions (provisioning resources, checking status, etc.). If you are a member of the admin group, you also have access to the admin-based functions that are exposed by the web portal (checking logs, checking system errors, etc.)
Shared Services Three key services are shared among the entire Solution Center Cloud infrastructure:
Domain Controller – an Active Directory domain is set up in the lab to control the authentication of the Web Portal. The Portal Server is a member of the domain, and the Web Portal leverages integrated authentication with the Active Directory domain.
DNS/DHCP (Not Pictured) – DNS and DHCP services are provided on the Domain Controller machine. The virtual machines that are provisioned on XenServer through the system are configured to acquire a DHCP IP address. This frees the system from having to maintain a list of available IP addresses and keep the list up to date as virtual machines are created/deleted. The one catch to this is that although users are free to install their own Active Directory domains on the provisioned virtual machines, they cannot install DHCP on the machines as this will interfere with the current DHCP process.
SMTP Server – the SMTP service has been installed on the Portal Server to allow emails to be sent to users at various points of the using the system. The Provisioner Windows Service is the primary component that leverages the SMTP service for sending out emails during the provisioning process and when resources are cancelled.
File Server – a file share containing ISO images has been set up the file server. When consultants provision base virtual machines on XenServer, they need access to CDs/DVDs to install products on those machines for testing out various things. To accommodate this, ISOs for various software were placed on a single file share on the file server. MagicDisc was installed on each base OS template to allow consultants to mount ISOs from their system tray. This allowed the ISO share to be accessed from within the virtual machines without having to grant each user access to XenCenter console.
Provisioner Windows Service The Provisioner Windows Service performs all of the provisioning duties of the backend resources. Different actions are performed based on the resource type being requested (see below). After the resource is provisioned, the IP address and credential are sent to the user via email. This service runs every few minutes to check for any new requests made from the Web Portal.
XenServer Virtual Machine (including VPX appliances) – The Provisioner Windows Service creates a virtual machine on XenServer by copying a base OS template hosted on the NetApp SAN. Also deletes virtual machines running on XenServer at their scheduled cancellation time. Utilizes the XenServer .NET SDK for communicating to XenServer.
Physical NetScaler Appliance – The Provisioner Windows Service resets the password of the “nsroot” account and reloads a base ns.conf file to provide a fresh NetScaler installation. Utilizes the Tamir.Sharp.Ssh library and NetScaler API for communicating to the physical NetScaler.
Physical Branch Repeater Appliance – The Provisioner Windows Service resets the Branch Repeater to factory settings and provisions a client VM on XenServer that acts as the client for the backend Branch Repeater environment. Utilizes the .NET HttpWebRequest and HttpWebResponse classes for communicating to the physical Branch Repeaters.
Discovery Windows Service The Discovery Windows Service performs the following duties:
• Updates the XenServer template list within the SQL Server database once a day. Admins are able to choose which templates are made available to users from the admin pages of the Web Portal.
• Updates the XenServer storage repository list within the SQL Server database once a day. Admins are able to choose which storage repositories are made available to the system from the admin pages of the Web Portal.
SQL Server The SQL Server hosts the database for the Solution Center Cloud system. The database stores details such as available templates, available storage repositories, and resource requests. The schema for this database is very similar to the database used by the Virtual Computing Demo Center mentioned in the previous blog. All SQL queries used by the Provisioner and Discovery Windows Services are written as stored procedures within SQL Server.
API Layer The Provisioner and Discovery Windows Services call different APIs on the backend components based on the type of resource being requested.
XenServer Virtual Machine (or VPX appliance) – the XenServer .NET SDK is leveraged for performing actions on XenServer.
Physical NetScaler – the Tamir.SharpSsh library is leveraged for sending NetScaler CLI commands to the physical NetScaler over SSH.
Physical Branch Repeater – the System.Net.HttpWebRequest and System.Net.HttpWebResponse .NET classes are leveraged for executing commands on the physical Branch Repeaters over HTTP. Branch Repeater provides web-based administration in general, so calling certain pages on the branch-repeater allows you to do certain actions such as restarting the appliance and resetting the factory settings.
XenServer Pool Two XenServers are configured as part of a pool to host the virtual machines provisioned by the system. When the system needs to provision virtual machines on XenServer, the Provisioner Windows Service checks the available memory and hard-drive space of both XenServers to determine which server is the best fit for hosting the machine.
XenServer provides APIs for checking the available resources of a XenServer host, and these APIs are leveraged for finding the most suitable and optimal XenServer for handling the request.
NetApp SAN The NetApp SAN stores the OS templates and NetScaler VPX template that can be provisioned on XenServer. The NetApp SAN is configured as a storage repository within XenServer so that XenServer can communicate with it. Base templates have been created for different OS types (e.g. Windows Server 2003 x86 and Windows Server 2003 x64) and these templates are stored within the NetApp storage repository.
When a virtual machine is to be provisioned, the Provisioner Windows Service makes a copy of the template hosted on NetApp and stores the copy on the local XenServer that will run the virtual machine. XenServer then runs the virtual machine from its local hard-drive.
Physical NetScalers A physical NetScaler appliance can be requested by the Solution Center portal. Only one physical NetScaler is contained within the environment, so users need to share the device. The NetScaler is already physically connected to the environment. When users request this resource, the Provisioner Windows Service resets the “nsroot” password to a new randomly generated password so that only the new user can access the device. The ns.conf file is also reset to bring the device back to its initial un-configured state. With the recent introduction of the NetScaler VPX template into the Solution Center Cloud, most users provision the virtual appliance over the physical device, so the physical NetScaler may be phased out of the Solution Center Cloud environment in the future.
Physical Branch Repeaters A physical Branch Repeater environment can be requested by the Solution Center portal. This environment consists of a client VM that resides on XenServer, two physical Branch Repeaters, an Apposite WAN emulator, and a physical file server (and file share). When users request this resource type, the Provisioner Windows Service provisions a client VM on XenServer and resets the physical Branch Repeater to its factory settings. The client VM can then be used to connect to the physical Branch Repeater environment to simulate a WAN connection from a “branch” office to the datacenter. I hope to provide more details about this particular setup in a future blog – stay tuned!

This blog provided a high-level overview of the Solution Center Cloud architecture. My goal here was to showcase a second real-world example of how we are using cloud computing at Citrix to help improve our way of doing business. The Solution Center Cloud provides a great test-bed for Citrix Consultants to spawn up new virtual machines, test software and configurations, and tear-down virtual machines within a low-risk environment.

This wraps up this blog series on architecting an on-demand provisioning system in the cloud. I hope you were able to gather some insights on designing such systems for use within your own environments. In future blogs I hope to expand on the specific architectural details of the Portal, Windows Services, and Database Schema that I discussed within the Virtual Computing Demo Center and Solution Center Cloud. Cheers!

Blogs in this series:
Part 1 – Conceptual architecture
Part 2 – Real-world example 1 – Virtual Demo Center
• Part 3 – Real-world example 2 – Solution Center Cloud (this one)

Ed York – Architect – Worldwide Technical Readiness
Ask-the-Architect Site:
Follow Me on twitter: