Under the Hood of XenApp & XenDesktop

Recently I had the honor of being a lead architect working on a new course for Citrix Education. I wanted to design something that I would enjoy attending myself, so we created the course with an emphasis on the use of PowerShell. The course is mostly hands-on and explains some core concepts of the (still rather new to many customers) FlexCast Management Architecture (FMA).

We focused on a lot of under-the-hood details needed for troubleshooting: what the FMA services are, how they work together and how you can recover them if anything unexpected happens.

The course was such a success, we released a companion course—CXD-302: XenApp and XenDesktop Advanced Concepts – Troubleshooting—in March 2016. I could spend hours talking about this course, but you can join me and curriculum product manager Amit Ben-Chanoch and watch a Behind the Design video at YouTube to learn more.

One whole module of this course is dedicated to troubleshooting of the FMA core components including FMA services. The code is broken down in class for a better learning experience to understand step by step the processes and communications of the services.

For this post, I am posting a more complex version using more advanced PowerShell functionality to reduce the lines of code to accomplish the same results.

Both XenApp and XenDesktop are now based on FMA architecture. In fact, XenApp and XenDesktop are using identical binaries and are just different feature editions. From the underlying architecture perspective, both XenApp and XenDesktop can be considered to be the same product. It would be more accurate to talk about FMA 7.8 rather than XenDesktop 7.8 or XenApp 7.8.

What is FMA architecture?

You have maybe heard different explanations of what FMA actually is. Marketing people will use a different definition than technical people. As my current title involves both “technical” and “marketing,” it can be hard for me to choose sides. Therefore, I’m going to give you my own, rather technical definition.

FMA architecture for me is a service oriented architecture where multiple controllers consist of multiple independent services. All these services are implemented following very well defined standards – as a result, it’s much easier to maintain such code and this also allows Citrix engineers to easily add new functionality if needed just by adding new services to the mix. When FMA was first released with XenDesktop 5, it consisted of 5 services, while most recent version XenDesktop 7.8 consist of 11 FMA services.

Services are independent of each other and communicate using service endpoints (URL format) – these are also known as Windows Communication Foundation (WCF) addresses. Each service uses a separate DB connection string to connect to the primary site database (we talk cover a lot more about DB in the CXD-302 training course). All services run under the NT AUTHORITY\Network service account – the controller machine account is used for authorization (which is topic for a separate blog post, but this elegantly solves the common problem with password management for service accounts).

The result is platform that is easier to maintain and at the same time can be easily extended (combining these two factors is a huge achievement for any software company).

Services, services, everywhere you look

As mentioned before, all services run independently of each other, so a failure of one service will not cause a disruption in the functionality of any other service(s). But not all services are equal – we can logically divide services into three different groups:

• Core Services – These services are essential for functionality of an FMA site.
  • CCS – Central Configuration Service acts as a centralized directory of all FMA services (more details later).
  • CLS – Configuration Logging Service is critical, because it needs to be involved in all changes to the environment to make sure that they’re recorded in central database.
  • DAS – Delegated Administration Service is also crucial, because it needs to determine if current user has required privileges for every call he is making.
• Desktops\Apps Services – MCS, brokering, hypervisor (host) and AD identity services are all required for provisioning of desktops or applications.
• Support Services – Analytics, StoreFront, Environment Testing and Monitoring services are considered supporting services.

Below is the diagram that we’ve included in the CXD-302 training. If this all sounds too complicated, don’t worry – all this complexity is hidden under the hood and you don’t really need to know it in order to use XenApp or XenDesktop although an understanding of this can really help when troubleshooting.

One of the great diagrams from CXD-302

Each of the FMA services has a set of accompanying cmdlets that you can use. You can use the following table to get list of cmdlets (use Get-Command to display all exported cmdlets). This table is one of the job aids that we’re providing with CXD-302 (quick fact sheets or reminders).

One service to rule them all?

I have promised to provide some details about Central Configuration Service (CCS). To quote Mick Glover (a.k.a. XDTipster), “CCS is the glue holding the FMA together.” I usually describe CCS as an equivalent of Yellow Pages for FMA services – it’s a list of all available FMA services (equivalent of “person”) with their registered service endpoints (equivalent of “contact details”).

As you can imagine, this could generate a lot of traffic, so each of the FMA services will keep a cache of CCS directory for five minutes. Quick tip – if you ever need to refresh the cache, just restart the Windows service, as cache is retrieved during the startup of the service.

The next section was provided by Bas Van Kaam from his upcoming book “Inside Citrix: The Flex Management Architecture,” which I cannot wait to see, as it already looks amazing:

All FMA services need to register with the configuration service on start-up so that it knows they are all good to go. This is one of the main reasons why the CCS has such a prominent role, it handles all inter service communication within the FMA. Located at the center of the FlexCast Management Architecture it holds and manages a list of all FMA services, allowing them to advertise their WCF addresses, or endpoints including the functionality that they provide. Only after a service successfully registers with the configuration service, when adding in additional Controllers, after a reboot or during Site creation for example, will it become active and able to communicate with other FMA Site services. Once done, the configuration service will share a listing of all active and registered services as being active Site members.

As soon as an individual service successfully queries the listing, this information will be cached for five minutes. As a side note, the Machine Creation Service and the Machine Identity service both communicate through the Host service to find out about the configuration, and connections of the hypervisor, including the storage and network configurations needed for virtual machine provisioning. This information will be stored for one minute as opposed to the five minutes mentioned earlier. Each FMA service can query the CCS to look up other services using the listing mentioned earlier, including their functionalities that they require to use. In short, service registration and communication are all relying on the configuration service. As a side note, it will also store configuration meta-data for all services.

Automation\scripting considerations

Making sure that the FMA site is fully functional is not too complicated and as we’re showing in a couple of different exercises in CXD-302, generally consists of four steps:

• Make sure that you have required permissions for the database
• Make sure that all services have correct DB connection configuration
• Make sure that all services are registered in CCS
• Make sure that all services are aware of location of CCS

Now, if you’re scripting this, there is one important rule to remember – always start with core services in the following order:

• CCS (Config)
• Admin (DAS)
• CLS (Log)

The order for core services is very important – you need to make sure that these three services are configured first (whatever changes you are making).

A second important rule for scripting is that what you need to do for one of the services, you need to do for all of them. If you change the DB connection string, you need to change it for all 11 services. If you are registering all instances with CCS, you need to do it for all 11 services. And if you are updating the location of CCS, you also need to do it for all of them.

Have a look at this blog post from XDTipster about a manual join of the new controller and read through all the steps to get some ideas of what needs to be done.

In step 3 (Register-ConfigServiceInstance), you are making changes to the CCS directory (updating entries). In step 4 (Reset-*ServiceGroupMembership), you are making changes to the other FMA services (updating the location of CCS directory), so they will know how to contact the CCS. Using our Yellow Pages analogy again – in the first step, you update all entries in the Yellow Pages, in the second step you send an email to all participants to let them know where they can access this directory.

Finally, if you really want to take your scripts to the next level and make them future-proof, you want to dynamically load all available services. The method that I’m using is through a parsing of the Get-Command output (which is probably the most important PowerShell cmdlet there is, we even have one dedicated exercise for it in CXD-302).

Below is an example script where I’m dynamically parsing through the cmdlets:
Download

Summary

Wow, that was really exhausting, but I hope you have found a lot of valuable information here. Just to quickly summarize what we’ve been discussing here:

• FMA is really cool!
• XenApp and XenDesktop are using the same FMA architecture.
• All FMA services are independent on each other, but are closely working together (communicating using service endpoints).
• Services are being added to provide additional functionality in new versions of XenApp\XenDesktop.
• Three core services are: CCS, DAS and CLS.
• CCS acts as a Yellow Pages directory for all FMA services.
• What you do to one service, you need to do to all of them.
• Order of the core services is important when making changes.
• You should join course CXD-302 if you want to learn even more
• FMA is really cool!