As Architect of Application Streaming, one of the things I do along with the other Citrix architects is worry about how to build the layers of cake.  Operating system on the bottom, Applications in the middle and user profile on the top.  We gathered a few months back to work on this problem and some pretty cool stuff came out of it.  The concepts apply to XenApp and XenDesktop.  Here’s the high level view:

I’m going to focus primarily on XenDesktop.  The same ideas apply to XenApp hosted on Virtual Machines as they do to XenDesktop running virtual desktops.  One way or another, the “system” has to be put together and that means assembling an operating system, applications and user data.

Starting at the top: User data.  With Roaming Profiles or Citrix Profile Manager, the application settings are already roamed as a common occurance.  This is the least interesting of the layers in this post.

The quick version is that if a user visits machine A today and machine Q tomorrow, the profile manager will move the applications settings from A to Q.  At logon/logoff the user profile is synchronized to some central machine, where it is later copied from to the new execution machine.  This kind of technology has been around for a very long time, though there still exists some art to “doing it properly”.  The “properly” part says that you cannot ASSUME that a user will not be logged on to multiple machines at once.  This is the classic “last logoff wins” problem.  Thankfully, the Citrix Profile Manager does not suffer these problems, so this top layer of the layers of cake is already baked and ready for icing!

A side note is “user data” is over stated, the user profile is really worrying about Application Settings.  The user’s data, as in DOCUMENTS, will normally be redirected to some network server.

Next, the “bottom”, the operating system:

Notice I skipped apps for the moment.  Apps are the point of this post, so I’m saving them for last.

Every machine, virtual or real, needs an operating system.  For real machines, no problem, INSTALL IT locally and it will still be there the next time you power the machine on.  For Virtual Machines, this same INSTALL IT solution could be used with one disk image per machine.  Store the image away at “power down” and bring it back at “power up”.  This will RUN, but it will sure be a PIG on disk usage.  It will also be harder to maintain as there will be a separate disk image for every virtual machine.  The whole appeal of bringing things into the data center is that it makes things easier to maintain.  If all you do is convert physical hard disks to virtual hard disks, you haven’t really worked on the problem of simplifying maintenance!

A side note: With real machines, you can also get to the ONE to Many maintenance aspects, Provisioning Services does this for real hardware just as it can for virtual hardware.

Back to the virtual world:

To get this “right”, we need ONE Operating System image and ONE image for each application to go along with ONE user profile.  In this world, the operating system maintainer, maintains the OS and each application owner maintains their respective application.  In some environments, these are the same person, but in concept, they are all separate and even if they are the same person, we do not want application content to pollute the OS image.  If that happens, then every time ANY application needs update, the OS image must be updated and versioned => inefficient.

Cashes, caches and more caches!

With a virtual machine, the virtual machine manager maintains a “write back cache” for the “machine”.  Pooled desktop here, there is ONE image for the operating system and each VM as it runs has disk space that holds the changes to the disk as the machine runs.  When the user logs off, the machine powers down and the virtual machine manager THROWS AWAY everything that was written to the base machine.

This “throw away” is necessary as the disk is a “block” entity and the only way that the single base image can be updated is if it is updated in ONE place.  On a future logon, the pool manager will have queued up a pristine machine, with the LATEST AND GREATEST version of the operating system image, which will bring the virtual machine to life – while implementing yet another write back cache for the machine.

Applications, the point of this post

Application Streaming in the graphic above is the Application Virtualization system bringing in application content.  This is the artist correctly now titled the “plugin for offline applications”.  I’m getting used to it.

The App Virtualization system could also be Microsoft App-V or potentially even some other solutions though I do find it easier to manage the caches for systems that have installed agents.  As Architect of App Streaming, I’m working the Citrix side today; the concepts are the same.

Given a virtual machine is running and you want to get apps onto it, what are your choices?

1) Install them into the base image.  We covered this already – it’s a losing gig.

2) Presentation delivery.  Publish from XenApp and deliver to XenDesktop; HDX gets it to the user.

3) App Virtualize the applications to get them “into” the hosted desktop.

Today, we’re worring about “3”.

Given we have a virtual machine, anything we populate into the virtual machine will be a DISK WRITE.

The fundamental guiding principle: Disk writes are BAD! We want everything to be a “read”.

When the OS runs, we want the system to populate pieces of the OS as needed via block based fills from the virtual machine manager or disk subsystem.  When we run applications, we want the virtual machine manager to READ data from the application store and … (finally getting to the meat of this post) … NEVER perform a disk write.

Application Streaming has a long history of delivering application content to physical machines.  On physical machines, disk writes are fine, they are a one time cost and the up front cost actually has long term benefit to minimize network activity on future executions.  With virtual machines, disk writes are evil and they are a recurring “every logon” cost.

We need disk writes to go away, which leads me to one of my favorite statements related to layers of cake.

  • We gotta get Application Streaming out of the STREAMING business.

This task much better to hand off to the virtual machine manager or to Provisioning Services, or to the enterprise disk subsystem.  Whatever the the technique of mounting disk space, all of these folks will do a more efficient job, if the streaming system somehow makes everything a “read”.

In the layers of glass, the streaming system populates stuff into the execution cache, which is called RadeCache, named after the directory that holds the application execution content.  I have also previously described that deploy should never happen on a virtual machine based execution, so thankfully we can avoid all of that disk activity.

How do we get application content into a virtual machine?  Answer: MOUNT IT!

The Application Hub is the file server that holds the execution content.  In a physical world, the streaming part of application streaming deals with COPYING content from the Application Hub onto the physical execution machine.  In a Virtual World, we want everything to already be in place, so that application execution is based on disk reads and the disk write generating “streaming” aspects of Application Streaming, never occur.

One complication to date has been that the streaming system stores stuff on the Application Hub in compresses container files, CAB files.  These are going away in a future release, to be replaced with decompressed directory images of the same content.

Putting it all together

Once we know that we don’t want the streaming system to “stream”, it can instead focus on application isolation and application delivery; delivering “installed” applications to single instance machines, but letting other pieces of the puzzle take care of the streaming aspects.

The Deploy space is not used; the RadeCache – is necessary.  Stick with me on how we make it go away too.

Given that the RadeCache space and the streaming profile execution content on the App Hub have the same format, there are things we can do to “mount” the Application Hub directly into the execution space.  With the “next” streaming client, this is easier because the App Hub storage format is changed to match the execution format commonly found in the RadeCache.

This can STILL be accomplished NOW using the released 5.2 version of the streaming client.

The example App I will use is Textpad.  To follow along, profile this up and store it some place convenient.  I’m using Windows 7 machine with the 5.2 level offline plugin and skipping all the publishing aspects.

Start up a command prompt (run as administrator).  I follow with a cut / pasting of a bunch of commands which I entered to prove this can work with 5.2.  After that, we’ll talk about what each of them do.

c:

md \apphub & cd \apphub

icacls . /grant:r Ctx_StreamingSvc:(R)

icacls . /grant:r Ctx_StreamingSvc:(OI)(CI)(IO)(R)

mkdir textpad

xcopy \\ConvenientPlaceFromEarlier\textpad textpad\. /s /e

cd textpad

md guid_v

cd guid_v

extract ..\guid_v.cab *.*   (Extract is from Microsoft CAB SDK and requires *.* rather than *)

cd \program files\citrix\radecache

mklink /d guid_v c:\apphub\textpad\guid_v   (Creates a JUNCTION to the mounted App Hub)

exit (close the command prompt window)

Finally run the thing (a really short version of publishing the the AMC).

start “” “%PROGFILES%\Citrix\Streaming Client\RadeRun.exe” /package:”c:\apphub\textpad\textpad.profile” /app:”Textpad”

WOW!  That was alot of stuff.

Given you did the start above, you’ve already seen the application run and come to life, with minimal disk writes!  Drum roll.

Here’s what happened.

The MKDIR of C:\AppHub simulated the MOUNTING of the App Hub onto the local machine.

The icacls commands gave the streaming service READ privilege to this space.  The streaming service runs on a named user account and by default it can only modify a few places on disk, like RadeCache and Deploy.  I think this is unnecessary as all users on the machine will automatically get read access to this space due to inherritance from the root.  Still, I stuck it in there in case it is a required element.  If so, you’ll need this in the system configuration when the mount point is created.

The streaming service will “believe” that this space is local when it is really backed up by some XenServer, Provisioning Server, Hyper-V, ESX, Disk magic system.  One way or another, it’s a MOUNT POINT and the streaming service has privilege to read it.

Why local?  The isolation system looks at all disk activity at the start and if it is “network”, it jumpts out of the way and doesn’t mess with it.  So, the streaming system MUST believe that this space is “local”.  For all these virtual worlds, this is “easy”.

Next trick is that the streaming sytem will want to populate the RadeCache at runtime, we don’t want this to occur.  Instead, we want the streaming system to BELIEVE that the RadeCache is “fully populated”.

Creating the JUNCTIOND at the RadeCache space, the streaming system will believe that the RadeCache space is already there, when it looks inside, it will see that the files are already in place and it won’t have any reason to want to write more to this space.  Notice that with the “linkd” commands in place, when the streaming system looks into the RadeCache space, we will have LIED to it to instead sent it to the C:\AppHub space.

Lying to the Lying system – wonderful!

Deep down in the streaming streaming, the device driver that does the file system filtering isn’t fooled by any of this application level nonsense; it sees the “real” location, C:\AppHub.  The isolation system insists that the RadeCache space for the application content be on local storage, and it is, as far as it knows.

C:\AppHub is “local” as far as this virtual machine is concerend.

With the call to RadeRun, the application is launched and comes to life; skipping publishing.

The application comes to life and things are happy.

Publishing

When publishing applications in the Access Management Console, one of the fields is the UNC path to the App Hub.  This can be a UNC path or a HTTP path, but either way, the AMC is told where the App Hub is located and when applications are enumerated via PNAgent/Web Interface, this information is relayed to the streaming launcher for app launch.  This means that the publishing has to reference C:\AppHub or possibly a local host based reference of local machine based UNC reference.  The AMC and the execution machine both have to believe that C:\AppHub is local storage.  You can also create a LINKD based junction on the AMC machine that reflects to the “real” app hub on the network.  The AMC will believe that this is the local machine space and will relay this to the execution machine.

I dream of a future where the streaming client could override what the AMC says to use and instead use a fixed location that the administrator says to use.  Just dreaming…

House cleaning

One of the things that the streaming system does is clean house in the RadeCache.  If it exceeds high water mark in size, the streaming system will launch a reaper thread that deletes things until the house is clean.  Here’s a link that describes how this works.  Since we only granted the streaming service “(R)” Read access to the local machine AppHub, the streaming services’ attempts to house clean in RadeCache space that is really the AppHub space will be unsuccessful – it will have no choice but to “push on” to look for some other file to erase.

For this reason, in a XenDesktop space, we should either put everything into the read-only space where the Streaming Service can’t delete it or we should set the cache high-water mark sufficiently high that the house cleaning thread will never trigger; in reality, we should do both.   Leaving the RadeCache space writable by the streamign service allows it to run applciations that haven’t been published or managed into the App Hub, so this is good – though that space would be loaded on each logon to the virtual machine.

Hope this is useful.

Joe Nord

Product Architect – Application Streaming and User Profile Manager

Citrix Systems