Some time back, I wrote a post on file system filter drivers and load altitude which included discussion of  kernel mode filtering and how “filter altitude” affects interaction of App Streaming file system filtering and anti-virus.  Today’s post discusses application level hooking of registry operations and how this can cause collisions between application level entities, specifically, Citrix App Streaming with AppSense Environment Manager (EM).

Application Streaming implements the layers of glass for registry operations at “application layer”.  AppSense EM implements its registry filtering also at “application layer”.  With the release of Citrix Application Streaming 5.2 in XenApp 5.0 Feature Pack 2, we cratered them; returning the isolated view of the registry to the application even if AppSense EM were trying to provide a view from their cached database backend.

Since then, AppSense have released version 8.x of EM which works with Citrix App Streaming “offline plugin” version 5.2.  The supporting version of EM is available now as a download from MyAppsense.com.  With the EM 8.x release, things are again working and more than working, it’s worth noting that AppSense have done extra work to ensure that EM works effectively with Citrix App Streaming based applications just as it does with locally installed applications. This works XenApp server side, end user client side as well as on XenDesktop hosted desktops.

Environment Manager introduction

AppSense Environment Manager is an application set which ensures that users receive a consistent view of their desktop and application settings regardless to the computer that they use to logon or how the desktop applications are delivered.  Similar to Microsoft Roaming Profiles and Citrix Profile Manager, EM gathers configuration information from applications and retains that information across logons, sessions and across machines where it can later be provided back to the application as if it came from the same machine where it was stored.  From a technology view, EM uses a database back-end and local machine cache to store information and uses application level “hooking” to provide configuration information to the applications on-demand at runtime.  For example, if an application stores a setting at a specific place in the registry, that operation is backed up by the database rather than the local registry and the EM hook code answers the question bypassing the registry.

The interesting part is that since the database is used for central storage, this can occur with or without actually roaming the user’s profile from session to session.  The backing store for the information is the database and this remains true no matter what happens on HKCU registry or file system.

In the collision case, App Streaming returned the isolated registry view of the setting, which could and often would be different than the “correct” setting from the AppSense EM central store.

How hooking works

Back in the beginning, there was DOS…

DOS was (“is”?) a single user, single tasking operating system.  In theory there could be only one application running at a time.  Clever programmers quickly figured out that they could simulate a multi-processing machine by hooking the execution of the single application.  Borland’s SideKick was the first commercial success here.  It ran at at machine startup (autoexec.bat), then went away and hung out in the background until the user typed a special key, which was pressing ctrl-alt at the same time.  On seeing the hot-key, the program would come to foreground and take over the machine screen and keyboard.  When done, execution would return to the application that was interrupted as if nothing had happened.

Two things to observe

  1. The single tasking OS was now running more than one app at the same time
  2. Citrix XenCenter has a hot-key collision with a 27 year old application.

How did SideKick accomplish this?  Answer, Terminate and Stay Resident API into DOS to leave a portion of application in memory after the application had officially terminated.  The resident portion was “HOOKED” into the BIOS IRQ-9 keyboard hardware interrupt processing so that when both Control and Alt were pressed, the program would save away the screen, bring itself to foreground, do it’s work and eventually put things back.  Impressive stuff for the day and the TSR was here to stay.

The concepts of hooking application activity have not changed significantly since the TSR days of DOS.  The Windows operating system and Windows based applications have a set of dispatch points and these dispatch points implement program or Win32 SDK functions.  To “hook” a function, the function dispatch points get “messed with” so that the hook get called instead of what the system/application really wants to call and then the hook has a chance to do things early and make intelligent decisions on whether or not to call the “real” software.  The concept is “unchanged” in 27 years and if someone is older than me, this probably goes back even further.

In the days of DOS, everyone soon wanted to climb onto interrupt 9 to watch the keyboard button press and release messages and pretty soon, dependencies arose on the “order” of placement onto the TSR stack.  Hook collisions were born!

Everyone wants to be on top

View the hook code as a stack of hook modules.  The application calls what it believes to be the registry API, but which in reality is the hook code of the first hooker.  That hooker does it’s work and calls what it believes to be the real system API, but which is in reality the next hooker in the chain.  Add more hook modules and the stack gets higher, more layers between the application and the real implementation of the registry API in the Win32 system.

The “top” hook module is the first one to see the API call.  In the DOS days, the “last” TSR to get installed was generally the first code to get called in the hook.   Today, that’s unchanged, the last hook module to hook is usually the FIRST entry called when the hooked API is called from the application.

If you’ll recall from my post on FSFD and altitude, “everyone overrates their own importance”. Everyone is worried about their own hooking and everyone wants it to be successful in their own testing, but when you step back and look at a full system and then ask, “who should be at the top of the hooking stack”, the answer is that it usually shouldn’t be you.  The challenge is that when placing yourself onto the API hook stack, you can’t really see the hookers below, they are indistinguishable from the real API.  You have little choice but to place yourself at the top of the stack and this means that you will get called before the people lower on the stack.

Application Streaming absolutely MUST be able to hook the registry to accomplish Application Virualization and AppSense Environment Manager absolutely MUST hook the registry to accomplish it’s mission of replacing the “registry” with a local cache and database back end.  Who should be “higher”.  Answer: AppSense.  Whether the application is run isolated or locally installed, the application settings must come from EM and getting the hookers out of order can prevent this from occurring.

With Application level hooking, altitude discussions are harder than with kernel mode, because there is no easy way for a programmer to control altitude for application level hooks. In the kernel, you tell the filter manager, “put me here”.  At application level, chaos is supreme.

Where it broke

When an isolated application calls RegQueryValueEx, the registry API is hooked by both Citrix App Streaming and AppSense Environment Manager.  Being on the top, the Citrix code gets first look.  For a registry key that is isolated, the registry query is converted from it’s before isolation location to its sandboxed location according to the layers of glass.  With the location converted, the operation is sent “down the stack” to have the Windows registry system fetch the value from the isolated registry.  To shorten the story, in the both case of App Streaming + Environment Manager, for isolated spaces, Citrix 5.2 streaming client would BYPASS the AppSense hook, preventing it from doing it’s job of looking up the value in the central database.

BUSTED – And no easy fix.

With Environment Manager 8.x,  AppSense have changed their hooking code to get themselves in “higher” in the registry stack than is Citrix App Streaming.  The end result is that EM starts working again, even for streamed applications.  To get here, AppSense had to be aware of Citrix App Streaming in their code and had to take programmatic action to work past some roadblocks that we put in their way.

What it comes down to is that AppSense have hooked the Citrix Hook code code.  Isolating the isolation system if you will, which is a pretty neat idea.

The present is a happy place

AppSense has Environment Manager working with Citrix App Streaming version 5.2.

The future is daunting

Soon, a new release of Citrix App Streaming will ship with XenApp 6.0.  Working with AppSense, we are 100% positive that changes in App Streaming hooking implemented for 6.0 will interfere with Environment Manager and will require an update to EM to address.   I’ll go on record that the fault for the break here is the Citrix end.  We changed the method of hooking and this has no choice but to break the EM hooking of the Citrix Hooks.

What did we change?  Well, AppInit_Dlls is no longer vogue, so it’s time to move off.  Also, implementing isolation of NT services requires some creative programming for filtering the Windows Service Control Manager.  Add it all up and we changed the hooking from AppInit_Dlls style application hooking to kernel implemented hooking of application APIs. The result is that “everything is different” and AppSense gets broken again.

I suppose this is the nature of the business.  Citrix climbs in deep into the operating system to do neat things and becomes dependent on the OS innards for function.  Microsoft changes the innards, and AppInit_Dlls, and we have to change our software to work with it.  The same analogy applies in the Citrix – AppSense case.  They are doing neat things – but these things require climbing in deep, and when we change the deep things, they get impacted.  The good news is that early tech reviews allow them to know of the problem sooner rather than later and we have already had conversations to discuss how to best resolve.  This is a partner relationship and we’re all motivated to make both successful.

So to my friends at AppSense.  Congratulations on your success with App Streaming 5.2 and … my apologies in advance for breaking you on 6.0.

Joe Nord

Citrix Systems

Product Architect for Application Streaming and User Profile Manager