This is a guest blog post by Trentent Tye, Technical Specialist at ControlUp and a Citrix Technology Professional.

I worked with a technology VIP who would occasionally reach out to tell me that Citrix was slow or had crashed over the last couple weeks. Getting this information long after it occurred usually made correlating data and finding a root cause difficult. You could look at the historical metrics and get some context for what happened, but it’s hard to find a silver bullet in the CPU, memory, or other data. Mostly, you just ended up looking at graphs, hoping something pops out.

If only there was a tool that would let me see what the VIP’s team was doing. If only I could see what they were doing when they experienced slowness or could provide some context for what they thought was a “crash.” The best I could do was shadow a session. Of course, everything worked perfectly then, just like when you visit the mechanic about a strange noise in your engine only to find that the car doesn’t make the noise when you’re at the garage.

If only there were a tool to help, that would enable me to see what the end user saw. It would provide the context I need and help me solve the problem.

This tool exists.

Citrix Session Recording

Citrix Session Recording is a powerful tool that can capture what the user sees during their remote session. This gives Citrix administrators a lot of power for security auditing or troubleshooting. In addition, Citrix Session Recording can annotate events. As of Citrix Session Recording 1909, these events are:

  • Log CDM mapped USB events
  • Log generic redirected USB events
  • Log app start events
  • Log app end events
  • Log sensitive file events
  • Log web browsing activities
Citrix Session Recording tracked the launch of msedge.exe processes.

Tracking when applications start or end or if they access certain paths or files is great! The process would exit if an application “crashes,” allowing us to visibly see what the user was doing at the time of the crash. This information can be fed back to the application developer if the root cause consistently ends up being a workflow-type issue. Or there might be enough information for an admin to come up with a clever workaround.

But even with all this great information and annotation, there are gaps.

For instance, some Citrix sessions can last hours or even days. Users might start and close applications frequently, generating noise. Or the slowness a user experiences might not be well explained by a recording. For example, if a user reports that their performance was degraded during one of these long sessions, it can be a daunting task to watch hours of footage to try and find a particular performance pain point.

With monitoring products, you can do some correlation between the historical performance graphs and the video recording, but this is a tedious task.

What if we could annotate performance problems within the session recording? Then we might be able to see the slowness or the workflow that is causing issues.

ControlUp Automated Actions – Augment Your Troubleshooting

With Citrix Session Recording we can add custom event annotations to the recording. ControlUp’s automated actions can take full advantage of this by creating events in your Citrix Session Recording as they occur. This can help you immediately narrow down to a specific issue or correlate the incident to a reported performance issue.

What can ControlUp automation annotate for you? Almost anything.

  • Slow user experience? Annotate with User Input Delay.
  • Slow network? Annotate with latency and/or bandwidth.
  • Session state? Annotate with Session State.
  • High CPU? Annotate with Session CPU.

How Do We Set This Up?

The prerequisites are a working Citrix Session Recording with “third-party applications to record custom data on this server” enabled.

Once completed, we can configure the automation by creating the following triggers:

Trigger Name: AAS_Citrix SessionRecording_User Input Delay

State/Record:

Filter:

Scope:

Follow-up Actions:

Name/Description:

Trigger Name: AAS_Citrix SessionRecording_Network Latency

State/Record:

Filter:

Scope:

Follow-up Actions:

Name/Description:

Trigger Name: AAS_Citrix SessionRecording_High CPU

State/Record:

Filter:

Scope:

Follow-up Actions:

Name/Description:

For recording the session state, we’ll leverage existing ControlUp community triggers used for resource optimization. Due to how Session Recording is implemented, we can only track when a session goes idle or from idle to active. To enable session state tracking, we only need to add the “Citrix Session Recording – Session State” script to two triggers:

I highly recommended you edit the triggers after enabling them and set the scope to only encompass your multi-session servers that have Session Recording.

For each of the Session State triggers, add the session recording script action to the triggers.

Script Actions

I’ve written the following script actions for each of the scenarios:

If you examine each script, you can see the differences between them are minor. If you wanted to add tracking for things not covered here, you can simply copy the script, modify the parameter and annotation text, and select the corresponding ControlUp metric in the parameters for the script action. This simplicity, power, and flexibility highlights how ControlUp can meet almost any of your needs to augment your troubleshooting capabilities.

Augmented Troubleshooting

To get a full sense of what additional capabilities you get with ControlUp custom event annotations, let’s look at what conclusions we can draw about issues within a Citrix session.

With ControlUp we can see numerous performance events have been recorded after a threshold was reached.

For example, when network latency is high, ControlUp will record the value and annotate within the Citrix Session Recording.

Clicking the event on the left will skip to when that event was annotated on the video. From this clip, we can see the user was watching a YouTube video while copying crossing the WAN to their client mapped drive. With this information, we can easily infer that the slow network performance can be due to the additional load due to streaming videos.

Another example is leveraging the new User Input Delay metric to measure user interactivity.

User interactivity was high in this session. That makes sense because these VDI machines do not have 3D graphics, and Adobe is rendering the 3D content in software. This additional CPU load has added substantial latency to the UI and it was captured by ControlUp and annotated within Citrix Session Recording.

Citrix Session Recording is a great feature that’s made even better with ControlUp custom event annotations! You can learn more about Citrix Session Recording here.


A Technical Specialist at ControlUp, Trentent Tye was first introduced to Citrix in 2011. Originally an Active Directory administrator, seeing Citrix XenApp in action was immediately engaging and he sought to know more. Applying for a Citrix position with no Citrix experience, Trentent became a Citrix Administrator by promising to become an expert efficiently. From here, he was introduced to complementary technologies like Microsoft’s App-V and UE-V and became a Microsoft MVP in 2015, successfully becoming renewed each year. He became a Citrix Technology Advocate in 2017 and a Citrix Technology Professional in 2019. Connect with him on Twitter at @trententtye, on LinkedIn and at https://theorypc.ca/.