Lately I was troubleshooting some performance issues in a XenApp 6.5 environment, the customer was observing 2 issues 1: degraded performance in user sessions and 2: scalability problems.  The root cause of this issues was fairly easy to detect when looking at the performance history of the XenApp servers :  A (cr)application was hitting the CPU constantly which causes CPU spikes and overall performance degradation. The environment wasn’t using any form of CPU management other than the default CPU Dynamic Fair Share Scheduling (DFSS) feature in 2008 R2. While DFSS does a great job in equally distributing CPU resources amongst user sessions, the problem with this application is that it’s running away in every user session resulting in overall performance degradation of other applications and lower scalability of the XenApp farm.

While the software vendor is looking at the issue (I will not call the vendor by name because I don’t want to “name and shame”) a community tool developed by Andrew Morgan named ThreadLocker came to mind. ThreadLocker is a tool that can change the CPU affinity and priority of processes on the fly.

In this blog post I will show you how ThreadLocker is helping to improve the overall performance and scalability by changing the CPU affinity of the process that causes the CPU spikes on the fly. Beneath is a snapshot of the CPU usage before ThreadLocker was running, this spikes are caused by the application running in different user sessions.

The task list reveals the application running away with the CPU resources :

So I installed ThreadLocker and configured the affinity of the process to CPU core 0 and 1, I didn’t changed the priority because this caused the application to hang completely  :

ThreadLocker is very light weight and runs as a background service so it does its job in every user session, nothing to configure at a per user basis. After ThreadLocker is running it changes the affinity of the process to CPU core 0 and 1 at a custom interval  :

After ThreadLocker is active notice CPU core 2 and 3 on the right when the users start using the application :

ThreadLocker will also write its activity in the event logs when the verbose logging option is switched on, in this way you can check how often ThreadLocker switches the affinity.

Conclusion :

In this blog post I showed you a good use case for ThreadLocker, with this tool you can tie processes to certain CPU cores to free others. In this way the run away application cannot consume all CPU resources on the XenApp servers so other applications have always free CPU resources to use. Please note that in this use case it’s not a permanent solution for this issue, by binding the process to less CPU cores the application itself has less resources to consume and will perform slower, but that weight is much lower than the performance and scalability impact in the whole environment. We will use ThreadLocker here till the software vendor has fixed the issue and ThreadLocker will be in my Toolbox when I come around this kind of situation again.

You can download and read more about ThreadLocker here.

Please note that the information in this blog is provided as is without warranty of any kind.