Modern IT architectures are designed and built to scale and be resilient. Systems are now usually built to handle failover and auto-scale up and down to handle varying demand and workloads with very different properties and needs. Redundancy and surge capacity is often built-in to systems by design, as is self-repair.<\/p>\n
Modern technologies such as containers, microservices and cloud services mean IT systems are distributed and often administrators handle 100s or 1000s of servers, VMs and users. Modern monitoring platforms also allow alert automation, whereby thresholds can be set on 1000s or even millions of individual metrics. This can lead to alert fatigue and alert storms with administrators being swamped by hundreds of alerts and messages regarding minor and even inconsequential events.<\/p>\n
Performance Ratings and User Experience Scores provide a way to aggregate the signals from multiple metrics to get a useful measurable for the health of components and help eliminate alarm storms.<\/p>\n
<\/span>What is a Performance Rating?<\/span><\/h2>\nPerformance Ratings enable the grouping of different measures for a single component, service or user. A single quantifiable rating\/index that is representative of multiple metrics. Comparable to an Apdex (Application Performance Index) score or rating<\/a> or a Customer Satisfaction Index (CSI) calculated by service providers. A very common application is for generating a \u201cUser Experience Score\u201d, \u201cExperience Score\u201d or \u201cUX Score\u201d. A single number that measures how good the performance of a service or an experience is. Other names and terms describing performance ratings include \u201chealth score\u201d, \u201chealth rating\u201d, \u201cstress level\u201d and so on.<\/p>\n<\/span>Performance Ratings \u2013 Example: VDI and DaaS user experience scores<\/span><\/h2>\nMonitoring tools and observability tools can collect hundreds of metrics, events and signals associated with the VDI user experience. Many of these metrics are interdependent and many remoting protocols dynamically adapt to compensate for resource constraints e.g., reducing frame rate (fps) count to reduce bandwidth needs or reducing frame quality to lower CPU encode\/decode demands.<\/p>\n
Consider the user experience metrics typically exposed for Citrix CVAD (Citrix Virtual Apps and Desktops) which include Application launch time, ICA \/ HDX latency, logon time, enumeration time, network latency, profile load time, user input delay and more \u2013 plus the associated session and resource metrics such as CPU and RAM usage. Whilst the details are essential for troubleshooting. No single one of these metrics is suitable as a performance rating.<\/p>\n
A meaningful comprehensive user-experience metric needs to enable administrators and IT management to:<\/p>\n
\n- Compare the user experience of one session to another<\/li>\n
- Allow the reporting and tracking of user experience over a period of time<\/li>\n
- Be a quantifiable measurable KPI that can be used to state if the experience has been consistent, degraded, or improved<\/li>\n
- Allow admins to segment users and focus on those users have or are having a frustrating experience with applications<\/li>\n<\/ul>\n
eG Enterprise includes out of the box performance ratings for user experience that can be used as a significant metric for measuring, reporting, and comparing user experiences, including:<\/p>\n
\n- Citrix User Experience Rating (for Citrix virtual apps)<\/li>\n
- Citrix XenApp User Experience Rating (for Citrix XenApp 6.5)<\/li>\n
- VDI User Experience Rating (for Citrix VDI)<\/li>\n<\/ul>\n
Details on how to enable the out-of-the-box ratings are covered in the eG Enterprise documentation, see: Performance Rating Tests (eginnovations.com).<\/a><\/p>\neG Enterprise also supports custom performance ratings that allow customers to build and tune their own measures based on multiple metrics and weight them accordingly.<\/p>\n
Performance Ratings can be defined for any of the 500+ technology stacks<\/a> that eG Enterprise supports across a wide range of use cases from user experience for web applications, cloud deployments and DaaS \/ digital workspaces such as Citrix, Microsoft AVD \/ Windows Cloud PCs and VMware, to \u201cStress Level\u201d measures for system or network resources.<\/p>\n![\"\"](\"https:\/\/www.eginnovations.com\/blog\/wp-content\/uploads\/2023\/05\/Performance-Rating.webp\")
<\/a><\/p>\nFigure 1: Performance ratings can be viewed, added and customized via the \u201cAdmin\u201d tab of the eG Enterprise console.
\nAdmin -> Agents-> Performance Ratings<\/div>\n![\"\"](\"https:\/\/www.eginnovations.com\/blog\/wp-content\/uploads\/2023\/05\/Performance-Rating-img.webp\")
<\/a><\/p>\nFigure 2: A GUI editor is provided to allow users to create and \/ or customize performance ratings, defining which metrics to include and how to weight them<\/div>\n<\/span>The benefits of performance ratings and user experience scores<\/span><\/h2>\nHaving a single number associated with a component such as a server, user session experience, a web service or similar condition means that there is a single quantifiable number allowing an admin to:<\/p>\n
\n- Assess health instantly at-a-glance without reviewing dozens of interdependent metrics<\/li>\n
- Use intelligent thresholds and alerting around a single value based on multiple metrics to avoid alarm fatigue<\/li>\n
- Gain instant visibility overviews. In eG Enterprise performance ratings such as Citrix User Experience or custom Performance Ratings can be included on dashboards to give a rapid overview of service levels and user satisfaction<\/li>\n
- Improve visibility for the wider organization. As a measure, performance ratings can be used with thresholds applied to automatically generate reports<\/li>\n<\/ul>\n
Detailed metrics are needed for performance experts when investigating specific issues and support incidents, however IT executives should not have to go through all of these to know and assess whether the user experience is good\/not and whether the average user experience is degraded or acceptable long term.<\/p>\n
<\/span>Evaluating the capabilities of performance ratings<\/span><\/h2>\nMany monitoring and AIOps Observability<\/a> platforms offer some sort of performance rating measurements, sometimes only pre-defined out of the box, but some allow some level of configuration and a few (including eG Enterprise) also offer the ability to build full blown custom performance ratings. Sometimes these ratings are called \u201cUX Score\u201d or \u201cUser Experience Score\u201d, sometimes simply just \u201cExperience Score\u201d or \u201cStress Level\u201d or similar according to proprietary branding and whether applied to a user session or a server \/ networking component.<\/p>\nWhen evaluating performance rating functionality and capabilities, you will probably want to evaluate:<\/p>\n
\n- What type of thresholds can you set on the rating to trigger alerts?<\/li>\n
- Are both static and dynamic thresholds<\/a> available to minimize noise and ensure intelligent anomaly detection \/ normal usage for time-of-day accounted for? Dynamic thresholds are particularly important when using aggregated metrics as they are particularly difficult to manually calculate.<\/li>\n
- How is the rating displayed and consumed by overview dashboards?<\/li>\n
- Can you easily trigger reports off the rating?<\/li>\n
- What performance ratings are available out-of-the-box?<\/li>\n
- Is there a GUI to build custom performance ratings?<\/li>\n
- Can you integrate alerts from performance ratings into your organization\u2019s other ITSM systems (e.g. helpdesk tools such as ServiceNow or Freshdesk)<\/a> and analytics systems (PowerBI etc.) as desired?<\/li>\n<\/ul>\n
<\/span>Performance Ratings vs. Metric Aggregation<\/span><\/h2>\nIn a future blog I hope to also cover eG Enterprise\u2019s Metric Aggregation capabilities. For some use cases Metric Aggregation may be a more appropriate feature to leverage. For now, bear in mind that a Performance Rating enables grouping of different measures for a single component.<\/span> Whilst Metric Aggregation enables the same measure to be aggregated across multiple components of the same type. <\/span><\/p>\n
<\/span>Performance Ratings \u2013 Example: VDI and DaaS user experience scores<\/span><\/h2>\nMonitoring tools and observability tools can collect hundreds of metrics, events and signals associated with the VDI user experience. Many of these metrics are interdependent and many remoting protocols dynamically adapt to compensate for resource constraints e.g., reducing frame rate (fps) count to reduce bandwidth needs or reducing frame quality to lower CPU encode\/decode demands.<\/p>\n
Consider the user experience metrics typically exposed for Citrix CVAD (Citrix Virtual Apps and Desktops) which include Application launch time, ICA \/ HDX latency, logon time, enumeration time, network latency, profile load time, user input delay and more \u2013 plus the associated session and resource metrics such as CPU and RAM usage. Whilst the details are essential for troubleshooting. No single one of these metrics is suitable as a performance rating.<\/p>\n
A meaningful comprehensive user-experience metric needs to enable administrators and IT management to:<\/p>\n
\n- Compare the user experience of one session to another<\/li>\n
- Allow the reporting and tracking of user experience over a period of time<\/li>\n
- Be a quantifiable measurable KPI that can be used to state if the experience has been consistent, degraded, or improved<\/li>\n
- Allow admins to segment users and focus on those users have or are having a frustrating experience with applications<\/li>\n<\/ul>\n
eG Enterprise includes out of the box performance ratings for user experience that can be used as a significant metric for measuring, reporting, and comparing user experiences, including:<\/p>\n
\n- Citrix User Experience Rating (for Citrix virtual apps)<\/li>\n
- Citrix XenApp User Experience Rating (for Citrix XenApp 6.5)<\/li>\n
- VDI User Experience Rating (for Citrix VDI)<\/li>\n<\/ul>\n
Details on how to enable the out-of-the-box ratings are covered in the eG Enterprise documentation, see: Performance Rating Tests (eginnovations.com).<\/a><\/p>\neG Enterprise also supports custom performance ratings that allow customers to build and tune their own measures based on multiple metrics and weight them accordingly.<\/p>\n
Performance Ratings can be defined for any of the 500+ technology stacks<\/a> that eG Enterprise supports across a wide range of use cases from user experience for web applications, cloud deployments and DaaS \/ digital workspaces such as Citrix, Microsoft AVD \/ Windows Cloud PCs and VMware, to \u201cStress Level\u201d measures for system or network resources.<\/p>\n<\/a><\/p>\nFigure 1: Performance ratings can be viewed, added and customized via the \u201cAdmin\u201d tab of the eG Enterprise console.
\nAdmin -> Agents-> Performance Ratings<\/div>\n<\/a><\/p>\nFigure 2: A GUI editor is provided to allow users to create and \/ or customize performance ratings, defining which metrics to include and how to weight them<\/div>\n<\/span>The benefits of performance ratings and user experience scores<\/span><\/h2>\nHaving a single number associated with a component such as a server, user session experience, a web service or similar condition means that there is a single quantifiable number allowing an admin to:<\/p>\n
\n- Assess health instantly at-a-glance without reviewing dozens of interdependent metrics<\/li>\n
- Use intelligent thresholds and alerting around a single value based on multiple metrics to avoid alarm fatigue<\/li>\n
- Gain instant visibility overviews. In eG Enterprise performance ratings such as Citrix User Experience or custom Performance Ratings can be included on dashboards to give a rapid overview of service levels and user satisfaction<\/li>\n
- Improve visibility for the wider organization. As a measure, performance ratings can be used with thresholds applied to automatically generate reports<\/li>\n<\/ul>\n
Detailed metrics are needed for performance experts when investigating specific issues and support incidents, however IT executives should not have to go through all of these to know and assess whether the user experience is good\/not and whether the average user experience is degraded or acceptable long term.<\/p>\n
<\/span>Evaluating the capabilities of performance ratings<\/span><\/h2>\nMany monitoring and AIOps Observability<\/a> platforms offer some sort of performance rating measurements, sometimes only pre-defined out of the box, but some allow some level of configuration and a few (including eG Enterprise) also offer the ability to build full blown custom performance ratings. Sometimes these ratings are called \u201cUX Score\u201d or \u201cUser Experience Score\u201d, sometimes simply just \u201cExperience Score\u201d or \u201cStress Level\u201d or similar according to proprietary branding and whether applied to a user session or a server \/ networking component.<\/p>\nWhen evaluating performance rating functionality and capabilities, you will probably want to evaluate:<\/p>\n
\n- What type of thresholds can you set on the rating to trigger alerts?<\/li>\n
- Are both static and dynamic thresholds<\/a> available to minimize noise and ensure intelligent anomaly detection \/ normal usage for time-of-day accounted for? Dynamic thresholds are particularly important when using aggregated metrics as they are particularly difficult to manually calculate.<\/li>\n
- How is the rating displayed and consumed by overview dashboards?<\/li>\n
- Can you easily trigger reports off the rating?<\/li>\n
- What performance ratings are available out-of-the-box?<\/li>\n
- Is there a GUI to build custom performance ratings?<\/li>\n
- Can you integrate alerts from performance ratings into your organization\u2019s other ITSM systems (e.g. helpdesk tools such as ServiceNow or Freshdesk)<\/a> and analytics systems (PowerBI etc.) as desired?<\/li>\n<\/ul>\n
<\/span>Performance Ratings vs. Metric Aggregation<\/span><\/h2>\nIn a future blog I hope to also cover eG Enterprise\u2019s Metric Aggregation capabilities. For some use cases Metric Aggregation may be a more appropriate feature to leverage. For now, bear in mind that a Performance Rating enables grouping of different measures for a single component.<\/span> Whilst Metric Aggregation enables the same measure to be aggregated across multiple components of the same type. <\/span><\/p>\n
eG Enterprise includes out of the box performance ratings for user experience that can be used as a significant metric for measuring, reporting, and comparing user experiences, including:<\/p>\n
- \n
- Citrix User Experience Rating (for Citrix virtual apps)<\/li>\n
- Citrix XenApp User Experience Rating (for Citrix XenApp 6.5)<\/li>\n
- VDI User Experience Rating (for Citrix VDI)<\/li>\n<\/ul>\n
Details on how to enable the out-of-the-box ratings are covered in the eG Enterprise documentation, see: Performance Rating Tests (eginnovations.com).<\/a><\/p>\n
eG Enterprise also supports custom performance ratings that allow customers to build and tune their own measures based on multiple metrics and weight them accordingly.<\/p>\n
Performance Ratings can be defined for any of the 500+ technology stacks<\/a> that eG Enterprise supports across a wide range of use cases from user experience for web applications, cloud deployments and DaaS \/ digital workspaces such as Citrix, Microsoft AVD \/ Windows Cloud PCs and VMware, to \u201cStress Level\u201d measures for system or network resources.<\/p>\n
<\/a><\/p>\nFigure 1: Performance ratings can be viewed, added and customized via the \u201cAdmin\u201d tab of the eG Enterprise console.
\nAdmin -> Agents-> Performance Ratings<\/div>\n<\/a><\/p>\nFigure 2: A GUI editor is provided to allow users to create and \/ or customize performance ratings, defining which metrics to include and how to weight them<\/div>\n<\/span>The benefits of performance ratings and user experience scores<\/span><\/h2>\n
Having a single number associated with a component such as a server, user session experience, a web service or similar condition means that there is a single quantifiable number allowing an admin to:<\/p>\n
- \n
- Assess health instantly at-a-glance without reviewing dozens of interdependent metrics<\/li>\n
- Use intelligent thresholds and alerting around a single value based on multiple metrics to avoid alarm fatigue<\/li>\n
- Gain instant visibility overviews. In eG Enterprise performance ratings such as Citrix User Experience or custom Performance Ratings can be included on dashboards to give a rapid overview of service levels and user satisfaction<\/li>\n
- Improve visibility for the wider organization. As a measure, performance ratings can be used with thresholds applied to automatically generate reports<\/li>\n<\/ul>\n
Detailed metrics are needed for performance experts when investigating specific issues and support incidents, however IT executives should not have to go through all of these to know and assess whether the user experience is good\/not and whether the average user experience is degraded or acceptable long term.<\/p>\n
<\/span>Evaluating the capabilities of performance ratings<\/span><\/h2>\n
Many monitoring and AIOps Observability<\/a> platforms offer some sort of performance rating measurements, sometimes only pre-defined out of the box, but some allow some level of configuration and a few (including eG Enterprise) also offer the ability to build full blown custom performance ratings. Sometimes these ratings are called \u201cUX Score\u201d or \u201cUser Experience Score\u201d, sometimes simply just \u201cExperience Score\u201d or \u201cStress Level\u201d or similar according to proprietary branding and whether applied to a user session or a server \/ networking component.<\/p>\n
When evaluating performance rating functionality and capabilities, you will probably want to evaluate:<\/p>\n
- \n
- What type of thresholds can you set on the rating to trigger alerts?<\/li>\n
- Are both static and dynamic thresholds<\/a> available to minimize noise and ensure intelligent anomaly detection \/ normal usage for time-of-day accounted for? Dynamic thresholds are particularly important when using aggregated metrics as they are particularly difficult to manually calculate.<\/li>\n
- How is the rating displayed and consumed by overview dashboards?<\/li>\n
- Can you easily trigger reports off the rating?<\/li>\n
- What performance ratings are available out-of-the-box?<\/li>\n
- Is there a GUI to build custom performance ratings?<\/li>\n
- Can you integrate alerts from performance ratings into your organization\u2019s other ITSM systems (e.g. helpdesk tools such as ServiceNow or Freshdesk)<\/a> and analytics systems (PowerBI etc.) as desired?<\/li>\n<\/ul>\n
<\/span>Performance Ratings vs. Metric Aggregation<\/span><\/h2>\n
In a future blog I hope to also cover eG Enterprise\u2019s Metric Aggregation capabilities. For some use cases Metric Aggregation may be a more appropriate feature to leverage. For now, bear in mind that a Performance Rating enables grouping of different measures for a single component.<\/span> Whilst Metric Aggregation enables the same measure to be aggregated across multiple components of the same type. <\/span><\/p>\n
![\"\"](\"https:\/\/www.eginnovations.com\/blog\/wp-content\/uploads\/2023\/05\/Performance-Rating-vs.-Metric-Aggregation.webp\")
<\/a><\/p>\n![\"\"](\"https:\/\/www.eginnovations.com\/blog\/wp-content\/uploads\/2023\/05\/04_Metric-Aggregation.webp\")
<\/p>\n