{"id":18902,"date":"2021-12-09T07:49:23","date_gmt":"2021-12-09T12:49:23","guid":{"rendered":"https:\/\/www.eginnovations.com\/blog\/?p=18902"},"modified":"2025-07-04T05:02:09","modified_gmt":"2025-07-04T09:02:09","slug":"cloud-performance-issues","status":"publish","type":"post","link":"https:\/\/www.eginnovations.com\/blog\/cloud-performance-issues\/","title":{"rendered":"Case Study: AWS Cloud Application Performance Troubleshooting"},"content":{"rendered":"
\n

<\/span>How a full stack monitoring solution helped our customer with Application Performance Troubleshooting on AWS Cloud<\/span><\/h2>\n
\n

<\/span>Summary<\/strong><\/span><\/h4>\n

\"AWS
\nHere’s a myth that needs to be debunked – the cloud (e.g., AWS or Azure) will take care of my performance problems!<\/p>\n

\"AI<\/p>\n

Our experience shows that cloud architecture usually introduces new layers of complexities that did not exist in the on-premises world. You need a modern AI-powered full stack monitoring solution to find the needle in the multi-layered haystack that is the cloud.<\/p>\n

\"\"<\/p>\n

Sometimes, it’s the cloud vendor who has to fix the issue. An example could be a noisy or defective physical host OS that you have no access or visibility into. You need the right information in the form of logs, metrics, traces and events to substantiate with evidence in conversations with the cloud provider’s support team.<\/p>\n

\"\"<\/p>\n

In this blog post, we describe a problem analysis and anomaly detection process for a cloud performance problem (high CPU in the JVM and SSL issues) that we encountered recently when working with a large customer, who had a significant footprint on AWS cloud.<\/p>\n<\/div>\n

<\/span>The Cloud Does Not Make Application Performance Monitoring and Troubleshooting Simpler<\/span><\/h2>\n

\"CloudCloud migration and digital transformation of business-critical applications on the cloud are on the rise. At the same time, some IT executives have the misconception that when applications are migrated to the cloud, monitoring their performance becomes easier. The cloud service provider (e.g., Amazon) takes care of most of the key elements that support the application, and hence, they believe that there are fewer challenges when applications are on the cloud vs. being deployed on-premises.<\/p>\n

This is often far from true.<\/p>\n

This case study exposes the challenges that organizations deploying applications on the cloud face when performance issues happen and how difficult it can be to diagnose such problems. While migration to the cloud does simplify and automate several routine administration activities, performance monitoring is one area that is not necessarily simplified!<\/p>\n

The scenario covered may be useful for those evaluating the level of insight and troubleshooting capabilities various cloud tools can offer.<\/p>\n

<\/span>Target Application and Infrastructure on AWS Cloud<\/span><\/h2>\n

\"AWS.<\/p>\n

Our customer is a mid-sized business that offers SaaS services to clients. The SaaS application was Java-based and was hosted on AWS cloud, and relied on AWS RDS for the backend database. eG Enterprise was used by the customer to monitor the SaaS application and the AWS services in use.<\/p>\n

The application had been operational for several months and there had been no complaints about performance.<\/p>\n

<\/span>The Initial Alert<\/span><\/h3>\n

The CPU usage of the AWS EC2 VM<\/a> hosting the key application had spiked all of a sudden around 5 am and remained high for several hours:<\/p>\n

\"CPU<\/a><\/p>\n

Figure 1: CPU usage of the VM had spiked suddenly<\/div>\n

At this time, application performance also suffered. This was evident from eG Enterprise\u2019s synthetic monitoring capability<\/a> that the customer had configured to proactively detect issues before real users experience issues. Initially when the problem started, application response time was poor. Over time, you see gaps in the response time in the graph in Figure 2 because the application became unavailable.<\/p>\n

\"Application<\/a><\/p>\n

Figure 2: Response time of the application, measured using synthetic monitoring<\/div>\n

Below is the graph of application availability during the same time scale. You can clearly see TCP connection availability dropping to zero several times indicating that users were not able to connect to the application. This information was also highlighted in critical alerts sent to administrators and displayed on the overview dashboards in eG Enterprise to enable actionable notification.<\/p>\n

\"AWS<\/a><\/p>\n

Figure 3: Application availability measured using synthetic monitoring<\/div>\n

<\/span>Analyzing Application Performance in Detail<\/span><\/h3>\n

To further analyze the problem, we analyzed the application performance in detail. Figure 4 below shows the CPU usage of the JVM (Java Virtual Machine)<\/a> used by the application on the problematic AWS VM. The JVM CPU usage follows the same pattern as the VM\u2019s CPU usage, indicating that the application\u2019s JVM had been affected.<\/p>\n

\"CPU<\/a><\/p>\n

Figure 4: CPU usage of the application’s JVM<\/div>\n

High CPU usage within the JVM can have numerous causes and often it can be a symptom of bugs within the Java code of the application. High CPU is a symptom – not a root cause. Reasons for high CPU could range from host OS issues, poorly sized JVMs, memory leaks to code-level deadlocks. My colleague has written a blog covering some common problems, see: How to Troubleshoot Java CPU Usage Issues | JVM High CPU Threads (eginnovations.com).<\/a><\/p>\n

\n

<\/span>The impact of Java Garbage Collection on application performance<\/strong><\/span><\/h4>\n

Java Garbage Collection (GC) is intrinsically a CPU intensive operation.<\/p>\n

\"ExcessiveWhen Java GC happens, all application threads are paused. Most Java GC algorithms (including the latest G1GC) must halt all application threads, a process referred to as stopping-the-world (STW) or pausing (a GC pause).<\/p>\n

The JVM (Java virtual machine) takes over all the computer’s cores to perform GC and reclaims memory before restarting the application threads.<\/p>\n

Best Practice:<\/strong> Look for full stack monitoring tools that can help you diagnose high CPU root-cause across various layers and tiers \u2013 Operating System, JVM (threads\/ heap), application code and connect the dots to the affected business transactions so you can quantify business impact. Full stack tools can correlate GC activity at the JVM level, thread activity at the code level and CPU utilization% even at a request-by-request level. This will help you triage faster and engage the right team to fix the issue.<\/p>\n<\/div>\n

The JVM Garbage Collector (GC)<\/a> can also be a source of high CPU, especially if application memory leaks are at play. The JVM knowing that it is running low on key resources, such as heap memory can get itself into a state where it keeps trying to desperately reclaim memory. A quick check eliminated the possibility of a GC issue in this case. See Figure 5, which shows the historical usage of GC in the JVM. The percentage of time spent by the JVM on GC activities had not changed significantly during the problematic period. Hence, GC activities were not the reason why the application\u2019s CPU usage had spiked.<\/p>\n

\"GC<\/a><\/p>\n

Figure 5: GC activity of the Java application indicated no anomalies<\/div>\n

Java threading issues are a common problem that cause application problems. So, the next step was to check if there were common threading issues present, and if so, were these root cause issues or simply manifestations and symptoms of issues elsewhere. Unfortunately, both are common possibilities!<\/p>\n

Figure 6 shows the total number of threads in the JVM over time. While there had been an increase, the increase was not too significant. This indicated that application processing in the JVM was not the issue.<\/p>\n

\"Application<\/a><\/p>\n

Figure 6: Tracking the number of threads in the application’s JVM<\/div>\n
\n

<\/span>3 Use Cases for which continuous thread analysis is key<\/span><\/h3>\n

\"\"Use case #1:<\/strong> Identify the dreaded Achilles heel of threaded applications \u2013 locks and deadlocks that impact your scalability.<\/p>\n

Use case #2:<\/strong> Automatically identify CPU-hungry threads and thread groups. On the cloud, CPU is money.<\/p>\n

Use case #3:<\/strong> Pinpoint the root-cause of the thread anomalies to the specific processes and microservices so the right team can be alerted to fix the issue.<\/p>\n<\/div>\n

The Java Thread Analysis modules within eG Enterprise further indicated that no specific thread in the JVM was taking a lot of CPU. There were many threads, each taking a small amount of CPU \u2013 1-2%, but a number of these threads caused the overall CPU to be high.<\/p>\n

\"Thread<\/a><\/p>\n

Figure 7: Details of threads in the JVM<\/div>\n

The auto-baselined thresholds for blocked threads had however triggered critical alerts when the problem happened. Reviewing the historical data, we could see that there was some significant thread blocking in the JVM, which was abnormal and anomalous. A quick check in the detailed diagnosis tool in eG Enterprise (see Figure 8) revealed there were issues in SSL processing at the JVM level \u2013 not in the application code.<\/p>\n

\"Thread<\/a><\/p>\n

Figure 8: Detailed diagnosis shows that thread blocking happened in the JVM<\/div>\n

Clicking through to identify the blocking thread, showed that SSL Memory Caching in the JVM seemed to have triggered the issue (see Figure 9). Many Java threads were stuck in SSL processing \u2013 and it was this that was consuming excessive CPU.<\/p>\n

\"Thread<\/a><\/p>\n

Figure 9: Thread blocking caused by synchronized access to the SSL Memory Cache in the JVM<\/div>\n

By simply googling the blocked class\/method, i.e., \u201csun.security.util.MemoryCache.put\u201d, one can find a number of links that point to known Java issues:<\/p>\n