{"id":32820,"date":"2023-09-12T09:07:53","date_gmt":"2023-09-12T13:07:53","guid":{"rendered":"https:\/\/www.eginnovations.com\/blog\/?p=32820"},"modified":"2025-03-18T06:30:51","modified_gmt":"2025-03-18T10:30:51","slug":"troubleshooting-spring-boot-microservices-a-real-world-case-study","status":"publish","type":"post","link":"https:\/\/www.eginnovations.com\/blog\/troubleshooting-spring-boot-microservices-a-real-world-case-study\/","title":{"rendered":"Troubleshooting Spring Boot Microservices – A Real World Case Study"},"content":{"rendered":"
\n

Today, I\u2019ll cover Shift Left Monitoring: A Pathway to Optimized Cloud Applications<\/a> and how left-shifted troubleshooting of Spring Boot code issues using observability tooling can avoid production issues, unnecessary costs and improve product quality.<\/p>\n

\"\" Shift-left is an approach to software development and operations that emphasizes testing, monitoring, and automation earlier in the software development lifecycle. The goal of the shift-left approach is to prevent problems before they arise by catching them early and addressing them quickly.<\/p>\n

When you identify a scalability issue or a bug early, it is quicker and more cost-effective to resolve it. Moving inefficient code to cloud containers can be costly, as it may activate auto-scaling and increase your monthly bill. Furthermore, you will be in a state of emergency until you can identify, isolate, and fix the issue.<\/p>\n

<\/span>Troubleshooting Java Spring Boot Pre-production<\/span><\/h2>\n

This is a recent real-world case where the data from their observability solution empowered a customer to avert a potential issue with an application that could have caused a major issue in a production environment.<\/p>\n

The customer was testing a new version of a Java Spring Boot microservice using MariaDB<\/a> as the backend, running behind Apache<\/a> reverse proxy and AWS<\/a> application load balancer. The end-to-end system was monitored using eG Enterprise throughout the testing cycle both prior to integrating the application changes and afterwards. Pre-production UAT (User Acceptance Testing) was performed and all UAT test cases passed.<\/p>\n

However, on review of the eG Enterprise performance reports of the UAT (User Acceptance Testing) infrastructure following the recent application changes. The performance charts in the MariaDB performance dashboard deviated significantly from pre-deployment patterns.<\/p>\n

On investigation a serious coding issue was discovered highlighting the importance of baselining both application and infrastructure behavior in such testing. Later I\u2019ll also cover why the application UAT tests succeeded.<\/p>\n

I\u2019ll walk-through the key metrics and information captured by eG Enterprise that allowed the customer to diagnose and analyze the full-stack behavior.<\/p>\n

\n

<\/span>Spring Boot and Kubernetes<\/span><\/h2>\n

Spring Boot is an open-source Java-based framework that simplifies the development of production-ready, stand-alone, and web-based applications. Kubernetes (K8s) is designed to manage containerized applications, and Spring Boot applications are well-suited for microservices architectures. Spring Boot’s lightweight and modular nature allows developers to create containerized microservices easily, and Kubernetes provides an excellent platform for orchestrating and scaling these services.<\/p>\n<\/div>\n

<\/span>The Timeline of the Java Spring Boot Application Migration<\/span><\/h2>\n

On August 6th at 14:13, The application was restarted with a new Spring Boot jar file containing an embedded Tomcat.<\/p>\n

\"\"<\/a><\/p>\n

Figure 1: eG Enterprise automatically captures all JVM restarts<\/div>\n

At 14:52, the query processing rate for MariaDB increased from 0.1 to 88 queries per second and then to 301 queries per second.<\/p>\n

\"\"<\/a><\/p>\n

Figure 2: eG Enterprise measured a significant increase in query rate for MariaDB shortly after the JVM restart<\/div>\n

Additionally, the system CPU was elevated from 1% to 6%.<\/p>\n

\"\"<\/a><\/p>\n

Figure 3: eG Enterprise capturing the change in CPU utilization<\/div>\n

Finally, the JVM<\/a> time spent on the G1 Young Generation Garbage Collection<\/a> increased from 0% to 0.1% and remained at that level.<\/p>\n

\"\"<\/a><\/p>\n

Figure 4: eG Enterprise visualizing the increase in GC time on the JVM<\/div>\n

The application, in its UAT phase, was behaving abnormally and issuing 300 queries\/sec, which is far beyond what it was designed to do. The new feature caused an increase in database connections, which is why the increase in queries is so dramatic.<\/p>\n

The monitoring dashboards also showed that the problematic metrics had been normal before the new version was deployed.<\/p>\n

<\/span>The Resolution of the Spring Boot Problems<\/span><\/h2>\n
\n

<\/span>What is JPA?<\/span><\/h2>\n

JPA<\/a> (Java Persistence API) is a Java-based specification and API for managing relational data in applications using Java. JPA provides a set of interfaces and standard annotations that allow Java developers to interact with relational databases in a way that is both portable and easy to use.<\/p>\n<\/div>\n

The application is a Spring Boot application that uses JPA to query a MariaDB. The application is designed to run on two containers when experiencing a minimal load but is expected to scale up to ten under peak demand.<\/p>\n

\"\"<\/p>\n

Figure 5: eG Enterprise topology maps show the dependencies of the full-stack including the web server – app sever – DB topology<\/div>\n

Logical questions naturally arise – If a single container can generate 300 queries per second, can the stack handle 3000 queries per second if all ten containers are operational? Will the database have enough connections to meet the needs of the other parts of the application?<\/p>\n

At this point we investigated the changes committed by the developer in Git.<\/a> The changes were relatively minor and added simple functionality to take a few records from a table and process them. However, this is what we observed in the service class.<\/p>\n

List findAll = this.xRepository.findAll();<\/span><\/p>\n

The unqualified use of a findAll()<\/span> method without pagination<\/a> in Spring’s CrudRepository which is not efficient. Most basic RDBMS tutorials cover this \u2013 like the use of WHERE in SQL, it is more efficient to request a subset of records by certain criteria than request all records and process them on similar criteria. Pagination helps to reduce the amount of time it takes to retrieve data from the database by limiting the amount of data fetched. Additionally, pagination helps to keep memory usage low to prevent the application from crashing due to an overload of data, as well as reducing the Garbage Collection effort of Java Virtual Machine, which was mentioned in the problem statement above.<\/p>\n

This test was conducted using only 2,000 records in one container. If this code were to move to production, where there are around 200,000 records in up to 10 containers, it could have caused the team a lot of stress and worry that day.<\/p>\n

The application was rebuilt with the addition of a \u201cBy\u201d clause, which is equivalent to the role of a WHERE<\/em> clause in SQL, to the method.<\/p>\n

List findAll = this.xRepository.findAllByY(Y);<\/span><\/p>\n

After making this change and redeploying the application, normal behavior was restored. The number of queries per second decreased from 300 to 30, and the amount of effort put into garbage collection returned to its original level. Additionally, the system’s CPU usage decreased.<\/p>\n

\"\"<\/a><\/p>\n

Figure 6: eG Enterprise data measuring the query rate returning to normal expected behavior<\/div>\n

<\/span>Improving and Tuning Observability and Monitoring for Pre-Production Testing with eG Enterprise<\/span><\/h2>\n

In this case, having observability expertise involved in reviewing the data overviews quickly identified the issues. Incidents like this one empower organizations to iterate and refine their monitoring during testing to automate issue diagnostics further.<\/p>\n

In this case the UAT tests did not realistically reflect the production environment usage and scalability expectations. Long-term the customer will refine their test coverage.<\/p>\n

The test system and the associated metric thresholds used for eG Enterprise alerting were out-of-the-box designed for the customers\u2019 production systems and naturally did not trigger with a CPU usage of 6%. Experienced users can however tune such thresholds to leverage auto-baselining whereby the AIOps engine<\/a> and machine learning technologies establish the normal behavior of a system and raise alerts automatically when anomalies occur. In this case the customer may wish to tune their thresholds during such testing to raise alerts when infrastructure resources such as CPU deviate by 200% of established behavior. Information on leveraging eG Enterprise\u2019s powerful auto-baselining and anomaly detection is covered in: White Paper | Make IT Service Monitoring Simple & Proactive with AIOps Powered Intelligent Thresholding & Alerting (eginnovations.com).<\/a><\/p>\n

The code changes were known to be minor and limited to Git check-ins by a single developer and so the customer sourced the coding issues directly from Git. In many scenarios though a new version of an application will integrate hundreds of changes from multiple developers within multiple Java methods and diagnosing the exact method that is the root-cause of the issue will be a tedious and often manual task. In such scenarios, the full stack DevOps monitoring capability of eG Enterprise can be leveraged to identify exact method call that is causing slowness. eG Innovations brings DevOps monitoring under a unified console so that the administrator has full visibility of the roll-out process. eG Enterprise supports important components of the CI-CD pipeline such as GitHub, JenKins, Docker, K8s, BitBucket, RedHat Ansible Tower, Jira etc. See: Identifying and Diagnosing a CPU Issue in the JVM (eginnovations.com)<\/a> and Java Code Level Visibility | eG Innovations.<\/a><\/p>\n

<\/span>Learning and Summary<\/span><\/h2>\n

Anyone who works in Site Reliability Engineering (SRE) will appreciate the significance of this discovery. We were able to act upon it without having to raise a Severity 1 flag. Without the pre-production troubleshooting of this flawed Spring Boot package, in production it could have triggered the customer’s auto-scaling threshold, resulting in new containers being launched even without any additional user load.<\/p>\n

There are three main takeaways from this story.<\/p>\n