Monitoring and Troubleshooting Synchronization Issues in Java Applications

Why Synchronize Java Threads?

Java threads can be used for performing different tasksJava applications deployed in production must be multi-threaded for scalability. Threads can be used for performing different tasks or they could be executing the same task to service different user requests.

Often, it is necessary to synchronize between different threads. For example, in an eCommerce application, different threads may be fulfilling different orders but to get an accurate count of the number of orders processed, synchronization may be used when incrementing the count of orders processed. Synchronization may be used to coordinate accesses when multiple threads write to a common location (e.g., a memory variable) or to coordinate between read and write activities.

Synchronization is Very Easy in Java

While synchronization in many other programming languages was difficult to implement, the Java programming language makes it extremely simple to synchronize between threads. You can define any method as being synchronized just by adding the “synchronized” keyword in the method definition. When a method is synchronized, the code inside that method will be executed by only one thread at a time.

class MyPrinter {
	public synchronized void print10(int value) {
		for(int i = 10; i < 10; i++) {
			System.out.print(value);
		}
	System.out.println(""); // newline after 10 numbers
	}
}

Synchronized method

You can even synchronize code blocks (instead of entire methods) on a common object. The common object may be an instance of a class or it could be a static instance of a class. In the former case, synchronization is limited to all accesses to that instance, whereas in the latter case, all accesses to all instances of the class will be synchronized.

public void add(int value) {
	Student s = new Student();
		synchronized(s) {
		this.count += value;
	}
}

Synchronized block

You can also synchronize between web requests by having servlets that implement a single threaded model. In this case, requests to the servlet will wait while the executing request is being processed.

import java.io.*;
import javax.servlet.*;
import javax.servler.http.*;

public class MyServlet extends HttpServlet  
	implements SingleThreadModel {
	servlet methods...
}

Single threaded servlet

In the above cases, the developer has explicitly chosen to synchronize between the executions of different threads. Many a time, you may even be using synchronization without explicitly knowing. Access to collection classes such as Vector and Hashtable are synchronized, even though you may not explicitly be accessing them from within synchronized blocks.

Multiple Java Threads

Synchronization can Impact Java Application Performance

Many Java developers and programmers use synchronization constructs without thinking about their performance implications. There are two types of overheads associated with synchronization:

  • First is the need to manage the lock and unlock activities in the JVM. This will result in lower throughput.
  • Furthermore, because only one thread can enter into a synchronized block or access a synchronized object, unnecessary use of synchronization can make the application slow. The larger the number of Java threads contending to execute a synchronized block or method, the worse the problem gets. Ultimately, this may show up as applications being very slow or becoming unresponsive.

Many Monitoring Tools“Synchronized areas of code affect performance in two ways. First, the amount of time an application spends in a synchronized block affects the scalability of an application. Second, obtaining the synchronization lock requires some CPU cycles and hence affects performance.”

Scott Oaks
Author of Java Performance: The Definitive Guide

Monitoring Java Thread Synchronization

When your Java application is slow, how do you know whether it is thread synchronization that is the cause of slowness? Many things can go wrong with multi-threaded Java applications and reproducing issues can be a nightmare. Debugging and troubleshooting can take hours and even days or weeks.

To monitor thread synchronization activities of Java applications, it is essential to monitor thread activity inside the JVM. Java Management Instrumentation (JMX) provides ways for monitoring tools to collect and report on thread activity. There are built-in tools like JConsole and JVisualVM that allow you to look at what threads exist in the JVM, their current state and what line of code they are executing. However, these tools are only appropriate to look at the JVM in real-time.

Historical analysis of thread activity is extremely important. You may encounter a situation where the Java application suddenly hung at midnight and your helpdesk staff may have restarted the application to bring it back up. Without historical insights, you will have no clue as to what caused the problem: was the JVM short on memory? Was garbage collection taking time? Was there a thread blocking issue? Could it be caused by a database bottleneck? Having the ability to look back in time, to see what happened in the JVM when the problem occurred is a key to effective diagnostics. This historical data can also be shared by operations with developers, so problems can be quickly identified and resolved.

“Even when a program contains only a single thread running on a single processor, a synchronized method call is still slower than an un-synchronized method call. If the synchronization actually requires contending for the lock, the performance penalty is substantially greater.”

John La Rooy
Senior Software Engineer at Planet Innovation

Historical analysis of JVM thread activity provides interesting insights. One of the key metrics to look at is the number of blocked threads in the JVM. A blocked thread is one that is waiting on a synchronized block or method for another thread to complete its execution of the same block or method.

Except in rare situations, a large number of blocked threads should not be a common occurrence. The operations team will want to be alerted to situations when thread blocking is high. For quick diagnosis, it is important to be able to see which threads are blocked, at which method/line of code, and which other thread is blocking this thread. Having this information both in real-time and historically is a key to quickly diagnosing Java performance bottlenecks.

Java monitoring helps identify deadlocked threads
How Java monitoring helps identify deadlocked threads

While thread blocking is a problem, deadlocks are an even bigger problem. When a deadlock occurs, the Java application may hang completely and become unresponsive.

Deadlocks occur because of poor coding – one thread usually acquires a lock and then it waits on a second thread which has acquired a lock and is waiting on the first thread to finish. As with blocked threads, during deadlock situations, it is important to know the names of the threads that are deadlocked and which line of Java code each of these threads is executing.

Stack trace in the Java thread diagnosis identifies the line of code causing deadlock
Looking up the stack trace to identify the line of code causing Java thread deadlock

Best Practices to Improve Java Application Performance and Scalability

Here are some best practices that you should take into account if you want to design a high-performance Java application:

Best Practice #1: Concurrency is tricky, so keep it to a minimum

  • Synchronization has a big cost. Use it only when absolutely necessary knowing that you are allowing only one thread at a time in the JVM to execute that method or code.
  • Keep the lines of code that must be synchronized to a minimum, so a thread spends the shortest possible time within a synchronized block.
  • Even when a program contains only a single thread running on a single processor, a synchronized method call is still slower than an unsynchronized method call.
  • You’ve done the hard work of limiting the scope, but do you have locks that are declared public? Make them protected or private. Someone else might lock your carefully picked objects, thus aggravating the situation.

Best Practice #2: Pick wisely after having done your research

  • Know which built-in Java classes are designed to be thread-safe, i.e., have synchronization built-in to their access methods. Use such classes judiciously. For example, if all your manipulations are within the context of a single thread, use ArrayList instead of Vector and HashMap instead of Hashtable.
  • Where possible, synchronize on instances – not on classes (i.e., globally).
  • Should you use synchronization or locks? Read the java docs and pick as appropriate for your use case.

Best Practice #3: Perform an audit and upgrade your APIs where necessary

  • Over the years, Java has added more tools in the toolbox. Java 5 and Java 6 added concurrent alternatives for synchronized ArrayList, Hashtable and synchronized HashMap collection classes. The java.util.concurrent package contains ConcurrentHashMap and BlockingQueue collection classes that can be used to build more scalable multi-threaded applications. The new concurrent collection classes offer significant performance wins.

Helpful Resources