JVM Threads Test

This test reports the status of threads on the JVM, and also reveals resource-hungry threads, so that threads that are unnecessarily consuming CPU resources can be killed.

Target of the test : An Oracle 9i Application Server

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for the server being monitored.

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	The IP address of the host for which this test is to be configured.
Port	Refers to the port at which the specified host listens to.
Mode	This test can extract metrics from the Java application using either of the following mechanisms: Using SNMP-based access to the Java runtime MIB statistics; By contacting the Java runtime (JRE) of the application via JMX To configure the test to use SNMP, select the SNMP option. On the other hand, choose the JMX option to configure the test to use JMX instead. By default, the JMX option is chosen here.
JMX Remote Port	This parameter appears only if the Mode is set to JMX. Here, specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in the management.properties file in the <JAVA_Home>\jre\lib\management folder used by the target application (refer to the Java Applications Monitoring Model for details).
User, Password, and Confirm Password	These parameters appear only if the Mode is set to JMX. If JMX requires authentication only (but no security), then ensure that the User and Password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to the Java Applications Monitoring Model. Confirm the password by retyping it in the Confirm Password text box.
JNDIName	This parameter appears only if the Mode is set to JMX. The JNDIName is a lookup name for connecting to the JMX connector. By default, this is jmxrmi. If you have registered the JMX connector in the RMI registery using a different lookup name, then you can change this default value to reflect the same.
Provider	This parameter appears only if the Mode is set to JMX. This test uses a JMX Provider to access the MBean attributes of the target Java application and collect metrics. Specify the package name of this JMX Provider here. By default, this is set to com.sun.jmx.remote.protocol.
Timeout	Specify the duration (in seconds) for which this test should wait for a response from the target Java application. If there is no response from the target beyond the configured duration, the test will timeout. By default, this is set to 240 seconds if the mode is JMX, and 10 seconds if the mode is SNMP.
SNMPPort	This parameter appears only if the Mode is set to SNMP. Here specify the port number through which the server exposes its SNMP MIB. Ensure that you specify the same port you configured in the management.properties file in the <JAVA_HOME>\jre\lib\management folder used by the target application (refer to the Java Applications Monitoring Model for details).
SNMPVersion	This parameter appears only if the Mode is set to SNMP. By default, the eG agent supports SNMP version 1. Accordingly, the default selection in the SNMPversion list is v1. However, if a different SNMP framework is in use in your environment, say SNMP v2 or v3, then select the corresponding option from this list.
SNMPCommunity	This parameter appears only if the Mode is set to SNMP. The SNMP community name that the test uses to communicate with the firewall. This parameter is specific to SNMP v1 and v2 only. Therefore, if the SNMPVersion chosen is v3, then this parameter will not appear.
UserName	This parameter appears only when v3 is selected as the SNMPVersion. SNMP version 3 (SNMPv3) is an extensible SNMP Framework which supplements the SNMPv2 Framework, by additionally supporting message security, access control, and remote SNMP configuration capabilities. To extract performance statistics from the MIB using the highly secure SNMP v3 protocol, the eG agent has to be configured with the required access privileges – in other words, the eG agent should connect to the MIB using the credentials of a user with access permissions to be MIB. Therefore, specify the name of such a user against this parameter.
Context	This parameter appears only when v3 is selected as the SNMPVersion. An SNMP context is a collection of management information accessible by an SNMP entity. An item of management information may exist in more than one context and an SNMP entity potentially has access to many contexts. A context is identified by the SNMPEngineID value of the entity hosting the management information (also called a contextEngineID) and a context name that identifies the specific context (also called a contextName). If the Username provided is associated with a context name, then the eG agent will be able to poll the MIB and collect metrics only if it is configured with the context name as well. In such cases therefore, specify the context name of the UserName in the Context text box. By default, this parameter is set to none.
AuthPass	Specify the password that corresponds to the above-mentioned UserName. This parameter once again appears only if the SNMPversion selected is v3.
Confirm Password	Confirm the AuthPass by retyping it here.
AuthType	This parameter too appears only if v3 is selected as the SNMPVersion. From the AuthType list box, choose the authentication algorithm using which SNMP v3 converts the specified username and password into a 32-bit format to ensure security of SNMP transactions. You can choose between the following options: MD5 – Message Digest Algorithm SHA – Secure Hash Algorithm
EncryptFlag	This flag appears only when v3 is selected as the SNMPVersion. By default, the eG agent does not encrypt SNMP requests. Accordingly, the this flag is set to No by default. To ensure that SNMP requests sent by the eG agent are encrypted, select the Yes option.
EncryptType	If this EncryptFlag is set to Yes, then you will have to mention the encryption type by selecting an option from the EncryptType list. SNMP v3 supports the following encryption types: DES – Data Encryption Standard AES – Advanced Encryption Standard
EncryptPassword	Specify the encryption password here.
Confirm Password	Confirm the encryption password by retyping it here.
Data Over TCP	This parameter appears only if the Mode is set to SNMP. By default, in an IT environment, all data transmission occurs over UDP. Some environments however, may be specifically configured to offload a fraction of the data traffic – for instance, certain types of data traffic or traffic pertaining to specific components – to other protocols like TCP, so as to prevent UDP overloads. In such environments, you can instruct the eG agent to conduct the SNMP data traffic related to the monitored target over TCP (and not UDP). For this, set this flag to Yes. By default, this flag is set to No.
Pct Low CPU Util Threads	This test reports the number of threads in the JVM that are consuming low CPU. This thread count will include only those threads for which the CPU usage is equal to or lesser than the value specified in the Pct Low CPU Util Threads text box. The default value displayed here is 30.
Pct Medium CPU Util Threads	This test reports the number of threads in the JVM that are consuming CPU to a medium extent. This thread count will include only those threads for which the CPU usage is higher than the Pct Low CPU Util Threads configuration and is lower than or equal to the value specified in the Pct Medium CPU Util Threads text box. The default value displayed here is 50.
Pct High CPU Util Threads	This test reports the number of threads in the JVM that are consuming high CPU. This thread count will include only those threads for which the CPU usage is either greater than the Pct Medium CPU Util Threads configuration, or is equal to or greater than the value specified in the Pct High CPU Util Threads text box. The default value displayed here is 70.
DD Frequency	Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 1:1. This indicates that, by default, detailed measures will be generated every time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency.
Detailed Diagnosis	To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled: The eG manager license should allow the detailed diagnosis capability Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Total threads	Indicates the total number of threads (including daemon and non-daemon threads).	Number
Runnable threads	Indicates the current number of threads in a runnable state.	Number	The detailed diagnosis of this measure, if enabled, provides the name of the threads, the CPU usage by the threads, the time for which the thread was in a blocked state, waiting state, etc.
Blocked threads	Indicates the number of threads that are currently in a blocked state.	Number	If a thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, then such a thread is called a blocked thread. The detailed diagnosis of this measure, if enabled, provides in-depth information related to the blocked threads.
Waiting threads	Indicates the number of threads that are currently in a waiting state.	Number	A thread is said to be in a Waiting state if the thread enters a synchronized block, tries to take a lock that is already held by another thread, and hence, waits till the other thread notifies that it has released the lock. Ideally, the value of this measure should be low. A very high value could be indicative of excessive waiting activity on the JVM. You can use the detailed diagnosis of this measure, if enabled, to figure out which threads are currently in the waiting state. While waiting, the Java application program does no productive work and its ability to complete the task-at-hand is degraded. A certain amount of waiting may be acceptable for Java application programs. However, when the amount of time spent waiting becomes excessive or if the number of times that waits occur exceeds a reasonable amount, the Java application program may not be programmed correctly to take advantage of the available resources. When this happens, the delay caused by the waiting Java application programs elongates the response time experienced by an end user. An enterprise may use Java application programs to perform various functions. Delays based on abnormal degradation consume employee time and may be costly to corporations.
Timed waiting threads	Indicates the number of threads in a TIMED_WAITING state.	Number	When a thread is in the TIMED_WAITING state, it implies that the thread is waiting for another thread to do something, but will give up after a specified time out period. To view the details of threads in the TIMED_WAITING state, use the detailed diagnosis of this measure, if enabled.
Low CPU threads	Indicates the number of threads that are currently consuming CPU lower than the value configured in the Pct Low CPU Util Threads text box.	Number
Medium CPU threads	Indicates the number of threads that are currently consuming CPU that is higher than the value configured in the Pct Low CPU Util Threads text box and is lower than or equal to the value specified in the Pct Medium CPU Util Threads text box.	Number
High CPU threads	Indicates the number of threads that are currently consuming CPU that is either greater than the percentage configured in the Pct Medium CPU Util Threads or lesser than or equal to the value configured in the Pct High CPU Util Threads text box.	Number	Ideally, the value of this measure should be very low. A high value is indicative of a resource contention at the JVM. Under such circumstances, you might want to identify the resource-hungry threads and kill them, so that application performance is not hampered. To know which threads are consuming excessive CPU, use the detailed diagnosis of this measure.
Peak threads	Indicates the highest number of live threads since JVM started.	Number
Started threads	Indicates the total number of threads started (including daemon, non-daemon, and terminated) since JVM started.	Number
Daemon threads	Indicates the current number of live daemon threads.	Number
Deadlock threads	Indicates the current number of deadlocked threads.	Number	Ideally, this value should be 0. A high value is a cause for concern, as it indicates that many threads are blocking one another causing the application performance to suffer. The detailed diagnosis of this measure, if enabled, lists the deadlocked threads and their resource usage.

Note:

If the Mode for the JVM Threads test is set to SNMP, then the detailed diagnosis of this test will not display the Blocked Time and Waited Time for the threads. To make sure that detailed diagnosis reports these details also, do the following:

Login to the server host.
Go to the <JAVA_HOME>\jre\lib\management folder used by the WebLogic server, and edit the management.properties file in that folder.
Append the following line to the file:

com.sun.management.enableThreadContentionMonitoring

Note:

While viewing the measures reported by the JVM Thread test, you can also view the resource usage details and the stack trace information for all the threads, by clicking on the stack trace link in the Measurements panel.

A stack trace (also called stack backtrace or stack traceback) is a report of the active stack frames instantiated by the execution of a program. It is commonly used to determine what threads are currently active in the JVM, and which threads are in each of the different states – i.e., alive, blocked, waiting, timed waiting, etc.

Typically, when the JVM begins exhibiting erratic resource usage patterns, it often takes administrators hours, even days to figure out what is causing this anomaly – could it be owing to one/more resource-intensive threads being executed by the WebLogic server? If so, what is causing the thread to erode resources? Is it an inefficient piece of code? In which case, which line of code could be the most likely cause for the spike in resource usage? To be able to answer these questions accurately, administrators need to know the complete list of threads that are executing on the JVM, view the stack trace of each thread, analyze each stack trace in a top-down manner, and trace where the problem originated.