JVM Threads Test

This test reports the status of threads running in the JVM. Details of this test can be used to identify resource-hungry threads.

Target of the test : A Java Application

Agent deploying the test : An internal agent

Outputs of the test : One set of results for the Java application being monitored.

Configurable parameters for the test
Parameter Description

Test Period

How often should the test be executed.

Host

The IP address of the host for which this test is to be configured.

Port

Refers to the port at which the specified host listens to.

Measurement Mode

This test can extract metrics from Tomcat using either of the following mechanisms:

  • By deploying the egtomcat.war file in the <EG_INSTALL_DIR>\lib directory of the eG agent host on the Tomcat server;
  • By contacting the Java runtime (JRE) of Tomcat via JMX

To configure the test to use egtomcat.war file, first, select the War File option. Then, refer to How does eG Enterprise Monitor Tomcat Servers? to know how to deploy the WAR file on the target Tomcat server.

On the other hand, if you want the test to use JMX instead, then first, select the JMX option. Then, follow the procedure detailed in the Configuring and Monitoring Tomcat Servers document to configure the test to use JMX. By default, the JMX option is chosen here.

JMX Remote Port

This parameter appears only if the Mode is set to JMX. Here, specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in the management.properties file in the <JAVA_Home>\jre\lib\management folder used by the target application (refer to the Monitoring Java Applications for details).

User, Password, and Confirm Password

These parameters appear only if the Mode is set to JMX. If JMX requires authentication only (but no security), then ensure that the User and Password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to the Monitoring Java Applications. Confirm the password by retyping it in the Confirm Password text box.

JNDIName

This parameter appears only if the Mode is set to JMX. The JNDIName is a lookup name for connecting to the JMX connector. By default, this is jmxrmi. If you have registered the JMX connector in the RMI registery using a different lookup name, then you can change this default value to reflect the same. 

JMX Provider

This parameter appears only if the Mode is set to JMX. This test uses a JMX Provider to access the MBean attributes of the target Java application and collect metrics. Specify the package name of this JMX Provider here. By default, this is set to com.sun.jmx.remote.protocol.

Timeout

Specify the duration (in seconds) for which this test should wait for a response from the target Java application. If there is no response from the target beyond the configured duration, the test will timeout. By default, this is set to 240 seconds if the mode is JMX, and 10 seconds if the mode is SNMP.

SNMPPort

This parameter appears only if the Mode is set to SNMP. Here specify the port number through which the server exposes its SNMP MIB. Ensure that you specify the same port you configured in the management.properties file in the <JAVA_HOME>\jre\lib\management folder used by the target application (refer to the Monitoring Java Applications for details).

SNMPVersion

This parameter appears only if the Mode is set to SNMP. By default, the eG agent supports SNMP version 1. Accordingly, the default selection in the SNMPversion list is v1. However, if a different SNMP framework is in use in your environment, say SNMP v2 or v3, then select the corresponding option from this list.

SNMPCommunity

This parameter appears only if the Mode is set to SNMP. The SNMP community name that the test uses to communicate with the firewall. This parameter is specific to SNMP v1 and v2 only. Therefore, if the SNMPVersion chosen is v3, then this parameter will not appear.

UserName

This parameter appears only when v3 is selected as the SNMPVersion. SNMP version 3 (SNMPv3) is an extensible SNMP Framework which supplements the SNMPv2 Framework, by additionally supporting message security, access control, and remote SNMP configuration capabilities. To extract performance statistics from the MIB using the highly secure SNMP v3 protocol, the eG agent has to be configured with the required access privileges – in other words, the eG agent should connect to the MIB using the credentials of a user with access permissions to be MIB. Therefore, specify the name of such a user against this parameter. 

Context

This parameter appears only when v3 is selected as the SNMPVersion. An SNMP context is a collection of management information accessible by an SNMP entity. An item of management information may exist in more than one context and an SNMP entity potentially has access to many contexts. A context is identified by the SNMPEngineID value of the entity hosting the management information (also called a contextEngineID) and a context name that identifies the specific context (also called a contextName). If the Username provided is associated with a context name, then the eG agent will be able to poll the MIB and collect metrics only if it is configured with the context name as well. In such cases therefore, specify the context name of the UserName in the Context text box.  By default, this parameter is set to none.

AuthPass

Specify the password that corresponds to the above-mentioned UserName. This parameter once again appears only if the SNMPversion selected is v3.

Confirm Password

Confirm the AuthPass by retyping it here.

PCT Medium CPU Util Threads

By default, the PCT Medium CPU Util Threads parameter is set to 50. This implies that, by default, the threads for which the current CPU consumption is between 50% and 70% (the default value of the PCT High CPU Util Threads parameter) will be counted as medium CPU-consuming threads. The count of such threads will be reported as the value of the Medium CPU threads measure.

This default setting also denotes that threads that consume less than 50% CPU will, by default, be counted as Low CPU threads. If need be, you can modify the value of this PCT Medium CPU Util threads parameter to change how much CPU should be used by a thread for it to qualify as a medium CPU-consuming thread. This will consequently alter the count of low CPU-consuming threads as well.

PCT High CPU Util Threads

By default, the PCT High CPU Util Threads parameter is set to 70. This implies that, by default, the threads that are currently consuming over 70% of CPU time are counted as high CPU consumers. The count of such threads will be reported as the value of the High CPU threads measure. If need be, you can modify the value of this parameter to change how much CPU should be used by a thread for it to qualify as a high CPU-consuming thread.

AuthType

This parameter too appears only if v3 is selected as the SNMPVersion. From the AuthType list box, choose the authentication algorithm using which SNMP v3 converts the specified username and password into a 32-bit format to ensure security of SNMP transactions. You can choose between the following options:

  • MD5 – Message Digest Algorithm
  • SHA – Secure Hash Algorithm

EncryptFlag

This flag appears only when v3 is selected as the SNMPVersion. By default, the eG agent does not encrypt SNMP requests. Accordingly, the this flag is set to No by default. To ensure that SNMP requests sent by the eG agent are encrypted, select the Yes option. 

EncryptType

If this EncryptFlag is set to Yes, then you will have to mention the encryption type by selecting an option from the EncryptType list. SNMP v3 supports the following encryption types:

  • DES – Data Encryption Standard
  • AES – Advanced Encryption Standard

EncryptPassword

Specify the encryption password here.

Confirm Password

Confirm the encryption password by retyping it here.

Data Over TCP

This parameter appears only if the Mode is set to SNMP. By default, in an IT environment, all data transmission occurs over UDP. Some environments however, may be specifically configured to offload a fraction of the data traffic – for instance, certain types of data traffic or traffic pertaining to specific components – to other protocols like TCP, so as to prevent UDP overloads. In such environments, you can instruct the eG agent to conduct the SNMP data traffic related to the monitored target over TCP (and not UDP). For this, set this flag to Yes. By default, this flag is set to No.

Useps

This flag is applicable only for AIX LPARs. By default, on AIX LPARs, this test uses the tprof command to compute CPU usage. Accordingly, the UsePS flag is set to No by default. On some AIX LPARs however, the tprof command may not function properly (this is an AIX issue). While monitoring such AIX LPARs therefore, you can configure the test to use the ps command instead for metrics collection. To do so, set the Useps flag to Yes.

Note:

Alternatively, you can set the AIXusePS flag in the [AGENT_SETTINGS] section of the eg_tests.ini file (in the <EG_INSTALL_DIR>\manager\config directory) to yes (default: no) to enable the eG agent to use the ps command for CPU usage computations on AIX LPARs. If this global flag and the Useps flag for a specific component are both set to no, then the test will use the default tprof command to compute CPU usage for AIX LPARs. If either of these flags is set to yes, then the ps command will perform the CPU usage computations for monitored AIX LPARs.  

In some high-security environments, the tprof command may require some special privileges to execute on an AIX LPAR (eg., sudo may need to be used to run tprof). In such cases, you can prefix the tprof command with another command (like sudo) or the full path to a script that grants the required privileges to tprof. To achieve this, edit the eg_tests.ini file  (in the <EG_INSTALL_DIR>\manager\config directory), and provide the prefix of your choice against the AixTprofPrefix parameter in the [AGENT_SETTINGS] section. Finally, save the file.  For instance, if you set the AixTprofPrefix parameter to sudo, then the eG agent will call the tprof command as sudo tprof.

DD Frequency

Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 1:1. This indicates that, by default, detailed measures will be generated every time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency. 

Detailed Diagnosis

To make diagnosis more efficient and accurate, the eG Enterprise suite embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability
  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measurements made by the test
Measurement Description Measurement Unit Interpretation

Total threads

Indicates the total number of threads (including daemon and non-daemon threads).

Number

 

Runnable threads

Indicates the current number of threads in a runnable state.

Number

The detailed diagnosis of this measure, if enabled, provides the name of the threads, the CPU usage by the threads, the time for which the thread was in a blocked state, waiting state, etc.

Blocked threads

Indicates the number of threads that are currently in a blocked state.

Number

If a thread is trying to take a lock (to enter a synchronized block), but the lock is already held by another thread, then such a thread is called a blocked thread.

The detailed diagnosis of this measure, if enabled, provides in-depth information related to the blocked threads.

Waiting threads

Indicates the number of threads that are currently in a waiting state.

Number

A thread is said to be in a Waiting state if the thread enters a synchronized block, tries to take a lock that is already held by another thread, and hence, waits till the other thread notifies that it has released the lock.

Ideally, the value of this measure should be low. A very high value could be indicative of excessive waiting activity on the JVM. You can use the detailed diagnosis of this measure, if enabled, to figure out which threads are currently in the waiting state.

While waiting, the Java application program does no productive work and its ability to complete the task-at-hand is degraded. A certain amount of waiting may be acceptable for Java application programs. However, when the amount of time spent waiting becomes excessive or if the number of times that waits occur exceeds a reasonable amount, the Java application program may not be programmed correctly to take advantage of the available resources. When this happens, the delay caused by the waiting Java application programs elongates the response time experienced by an end user. An enterprise may use Java application programs to perform various functions. Delays based on abnormal degradation consume employee time and may be costly to corporations.

Timed waiting threads

Indicates the number of threads in a TIMED_WAITING state.

Number

When a thread is in the TIMED_WAITING state, it implies that the thread is waiting for another thread to do something, but will give up after a specified time out period.

To view the details of threads in the TIMED_WAITING state, use the detailed diagnosis of this measure, if enabled.

Low CPU threads

Indicates the number of threads that are currently consuming CPU lower than the value configured in the PCT Medium CPU Util Threads text box.

Number

 

Medium CPU threads

Indicates the number of threads that are currently consuming CPU that is higher than the value configured in the PCT Medimum CPU Util Threads text box and is lower than or equal to the value specified in the PCT High CPU Util Threads text box.

Number

 

High CPU threads

Indicates the number of threads that are currently consuming CPU that is greater than the percentage configured in the PCT High CPU Util Threads text box.

Number

Ideally, the value of this measure should be very low. A high value is indicative of a resource contention at the JVM. Under such circumstances, you might want to identify the resource-hungry threads. To know which threads are consuming excessive CPU, use the detailed diagnosis of this measure.

Peak threads

Indicates the highest number of live threads since JVM started.

Number

 

Total threads

Indicates the total number of threads started (including daemon, non-daemon, and terminated) since JVM started.

Number

 

Daemon threads

Indicates the current number of live daemon threads.

Number

 

Deadlock threads

Indicates the current number of deadlocked threads.

Number

Ideally, this value should be 0. A high value is a cause for concern, as it indicates that many threads are blocking one another causing the application performance to suffer. The detailed diagnosis of this measure, if enabled, lists the deadlocked threads and their resource usage.

Note:

If the Mode for the JVM Threads test is set to SNMP, then the detailed diagnosis of this test will not display the Blocked Time and Waited Time for the threads. To make sure that detailed diagnosis reports these details also, do the following:

  • Login to the application host.
  • Go to the <JAVA_HOME>\jre\lib\management folder used by the target application, and edit the management.properties file in that folder.
  • Append the following line to the file:

    com.sun.management.enableThreadContentionMonitoring

  • Finally, save the file.