Tomcat Executor Test

Connector elements are Tomcat's links to the outside world, allowing Catalina to receive requests, pass them to the correct web application, and send back the results through the Connector as dynamically generated content. In the default configuration, Tomcat will always create a bounded worker-thread pool for each Connector (with max-size 200). Mostly, this is not something that you will need to change. However, Tomcat has a propensity for caching a lot of scaffolding objects (like PageContext and tag buffers) in thread-local context in each worker thread. Because of this, there are instances where you might want Tomcat to be able to close threads down to clean out some memory. Also, having each connector maintaining it’s own pool makes it harder to set a firm top-limit on what load your server will accept. The answer to this is to use a shared Executor.

The Executor represents a thread pool that can be shared between Connectors in Tomcat. By having all connectors share the same executor, you can configure with more predictability how many simultaneous requests that is allowed to run across your entire application. The Executor also brings the ability of having a thread-pool that can shrink as well as grow to accomodate load.

As with thread pools for individual connectors, executors create threads based on request demand, up to the number of maximum threads allotted for the pool. All additional requests are placed in the executor queue to wait for available threads. If this queue is full, the executor will reject new request processing tasks.

To prevent the queue from filling up, administrators should ensure that there are always enough spare threads in the thread pools to service the current and anticipated workload. For this purpose, they need to periodically check if the executor thread pools are configured commensurate to the load, isolate the pools that are poorly configured, and adjust the size/specification of such pools to match the load dynamics. This is where the Tomcat Executor test helps!

This test auto-discovers Executor thread pools, and monitors the load, usage, and configuration of each pool. In the process, the test pinpoints pools that consistently process a large number of tasks, but do not have the processing power to service these tasks effectively causing many tasks to be enqueued in the Executor queue. The test also reports the current configuration of every pool, thus pointing administrators to those pools that are not configured according to their task load. This way, the test prompts administrators to fine-tune the pool configuration, so as to improve load processing.

Target of the test : A Tomcat Server

Agent deploying the test : An internal agent

Outputs of the test : One set of results for every executor thread pool configured on the Tomcat server being monitored.

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	The IP address of the host for which this test is to be configured.
Port	Refers to the port at which the specified host listens to.
Measurement Mode	This test can extract metrics from Tomcat using either of the following mechanisms: By deploying the egtomcat.war file in the <EG_INSTALL_DIR>\lib directory of the eG agent host on the Tomcat server; By contacting the Java runtime (JRE) of Tomcat via JMX To configure the test to use egtomcat.war file, first select the War file option. Then, refer to the How Does eG Enterprise Monitor Tomcat Servers? to know how to deploy the WAR file on the target Tomcat server. On the other hand, if you want the test to use JMX instead, then first, select the JMX option. Then, follow the procedure detailed in the How Does eG Enterprise Monitor Tomcat Servers? to configure the test to use jmx. By default, the JMX option is chosen here.
JMX Remote Port	This parameter appears only if the Measurement Mode is set to JMX. Here, specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in the catalina.sh (or catalina.bat) file in the <CATALINA_HOME_DIR>/bin folder of the target Tomcat server (refer to the How Does eG Enterprise Monitor Tomcat Servers? for more details).
JMX User, JMX Password, and Confirm Password	These parameters appear only if the Measurement Mode is set to JMX. If JMX equires authentication only (but no security), then ensure that the JMX User and JMX Password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to the How Does eG Enterprise Monitor Tomcat Servers?. Confirm the password by retyping it in the Confirm Password text box.
JNDI Name	This parameter appears only if the Measurement Mode is set to JMX. The JNDIName is a lookup name for connecting to the JMX connector. By default, this is jmxrmi. If you have registered the JMX connector in the RMI registery using a different lookup name, then you can change this default value to reflect the same.
Provider	This parameter appears only if the Measurement Mode is set to JMX. This test uses a JMX Provider to access the MBean attributes of the target Java application and collect metrics. Specify the package name of this JMX Provider here. By default, this is set to com.sun.jmx.remote.protocol.
SSL	This parameter appears only if the Measurement Mode is set to War File. Indicate Yes if the Tomcat server is SSL-enabled.
URL	This parameter appears only if the Measurement Mode is set to War File. Specify the URL of the managed Tomcat server to enable the test to connect to it and extract measures from it. The URL specification should be of the format: http://{TomcatIP}:{TomcatPort}.
UserName, Password, and Confirm Password	These parameters appear only if the Measurement Mode is set to War File. In the UserName text box, specify a name of a user who has been assigned the Manager role on the Tomcat server to be monitored; these users are typically allowed to control web applications deployed on the Tomcat server. Specify the Password of this user, and confirm the password by retyping it in the Confirm Password text box.
EncryptPass	This parameter appears only if the measurement modeis set to War File. Select Yes if you want to encrypt the password.
Timeout	Specify the duration (in seconds) for which this test should wait for a response from Tomcat. If there is no response from Tomcat beyond the configured duration, the test will timeout. By default, this is set to 240 seconds.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Active threads	Indicates the number of threads in this pool that are actively used.	Number	If the value of this measure is equal to the value of the Max no of allocated threads measure, it implies that the pool has no spare threads left. Without spare threads, Tomcat will not be able to service subsequent requests. This in turn can result in a serious thread contention, which can adversely impact application performance.
Recently completed task	Indicates the number of tasks that were recently completed by the threads in this pool.	Number
Thread pool core size	Reports the core size of this thread pool.	Number
Peak threads	Indicates the high watermark of threads in this pool.	Number
Max idle time	Indicates the number of milliseconds before a thread can remain idle in this pool before it is shut down, shrinking the thread pool.	Milliseconds	This measure reports the value of the maxIdleTime parameter of the Executor thread pool. Idle threads are a drain on the resources of the server. In times of minimal load, a low maxIdleTime setting will make sure that threads time out quickly, thereby shrinking the thread pool to suit the load. The same idle time setting will not be suitable when load is heavy, as it will risk flapping behaviour where threads are killed too soon. It is therefore imperative that you prudently set the maxIdleTime.
Max queue size	Reports the number of request processing tasks that can be placed in the executor queue to wait for an available worker thread.	Number	This measure reports the value of the maxQueueSize parameter of the Executor thread pool. If this threshold is reached - i.e,, if the number of tasks in the queue is equal to the maxQueueSize setting - then Tomcat will start rejecting processing tasks. To avoid this, its best that you track changes to the Queue size measure over time, periodically compare the value of the Queue size measure with that of this measure, and proactively determine whether the maxQueueSize limit is about to be reached. You may then want to consider increasing the maxQueueSize to avoid request rejection.
Max no of allocated threads	Indicates the maximum number of threads that can be allocated/created in this pool.	Number	This measure reports the value of the maxThreads parameter of the Executor thread pool. Typically, the Executor creates threads in a pool based on request demand, up to the limit specified by the maxThreads parameter. If load is high, its wise to increase the maxThreads setting too, as it will ensure that there are always enough threads for request processing.
Min no of allocated threads	Indicates the minimum number of active and idle threads that should always be available in this pool.	Number	This measure reports the value of the minSpareThreads parameter of the Executor thread pool. This parameter is set to 25 by default to ensure that no pool is shrunk to contain less than 25 threads. You can increase the value of this parameter if your applications require higher service levels.
No of threads in pool	Indicates the number of threads in this pool currently.	Number	If the value of this measure is equal to the value of the Max no of allocated threads measure, it implies that no more threads can be allocated/created in the pool. If a new request comes in subsequently, there may not be any thread to service it. To pre-empt this occurrence, you may want to increase the maxThreads setting for the thread pool.
Queue size	Indicates the number of request processing tasks currently enqueued in the queue of this thread pool.	Number	If the value of this measure is equal to that of the Max queue size measure, then the pool will reject new processing tasks, causing Tomcat to throw a RejectedExecutionException error. To avoid this, once you find that the value of this measure is dangerously close to that of the Max queue size measure, consider increasing the maxQueueSize setting of the thread pool. If this is done, then the queue will be able to accommodate and process more processing requests, thereby ensuring that no requests are rejected.
Percent of tasks in the queue	Of the maximum number of tasks that can be in queue, what percentage of tasks is currently in queue.	Percent	A value close to 100% indicates that the queue is rapidly filling up. This is a good time to increase the maxQueueSize of the thread pool and avoid rejection of requests.
Pool utilization	Indicates the percentage of pool size that is currently utilized.	Percent	A value close to 100% indicates that almost all the threads in the pool are actively engaged in processing request processing tasks. In such a situation, you may want to check the maxThreads and minSpareThreads setting of the pool to determine whether the pool has enough threads to process subsequent requests. If not, you may want to increase these settings.