Work Queue Status Test

The job queue contains Connection Broker processes that are independent of the Web interface. When the Connection Broker is not functioning correctly, you can use the job queue as a diagnostics tool. For instance, if you requested an action and it has not taken place, check if the action is pending in the job queue.

This test monitors the job queues and reports the status of jobs queued on the Connection Broker.

Target of the test : A Leostream Connection Broker

Agent deploying the test : An internal agent

Outputs of the test : One set of results for the Connection Broker being monitored.

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	Specify the IP address of the Connection Broker for which the test is to be configured.
Port	Refers to the port used by the Connection Broker is listening.
SSL	By default, this flag is set to No, indicating that by default, the test connects to the Connection Broker via HTTP. If the Connection Broker is SSL-enabled, then set this flag to Yes.
User	To enable the test to log into the Connection Broker and extract statistics of interest from it, specify the login name of a user on the Connection Broker who is assigned the Administrator role. If need be, you can even create a special Administrator user on the Connection Broker for this purpose, and pass the credentials of that user to the test. Refer to Pre-requisites for Monitoring the Leostream Connection Broker to know how to create such a user.
Password	Specify the password of the user here.
Confirm Password	Confirm the password by retyping it here.
Timeout	Specify the duration (in seconds) within which the test should timeout. The default duration is 30 seconds.
ShowFinishedDD	This test also reports the number of jobs in the queue that were recently completed. The detailed diagnosis of this measure, if enabled, will provide the details of all the newly finished jobs. In some environments, the count of the newly finished jobs could run to a thousand. In such environments naturally, the detailed diagnosis data for this measure, if available, will consume a considerable amount of database space, and can sometimes even choke the database. To reduce the strain on the database, the eG agent, by default, does not collect detailed diagnosis information for the New finished jobs measure. Accordingly, the ShowFinishedDD flag is set to No, by default. If you want to view the detailed diagnosis for this measure, then set this flag to Yes.
Detailed Diagnosis	To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled: The eG manager license should allow the detailed diagnosis capability Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Total jobs	Indicates the total number of jobs that are currently in queue.	Number	This is mostly a measure of the complexity of your Connection Broker configuration, in the sense of how many external systems you integrate with. For example, the more centers you have, the more refresh jobs will be in the queue.
Waiting jobs	Indicates the number of jobs that are currently waiting.	Number	Waiting jobs are those jobs that are pending execution. A high value of this measure need not necessarily indicate a bottleneck. You can have a number of pending jobs if the work queue has been suspended in order to process user logins, wherein this becomes a measure of how densely populated your Connection Broker is. Also, some jobs are pending because the Broker has actually scheduled them for a later time. Before a pending job should send up any flag, check that the Scheduled time is not before the Current time. Use the detailed diagnosis of this measure to know which jobs are pending, and their related details.
Inprogress jobs	Indicates the number of jobs that are currently executing.	Number	This measure reports the number of records from the work_queue table where the status is ‘I’. If a lot of jobs are simultaneously in progress, one of the center scans may be taking a long time, for example, large VirtualCenter / Active Directory scans. Alternatively, this could be an indication that the Broker is unhealthy. Check the scheduled time against the start time for the job. If there are a large number of jobs that start after they are scheduled, your Broker is busy. The detailed diagnosis of this measure reveals the details of jobs that are currently in progress.
Finished jobs	Indicates the number of jobs that have completed execution currently.	Number	This measure reports the number of records in the work_queue table with the status ‘F’. Keep in mind that some jobs cannot start before others finish. For example, if a VM that is scheduled to power off then power on, the Power On job will be pending until the power off is complete.
Aborted jobs	Indicates the number of jobs that aborted currently.	Number	This measure refers to those jobs that stopped executing on their own – i.e., it returns the number of records in the work_queue table where the status is ‘A’. Aborted jobs are generally not good. However, you will also get aborted jobs when you update or restart your Connection Broker. Be aware of when those events happen, so you can account for aborted jobs during those times. You can use the detailed diagnosis of this measure to view the list of aborted jobs and investigate the reasons behind the abnormal termination of the jobs.
Cancelled jobs	Indicates the number of jobs that were currently cancelled.	Number	An upcoming job that is terminated by the user is considered a cancelled job. To view the list of the jobs that were cancelled, use the detailed diagnosis of this measure.
Increase in qlength	Indicates the number of jobs that were newly added to the queue in this measurement period.	Number	While the total number of jobs in the work queue is useful, of more importance is whether the number of jobs in the work queue is increasing or decreasing. A consistent increase in length of the work queue may indicate that the Connection Broker is starting to bottleneck. Check the increase in pending and finished jobs. If the pending jobs in the work queue are increasing, this is a clear sign of a bottleneck on the Connection Broker.
New wait jobs	Indicates the number of new pending jobs in the queue in this measurement period.	Number	If there is a sudden/gradual increase in the value of this measure, it could indicate a processing bottleneck on the Connection Broker. Use the detailed diagnosis of this measure to know the new pending jobs and their related details.
New inprogress jobs	Indicates the number of new jobs in execution in this measurement period.	Number	The detailed diagnosis of this measure provides the details of the jobs that are in progress in this measurement period.
New finish jobs	Indicates the number of jobs that finished execution in this measurement period.	Number	The detailed diagnosis of this measure provides the details of the newly finished jobs. Note that the detailed diagnosis for this measure wil be available only if the showfinisheddd flag is set to Yes.
New abort jobs	Indicates the number of the jobs that were newly aborted in this measurement period.	Number	Ideally, the value of this measure should be low. A high value could indicate a problem condition. You can use the detailed diagnosis of this measure to view the list of newly aborted jobs and investigate the reasons behind the abnormal termination of the jobs.
New cancel jobs	Indicates the number of jobs that were newly cancelled.	Number	The detailed diagnosis of this measure provides the details of the newly cancelled jobs.

The detailed diagnosis of the Waiting jobs measure reveals the details of all jobs that are currently pending in the worker queue of the Connection Broker. The details displayed include:

The name of the user who requested the job
The ID of the job following which this job should run
The ID and name of the object on which the job should be performed
The name of the job/action
The parameters to be passed to the action
The time at which the job is to be executed
The result of the job; whether successful or not
Data and time at which the processing of the job was begun by the queue
Data and time at which the processing of the job was completed by the queue
The PID of the work queue that controls the job
The site ID, if the test connects to a remote database
The number of times the job has run

Using this information, administrators can accurately identify the pending jobs, and also determine the users who have initiated the jobs. This information helps administrators in investigating the reasons for the delay in job processing.

waitingjobsDD-final

Figure 1 : The detailed diagnosis of the Waiting jobs measure

The detailed diagnosis of the Inprogress jobs measure reveals the details of all jobs that are currently executing on the Connection Broker.

inprogressjobsDD-final

Figure 2 : The detailed diagnosis of the Inprogress jobs measure

The detailed diagnosis of the Aborted jobs measure reveals the details of all jobs that have stopped executing currently, without being manually terminated by the user. By analyzing this information, administrators can zero-in on the reasons for the abnormal termination of a job, and ensure such process abortions do not recur.

abortedjobsDD-final

Figure 3 : The detailed diagnosis of the Aborted jobs measure

The detailed diagnosis of the New inprogress jobs measure reveals the details of the recent additions to the list of jobs that are in progress.

newinprogressjobsDD-final

Figure 4 : The detailed diagnosis of the New Inprogress jobs measure