Azure Service Bus Queue Test

Azure Service Bus supports a set of cloud-based, message-oriented middleware technologies including reliable message queuing and durable publish/subscribe messaging. The messaging entity that forms the core of the messaging capabilities in Service Bus is the Queue.

Messages are sent to and received from queues. Queues store messages until the receiving application is available to receive and process them. With queues, the producers (senders) and consumers (receivers) don't have to send and receive messages at the same time. That's because messages are stored durably in the queue. Furthermore, the producer doesn't have to wait for a reply from the consumer to continue to process and send messages. Queues also enable producers and consumers to send and receive messages at different rates.

However, if a queue does not have enough processing power in the form of memory, then it may not be able to process the messages quickly and send them to the consumers. Similarly, if adequate worker processes are not available for a queue to service its load, then processing latencies are a given. The result? - the message consuming application may experience delays and performance deficiencies. Leave alone delays! If queues consistently and completely fail to process messages, then enterprises may be forced to look for other middleware technologies, the aforesaid benefits of Azure Service Bus queues notwithstanding! Such an outcome becomes inevitable if queues fail to process transactions that combine a set of business-critical operations!.

To avoid this, administrators should track the workload of each queue, periodically check if every queue has the resources (memory and worker processes) it requires to process the load, identify queues that are poorly-sized, and increase their processing power. Additionally, administrators should also be able to promptly capture queues that are unable to process messages and take appropriate action, so as to ensure the uninterrupted flow of messages. This is where the Azure Service Bus Queue test helps!

This test auto-discovers the Azure Service Bus queues that have been created for each Service Bus of the monitored subscription. For each queue so discovered, the test then reports the current status of the queue, and alerts administrators to disabled queues. The test also reports the memory used per queue, thus pointing you to queues that are rapidly running out of memory. Additionally, the test tracks the workload of each queue in terms of the count of messages in that queue and the number of messages actively processed by it. In the process, the test pinpoints queues that may not have require more processing power to process its workload. Furthermore, the test also leads you to queues that are unable to process one/more messages/transactions. Using detailed diagnostics, you can deduce the reason for such failures - did an improper configuration setting (eg., a wrong expiration setting) on the queue force the failure? or did the application explicity reject messages and cause the failure? With the help of these pointers, you can eliminate the underlying cause of the failure, and ensure that business-critical operations that may have halted because of the failure resume quickly.

Target of the Test: A Microsoft Azure Subscription

Agent deploying the test: A remote agent

Output of the test: One set of results for each Service Bus queue created in every Service bus configured for every resource group in the target Azure Subscription

Configurable parameters for the test
Parameters Description

Test Period

How often should the test be executed.

Host

The host for which the test is to be configured.

Subscription ID

Specify the GUID which uniquely identifies the Microsoft Azure Subscription to be monitored. To know the ID that maps to the target subscription, do the following:

  1. Login to the Microsoft Azure Portal.

  2. When the portal opens, click on the Subscriptions option (as indicated by Figure 1).

    Clicking Subscriptions Option

    Figure 1 : Clicking on the Subscriptions option

  3. Figure 2 that appears next will list all the subscriptions that have been configured for the target Azure AD tenant. Locate the subscription that is being monitored in the list, and check the value displayed for that subscription in the Subscription ID column.

    Determining Subscription ID

    Figure 2 : Determining the Subscription ID

  4. Copy the Subscription ID in Figure 2 to the text box corresponding to the SUBSCRIPTION ID parameter in the test configuration page.

Tenant ID

Specify the Directory ID of the Azure AD tenant to which the target subscription belongs. To know how to determine the Directory ID, refer to Configuring the eG Agent to Monitor a Microsoft Azure Subscription Using Azure ARM REST API.

Client ID, Client Password, and Confirm Password

To connect to the target subscription, the eG agent requires an Access token in the form of an Application ID and the client secret value. For this purpose, you should register a new application with the Azure AD tenant. To know how to create such an application and determine its Application ID and client secret, refer to Configuring the eG Agent to Monitor a Microsoft Azure Subscription Using Azure ARM REST API. Specify the Application ID of the created Application in the Client ID text box and the client secret value in the Client Password text box. Confirm the Client Password by retyping it in the Confirm Password text box.

Proxy Host and Proxy Port

In some environments, all communication with the Azure cloud be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none, indicating that the eG agent is not configured to communicate via a proxy, by default.

Proxy Username, Proxy Password and Confirm Password

If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy Username and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box.

Detailed Diagnosis

To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability
  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measures made by the test:
Measurement Description Measurement Unit Interpretation

Queue entity status

Indicates the current status of this queue.

 

The values reported by this measure and its numeric equivalents are mentioned in the table below:

Measure Value Numeric Value
Active 1
Creating 2
Deleting 3
Disabled 4
Receive Disabled 5
Renaming 6
Restoring 7
Send Disabled 8
Unknown 9

Note:

By default, this measure reports the Measure Values listed in the table above to indicate the current status of the the service bus queue. In the graph of this measure however, the same is represented using the numeric equivalents only.

Use the detailed diagnosis of this measure to know the location of the service bus queue, when it was created, when it was last modified, and configuration settings governing the 'life' of messages in the queue.

Total memory

Indicates the total memory capacity of this service bus queue.

MB

 

Used memory

Indicates the amount of memory currently used by this service bus queue.

MB

Ideally, the value of this measure should be low.

Free memory

Indicates the amount of memory that is still available for use by this queue.

MB

Ideally, the value of this measure should be high.

Memory utilization

Indicates the percentage of memory used by this queue.

Percent

A value close to 100% indicates that the service bus queue is running out of free memory. Without sufficient free memory, the queue will not be able to store/process messages. To avoid this, you may want to increase the memory capacity of the queue or declutter the queue by deleting unwanted messages from it.

Total messages

Indicates the total number of messages in this queue.

Number

 

Active messages

Indicates the number of messages in this queue that are being actively processed.

Number

This is a good indicator of the current workload of a queue.

 

Scheduled messages

Indicates the number of messages in this queue that are scheduled to be delivered at a later time.

Number

In messaging, you can send a message to a queue/topic for delayed processing. Those messages are called scheduled messages.

Unlike regular messages, scheduled messages don’t appear in the queue until the defined enqueue time. For example, you can schedule a message at 12 pm and send it to a queue at 10 pm. The message will be invisible for consumers until 12 pm. It will be appended to the queue as a new message at 12 pm.

Dead-letter messages

Indicates the number of messages in this queue that were moved to the dead-letter queue during the last measurement period.

Number

Azure Service Bus queues and topic subscriptions provide a secondary subqueue, called a dead-letter queue (DLQ). The purpose of the dead-letter queue is to hold messages that cannot be delivered to any receiver, or messages that could not be processed.

Ideally therefore, the value of this measure should be 0. A non-zero value implies that one/more messages have been moved to the DLQ. In this case, its important to know why messages could not be delivered/processed. In fact, there are several activities in Service Bus that cause messages to get pushed to the DLQ from within the messaging engine itself. The common reasons are as follows:

  • HeaderSizeExceeded: A message is moved to the DLQ if the size quota for that message stram has exceeded.

  • TTLExpiredException: If a message expires, it is automatically dead-lettered. The expiration for any individual message can be controlled by setting the time-to-live system property. If a message remains in the queue/topic for a duration beyond the configured time-to-live, you can configure that message to be dead-lettered.

    All messages sent into a queue or topic are subject to a default expiration that is set at the entity level. The default expiration is used for all messages sent to the entity where time-to-live is not explicitly set.

    Sometimes, if extremely low values are set for time-to-live, messages may expire and move to the DLQ before receiver applications receive it. To know if such a value has been set at the queue-level, go to the detailed diagnosis of the Queue entity status measure of this test, and look up the value in the Message time to live column. If the value of this measure is high, then the Message time to live configuration will tell you if a low TTL is the reason for the dead-lettering.

    If a message is protected from expiration while under lock, that message is moved to the dead-letter queue once the lock expires or is abandoned. If the value of this measure is abnormally high, then go to the detailed diagnosis of the Queue entity status measure and look up the time displayed in the Lock duration column. If a low value is displayed in this column, then you can attribute the frequent dead-lettering to the quick expiry of the lock.

  • Session ID is null: If a session enabled entity disallows a message whose session identifier is null, then that message is dead-lettered.

  • MaxTransferHopCountExceeded: If a message being forwarded between queues exceeds the maximum number of hops that are allowed, then that message is dead-lettered.

  • MaxDeliveryCountExceeded: There is a limit on number of attempts to deliver messages for Service Bus queues and subscriptions. The default value is 10. Whenever a message has been delivered under a peek-lock, but has been either explicitly abandoned or the lock has expired, the delivery count on the message is incremented. When the delivery count exceeds the limit, the message is moved to the DLQ. To determine whether a low 'Maximum Delivery Count' setting is the reason why many messages were dead-lettered, go to the detailed diagnosis of the Queue entity status measure and look up the value displayed in the Maximum Delivery Count column. If this value is indeed low, then increasing it can reduce the count of dead-lettered messages.

An application can also explicitly move messages to the DLQ for reasons other than the ones stated above. Messages can be so moved owing to system issues, it they hold malformed payload, or if they fail authentication when some message-level security scheme is used.

Transfer messages

This measure will be reported only if the queue being monitored is a transfer queue. In which case, this measure indicates the number of messages in this transfer queue.

Number

A transaction groups two or more operations together into an execution scope. By nature, such a transaction must ensure that all operations belonging to a given group of operations either succeed or fail jointly. Service Bus supports grouping operations against a single messaging entity (queue, topic, subscription) within the scope of a transaction. For example, you can send several messages to one queue from within a transaction scope, and the messages will only be committed to the queue's log when the transaction successfully completes.

To enable transactional handover of data from a queue or topic to a processor, and then to another queue or topic, Service Bus supports transfers. In a transfer operation, a sender first sends a message to a transfer queue or topic, and the transfer queue or topic immediately moves the message to the intended destination queue or topic using the same robust transfer implementation that the autoforward capability relies on.

Trransfer dead-letter messages

This measure will be reported only if the queue being monitored is a transfer queue. In which case, this measure indicates the number of messages in this transfer queue that were sent to the transfer dead-letter queue.

Number

Ideally, the value of this measure should be 0. If a non-zero value is reported, it means the one/more messages in the transfer queue have been dead-lettered.

Messages will be sent to the transfer dead-letter queue under the following conditions:

  • A message passes through more than four queues or topics that are chained together.

  • The destination queue or topic is disabled or deleted.

  • The destination queue or topic exceeds the maximum entity size.

Use the detailed diagnosis of the Queue entity status measure to know the location of the service bus queue, when it was created, when it was last modified, and the configuration settings governing the 'life' of messages in the queue. These settings will help you troubleshoot abnormal dead-lettering on the queue.

Queue Entity Status Measure Detailed Diagnosis

Figure 3 : The detailed diagnosis of the Queue entity status measure