RabbitMQ Virtual Host Test

A Virtual host provides a way to segregate applications using the same RabbitMQ instance. Different users can have different access privileges to different vhosts, and queues and exchanges can be created so they only exist in one vhost.

In fact, RabbitMQ employs a more tangible realization of virtual hosts by effectively making them "virtual clusters" on top of the broker. This means that much like RabbitMQ clusters, the workload of a virtual host is defined by the messages it receives from publishers and delivers to consumers and the message read-write activity on its disk. Understanding the current workload of a virtual host and measuring how well the virtual host handles the workload is important to figure out if the virtual host has been sized and configured right to perform well at peak load. The RabbitMQ Virtual Host test enables administrators to gain this understanding.

This test tracks the messages published to and delivered by each of the virtual hosts the configured user (i.e., the user configured for this test) has access to. Furthermore, the test measures the rate at which each virtual host processes the messages, and thus reveals bottlenecks (if any) in message processing. Additionally, the test also reports the level of I/O activity on the cluster disks, and reveals the count of unacknowledged and redelivered messages. This will enable administrators to assess the impact of IOPS and messages on the performance of the virtual host. This way, the test helps administrators gauge the current workload on each virtual host, understand the present processing power of every virtual host, and identify those virtual hosts that could be experiencing processing bottlenecks. Pointers provided by the test can also be used to figure out how the virtual host configuration can be fine-tuned to ensure optimal performance.

Target of the test : A RabbitMQ Cluster

Agent deploying the test : A remote agent

Outputs of the test : One set of results for the each virtual host to which the configured user has access

Configurable parameters for the test
Parameters	Description
Test period	How often should the test be executed
Host	The host for which the test is to be configured.
Port	The port at which the configured Host listens; by default, this is 15672
Username, Password, and Confirm Password	The eG agent connects to the Management Interface of the rabbitmq-management plugin of the target node, and runs HTTP-based API commands on the node using the plugin to pull metrics of interest. To connect to the plugin and run the API commands, the eG agent requires the privileges of a user on the cluster who has been assigned the 'monitoring' tag. If such a user pre-exists, then configure this test with the Username and Password of that user. On the other hand, if no such user exists, then you will have to create a user for this purpose using the Management Interface. The steps for this have been detailed in How Does eG Enterprise Monitor a RabbitMQ Cluster? In this case, make sure you configure this test with the Username and Password of the new user. Finally, confirm the password by retyping it in the Confirm Password text box.
SSL	By default, this flag is set to No, as the target node is not SSL-enabled by default. If the node is SSL-enabled, then set this flag to Yes.

Measurements made by the test
Measurement	Description	Measurement Unit	Interpretation
Ready messages	Indicates the number of messages on this virtual host that are available to be delivered now.	Number
Unacknowledged messages	Indicates the number of messages for which this virtual host is waiting for acknowledgement.	Number	A low value is desired for this measure. This is because, all unacknowledged messages have to reside in RAM on the servers. If you have too many unacknowledged messages you will run out of memory. An efficient way to limit unacknowledged messages is to limit how many messages your clients prefetch.
Current messages	Indicates the total number of messages on this virtual host currently.	Number	This is the sum total of the value of the Ready messages and Unacknowledged messages. The value of this measure is a good indicator of the current message load on the virtual host.
Published message rate	Indicates the rate at which publishers are publishing messages on this virtual host.	Messages/Sec
Publisher confirmation rate	Indicates the rate at which this virtual host confirms the receipt of a message to a publisher.	Messages/Sec	A 'Publish Confirm' is nothing but a acknowledgement sent to a publisher confirming the receipt of a message from that publisher. Publish Confirms have a performance impact. This means, the lower the value of this measure, the better. However, one should keep in mind that a Publish Confirm is required if the publisher needs at-least-once processing of messages.
Manually acknowledged message delivery rate	Indicates the rate at which messages on this virtual host are delivered to consumers that use manual acknowledgements.	Messages/Sec	Messages in transit might get lost in an event of a connection failure, and such a message might need to be retransmitted. Acknowledgements let the server and clients know when to retransmit messages. A manual acknowledgement is an 'explicit' acknowledgement that is received from the consumer. Manually sent acknowledgements can be positive or negative. Positive acknowledgements simply instruct RabbitMQ to record a message as delivered and can be discarded. Negative acknowledgements with basic.reject have the same effect. The difference is primarily in the semantics: positive acknowledgements assume a message was successfully processed while their negative counterpart suggests that a delivery wasn't processed but still should be deleted. Whether positive or negative, manual acknowledgements deliver low throughput and hence, should be avoided. A low value is therefore desired for this measure.
Auto-acknowledged message delivery rate	Indicates the rate at which messages on this virtual host are delivered to consumers that use automatic acknowledgements.	Messages/Sec
Consumer acknowledgement rate	Indicates the rate at which messages on this virtual host are being acknowledged by consumers.	Messages/Sec	If the Delivery rate of messages requiring acknowledgement measure registers an abnormally low value, then, you may want to check the value of this measure at around the same time to determine whether the a delay by consumers in acknowledging the messages was what caused the delivery delay.
Message redelivery rate	Indicates the rate at which messages with the 'redelivered' flag set are being delivered by this virtual host.	Messages/Sec
Delivery rate of messages requiring acknowledgement	Indicates the rate at which messages requiring acknowledgement are being delivered by this virtual host in response to basic.get.	Messages/Sec	Compare the value of this measure with that of the Delivery rate of messages not requiring acknowledgement measure to figure out what type of messages are being delivered much slower than the rest.
Delivery rate of messages not requiring acknowledgement	Indicates the rate at which messages not requiring acknowledgement are being delivered by this virtual host in response to basic.get.	Messages/Sec	Compare the value of this measure with that of the Delivery rate of messages requiring acknowledgement measure to figure out what type of messages are being delivered much slower than the rest.
Message return rate	Indicates the rate at which this virtual host sent unrouteable messages with 'mandatory' flag set to 'true', to publishers.	Messages/Sec	An unroutable message is a message without a destination. For example, a message sent to an exchange without any bound queue. If the 'mandatory' flag is set to 'true', then an unroutable message is returned to the producer with a `basic.return` AMQP method.
Disk read rate	Indicates the rate at which queues in this virtual host read messages from disk.	Messages/Sec	A high value could indicate that messages are frequently read from the disk and not from the RAM. This could be owing to high memory pressure, which may have forced RabbitMQ to move messages from RAM to disk.
Disk write rate	Indicates the rate at which queues in this virtual host wrote messages to disk.	Messages/Sec	A high value for this measure could indicate any of the following: Many messages have been published in such a way that they must be written to disk; A very high memory pressure on RabbitMQ has caused the cluster to move majority of messages from RAM to disk;
Published messages	Indicates the total number of messages entering this virtual host.	Number
Publisher confirmed messages	Indicates the total number of messages that this virtual host is confirming to publish.	Number	A 'Publish Confirm' is nothing but a acknowledgement sent by the cluster to a publisher confirming the receipt of a message from that publisher. Publish Confirms have a performance impact. This means, the lower the value of this measure, the better. However, one should keep in mind that a Publish Confirm is required if the publisher needs at-least-once processing of messages.
Messages delivered with manual acknowledgement	Indicates the total number of messages are delivered to consumers that use manual acknowledgements.	Number	Messages in transit might get lost in an event of a connection failure, and such a message might need to be retransmitted. Acknowledgements let the server and clients know when to retransmit messages. A manual acknowledgement is an 'explicit' acknowledgement that is received from the consumer. Manually sent acknowledgements can be positive or negative. Positive acknowledgements simply instruct RabbitMQ to record a message as delivered and can be discarded. Negative acknowledgements with basic.reject have the same effect. The difference is primarily in the semantics: positive acknowledgements assume a message was successfully processed while their negative counterpart suggests that a delivery wasn't processed but still should be deleted. Whether positive or negative, manual acknowledgements deliver low throughput and hence, should be avoided. A low value is therefore desired for this measure.
Messages delivered with auto-acknowledgement	Indicates the total number of messages that are delivered to consumers that use automatic acknowledgements.	Number	In automatic acknowledgement mode, a message is considered to be successfully delivered immediately after it is sent. This mode trades off higher throughput (as long as the consumers can keep up) for reduced safety of delivery and consumer processing. This mode is often referred to as "fire-and-forget". Unlike with manual acknowledgement model, if consumers's TCP connection or channel is closed before successful delivery, the message sent by the server will be lost. Therefore, automatic message acknowledgement should be considered unsafe and not suitable for all workloads. The value 0 is hence ideal for this measure.
Messages delivered with consumer acknowledgement	Indicates the total number of messages that are being acknowledged by consumers.	Number
Redelivered messages	Indicates the total number of messages that are being delivered with the 'redelivered' flag set.	Number	If a message is delivered to a consumer and then requeued (because it was not acknowledged before the consumer connection dropped, for example) then RabbitMQ will set the 'redelivered' flag on it when it is delivered again (whether to the same consumer or a different one). This is a hint that a consumer may have seen this message before (although that's not guaranteed, the message may have made it out of the broker but not into a consumer before the connection dropped). Conversely if the redelivered flag is not set then it is guaranteed that the message has not been seen before.
Messages requiring acknowledgements	Indicates the total number of messages requiring acknowledgement that are being delivered in response to basic.get.	Number	For best performance and high throughput, the value of this measure should be low.
Messages not requiring acknowledgements	Indicates the total number of messages not requiring acknowledgement that are being delivered in response to basic.get.	Number
Messages returned	Indicates the rate at which unrouteable messages with 'mandatory' flag set to 'true', were sent to publishers.	Messages/Sec	An unroutable message is a message without a destination. For example, a message sent to an exchange without any bound queue. If the 'mandatory' flag is set to 'true', then the cluster return an unroutable message to the producer with a `basic.return` AMQP method. To know which nodes in the cluster returned the maximum number of messages to publishers, use the detailed diagnosis of this measure.
Disk reads	Indicates the total number of messages read from disk.	Number	A high value could indicate that many messages are read from the disk and not from the RAM. This could be owing to high memory pressure, which may have forced RabbitMQ to move messages from RAM to disk. If this measure reports an abnormally high value, then use the detailed diagnosis of this measure to know which nodes in the cluster are performing the maximum reads from the disk. Such nodes could be running out of memory.
Disk writes	Indicates the total number of messages written to disk.	Number	A high value for this measure could indicate any of the following: Many messages have been published in such a way that they must be written to disk; A very high memory pressure on RabbitMQ has caused the cluster to move majority of messages from RAM to disk; If this measure reports an abnormally high value, then use the detailed diagnosis of this measure to know which nodes in the cluster are performing the maximum reads from the disk. Such nodes could be running out of memory.