RabbitMQ Nodes Test

The values that this measure can report and their corresponding numeric values are listed in the table below:

Note:

This test reports the Measure Values listed in the table above to indicate the current operational state of a node. In the graph of this measure however, the same will be represented using the numeric equivalents.

Compare the value of this measure across nodes to identify the node that has been down for the longest time.

By default, a node can use up to a maximum of 1024 file descriptors.

A file descriptor (FD, less frequently fildes) is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket.

This count includes both file and socket descriptors.

A file descriptor (FD, less frequently fildes) is an abstract indicator (handle) used to access a file or other input/output resource, such as a pipe or network socket.

A socket is an abstraction of a communication endpoint. Just as they would use file descriptors to access a file, applications use socket descriptors to access sockets. Socket descriptors are implemented as file descriptors in the UNIX System.

A value close to 100% is a cause for concern, as it indicates that the node is running out of file handles. If the value reaches 100%, then the node will block all incoming connections. To avoid this, you may want to consider increasing the maximum file descriptor configuration of the node.

A socket is an abstraction of a communication endpoint. Just as they would use file descriptors to access a file, applications use socket descriptors to access sockets.

A value close to 100% is a cause for concern, as it indicates that the node is running out of socket descriptors. If a node exhausts socket descriptors, then that node will block all incoming connections. To avoid this, you may want to consider increasing the maximum socket descriptor configuration of the node.

Queues, connections, and channels are the main components of RabbitMQ that consume processes. This means, higher the number of queues, connections, and channels, higher will be the usage of Erlang processes.

If the value of this measure is close to 100% for a node, it implies that that node is running out of Erlang processes. This could cause databases to hang and messages to start piling up on RabbitMQ. To avoid this, you may want to consider increasing the maximum number of Erlang processes configured for the node.

A value close to 100% indicates that the node is running out of memory. If the situation is allowed to persist, then the node may soon exhaust its memory completely. This could bring messaging operations to a standstill. 

At this juncture, you can start by looking at some of the common memory consumers on a node are as follows:

• Connections
• Channels
• Queue masters, indices, and messages kept in memory
• Queue mirrors, indices, and messages kept in memory
• Binaries containing message bodies and metadata
• Plugins
• Mnesia tables and other ETS tables that keep an in-memory copy of their data
• Memory used by code and atoms

If any of the afore-mentioned factors are large in number, then memory consumption too is likely to increase. In such a situation, to conserve memory, see if you can control the memory consumption of the aforesaid components by decreasing their count. For instance, see if you can optimize the number of channels your applications typically use and bring that number down.

Alternatively, you may want to consider increasing the memory allocation to the node.

An Mnesia transaction is a mechanism by which a series of database operations can be executed as one functional block.

If the transaction is performed on data stored exclusively in memory, it is a RAM-only Mnesia transaction. An example of such a transaction is creation/deletion of transient queues.

If the transaction is performed on data stored in disk, it a disk transaction. An example of such a transaction is creation/deletion of durable queues.

Each record in a queue index journal represents a message being published to a queue, being delivered from a queue, and being acknowledged in a queue.

If the value of this measure keeps increasing consistently, it could indicate that one/more queues reside on the target node and that there is a high level of messaging activity on that node.

Messages (the body, and any properties and / or headers) can either be stored directly in the queue index, or written to the message store.

The message store is a key-value store for messages, shared among all queues in the server. There are technically two message stores (one for transient and one for persistent messages) but they are usually considered together as "the message store".

If the value of the Store writes measure increases consistently, it could indicate the presence of many lazy queues on the node with many messages. To accommodate all these messages in the message store, the node will have to be sized with sufficient disk space and file descriptors.

The queue index is responsible for maintaining knowledge about where a given message is in a queue, along with whether it has been delivered and acknowledged. There is therefore one queue index per queue.

If the values reported by these measures are consistently high for a node, it could mean that one/more queues reside on the node and many messages are stored in the queues.

Compare the values of these measures across nodes to identify the node that is experiencing a high level of disk activity. In the process, you can identify which type of I/O operations are very common on that node - reads? writes? seeks? or syncs?

These measures are good indicators of the bandwidth used by a node when reading/writing. You can compare the value of this measure across nodes to know which node is consuming maximum bandwidth.

Compare the values of these measures across nodes to identify the most latent node. In the process, you can identify which type of I/O operations are taking the longest on each node - reads? writes? seeks? or syncs?