Message Queue

What is a Message Queue?

A message queue is a form of communication used in distributed systems and software architectures where different components or processes need to communicate asynchronously. It is a mechanism that allows messages (packets of data) to be sent and received between different parts of a system without the components being directly connected or synchronized in time.

The basic idea is that a sender (producer) can place messages in a queue, and a receiver (consumer) can retrieve and process those messages from the queue. This decouples the sender and receiver, enabling them to operate independently and at different rates.

Some key concepts associated with message queues include:

  • Producer: The component or process that sends messages to the queue.
  • Consumer: The component or process that retrieves and processes messages from the queue.
  • Message: The data packet containing information to be communicated between components. It could be a command, an event, or any other form of data.
  • Queue: A data structure that holds the messages in the order they were sent. Messages are usually processed in a first-in, first-out (FIFO) manner.
  • Asynchronous Communication: Message queues facilitate asynchronous communication, meaning that the sender and receiver do not need to be actively communicating at the same time. The sender sends a message, and the receiver processes it whenever it is ready.

When would you use a Message Queue?

Message queues are used in various scenarios, including:

  • Load Balancing: Distributing work among multiple processing units.
  • Event-Driven Architectures: Handling events and notifications between different components.
  • Decoupling Components: Allowing different parts of a system to evolve independently.
  • Fault Tolerance: Providing a buffer for temporary disruptions in communication.

What about Linux Message Queues?

In Linux, the kernel itself provides a set of message queue facilities that are commonly used for inter-process communication. These facilities are part of the System V IPC (Inter-Process Communication) mechanisms, and among them is the Message Queue.

It's important to note that System V IPC, including Message Queues, has been part of Unix-like operating systems for a long time, but it's not the only inter-process communication mechanism available on Linux.

Many newer Linux systems often provide an alternative IPC mechanism called POSIX Message Queue. POSIX Message Queues are generally considered more modern and flexible than the older System V Message Queues, but both are available on many Linux systems.

What are the most popular Message Queue systems?

Popular message queue systems include Apache Kafka, RabbitMQ, ActiveMQ, and Microsoft Message Queue (MSMQ). Many of these systems offer additional features such as durability, scalability, and support for different messaging patterns, making them suitable for a wide range of distributed systems.

What is the role of Message Queues in microservices architecture?

In microservices, message queues foster independence and enable asynchronous communication. They act as buffers during service failures, ensuring message delivery and enhancing system resilience. Message queues support event-driven architectures, maintaining message order and facilitating microservices orchestration. They contribute to scalability and responsiveness, allowing microservices to operate independently and scale based on workload. Additionally, message queues aid in load balancing by evenly distributing messages across multiple instances, optimizing resource utilization. Used in event sourcing, they store events for system history, supporting dynamic service discovery. With transactional features, message queues ensure microservices-based transactions' integrity, forming a critical component for resilient microservices.

As message queue systems are so well-suited to microservice architectures it is very common to see them used in conjunction with containerization and container orchestration technologies, such as Red Hat OpenShift, Kubernetes, VMware Tanzu and Docker amongst others.

What is the difference between a Message Queue and a Message Broker?

A message queue is a fundamental component that enables communication between different components or systems by storing and managing messages in a sequential manner. It provides a basic mechanism for asynchronous communication, allowing one component to send a message to a queue, and another component to retrieve and process the message. The Linux message queues of POSIX and System V IPC are truer to this conventional definition than the message queue systems commonly seen in modern microservice architectures.

On the other hand, a message broker is a more advanced entity that goes beyond basic queuing functionality. It acts as an intermediary in message-based communication between different components or systems. In addition to queuing, a message broker provides additional features such as message routing, translation, and support for various messaging patterns like publish-subscribe and point-to-point.

In essence, while a message queue is primarily focused on the storage and retrieval of messages in a sequential manner, a message broker extends this functionality by providing additional features for managing and facilitating communication between different components or systems in a distributed environment.

Indeed, many of the popular message queue systems define themselves as brokers. RabbitMQ describes itself as “RabbitMQ is a messaging broker - an intermediary for messaging. It gives your applications a common platform to send and receive messages, and your messages a safe place to live until received”. In practice common terminology often uses “Message Queue” to refer to systems that are a “Message Broker”.

Many “Message Queue Systems” go further and describe themselves and their broad technologies in terms of services and platform, for example Apache Kafka describes itself as “an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications”.

What features should a Message Queue monitoring tool have?

To proactively monitor message queue health and performance data, tools such as eG Enterprise have key capabilities such as:

  • The ability to monitor message queues in real-time, providing visibility into message processing times, queue sizes, and message throughput. This can help identify bottlenecks and potential performance issues in your message queue infrastructure
  • eG Enterprise propagates the context through message headers to enable auto-tracing of client-side and server-side calls to messaging services
  • End-to-end tracing coupled with technology-specific message queue performance metrics, events and logs, offer full insight into anomalies and issues impacting mission-critical apps
  • Trigger alerts on message processing backlogs or message buildup
  • Monitor dead-letter queues
  • Monitor unbalanced message processing that could result in queue overflows
  • Support for all the common messaging protocols – JMS, AMQP, MQTT, STOMP
  • Unified topology view to visualize connected producers and consumers

Learn more about eG Enterprise’s support for messaging queue monitoring, here: Message queue monitoring | eG Innovations.

What are the key metrics a Message Queue monitoring tool should monitor?

The metrics to monitor depend on the specific characteristics and requirements of your message queue system. You should always pick a tool that explicitly supports your message queue technology of choice. However, here is a general list of metrics that a comprehensive monitoring tool for message queues would usually capture and / or alert on:

  • Message Rates: Monitor the rates at which messages are produced and consumed. Track the throughput to understand the volume of messages flowing through the system.
  • Queue Depths / Lengths: Keep an eye on the depth of each queue to understand how many messages are awaiting processing. This metric helps in identifying potential bottlenecks.
  • Consumer Lag: For systems with multiple consumers, monitor the lag between message production and consumption. Lagging consumers may indicate processing delays.
  • Error Rates: Track the rate of message processing errors. High error rates may indicate issues with message content or processing logic.
  • Connection Counts: Monitor the number of active connections to the message broker. Sudden spikes may indicate increased activity or potential issues.
  • Resource Utilization: Keep an eye on system-level metrics such as CPU, memory, and disk usage. Unusual patterns may indicate resource constraints.
  • Latency and Message Processing Times: Measure the time it takes for messages to traverse the system. High latency may impact the responsiveness of your application.
  • Dead Letter Queue Activity: If your message broker supports dead letter queues, monitor activity to identify messages that failed processing and were moved to the dead letter queue.
  • Connection Durations: Track the duration of connections to identify long-lived connections that may impact system resources.
  • Java and JVM metrics: Many systems such as Apache Kafka are implemented in Java and run on the Java Virtual Machine (JVM). Kafka's architecture is built around the JVM, with Kafka brokers being Java processes that manage message storage, replication, and data distribution. The JVM is responsible for memory management, garbage collection, and efficient execution of Kafka broker processes. Monitoring Garbage Collection and Heap Memory Usage, JVM performance and so on is essential for Java based systems.
  • Network Metrics: Monitor network metrics such as bandwidth usage and latency, especially if your message queue system spans multiple nodes, clouds or data centers.
  • Replication Metrics (for Replicated Systems): If your message broker uses replication, monitor replication lag and the synchronization status between replicas.
  • Broker-Specific Metrics: Depending on the message broker in use (e.g., RabbitMQ, Apache Kafka), monitor broker-specific metrics provided by the broker's management interface. This is an example of why it is necessary to use a tool with an explicit integration with the message queue vendor’s APIs and similar interfaces.