What is Apache Kafka Message System?

Apache Kafka is a publish-subscribe based durable messaging system. A messaging system sends messages between processes, applications, and servers.

Apache Kafka is a software where topics can be defined (think of a topic as a category), applications can add, process and reprocess records.

Applications connect to this system and transfer a record onto the topic. A record can include any kind of information; for example, information about an event that has happened on a website, or an event that is supposed to trigger an event. Another application may connect to the system and process or re-process records from a topic. The data sent is stored until a specified retention period has passed by.

Records are byte arrays that can store any object in any format. A record has four attributes, key and value are mandatory, and the other attributes, timestamp, and headers are optional. The value can be whatever needs to be sent, for example, JSON or plain text.

These are four main parts in a Kafka system:

Broker: Handles all requests from clients (produce, consume, and metadata) and keeps data replicated within the cluster. There can be one or more brokers in a cluster.

Zookeeper: Keeps the state of the cluster (brokers, topics, users).

Producer: Sends records to a broker.

Consumer: Consumes batches of records from the broker.

Why Monitor Apache Kafka Message System?

The requests from producers/consumers to the brokers and the responses from the broker to the producer/consumers should be monitored to ensure the Apache Kafka Message System is communicating efficiently. Kafka's request rate could rise as producers send more traffic.By closely monitoring the target Apache Kafka Message System, administrators can be proactively alerted to issues in the communication between producers, consumers and brokers that could affect the overall performance of the target system.