Apache Kafka Broker layer

Using the tests associated with the Apache Kafka Broker layer (see Figure 1), you can monitor the following:

  • Rate at which data sent from producers is consumed by the broker.

  • Rate at which individual messages are consumed by the broker.

  • Fetch request rate for requests that have failed.

  • Rate at which data is sent from one broker to other brokers.

  • Rate at which data sent from other brokers is consumed by the follower broker.

  • Outgoing/Incoming byte rate of reassignment traffic.

  • Number of brokers designated as controllers.

  • Number of partitions without an active leader and to check whether/not they are writable or readable.

  • Imbalance count in the preferred replica.

  • Number of coordinator groups currently in stable/unstable state.

  • Number of consumer groups that are currently in preparing/completing re-balance state.

  • Number of offline log directories on a live broker.

  • Size of the request/response queues.

  • Total number of disconnections across all processors.

  • Number of replicas for which this broker is the leader.

  • Number of offline replicas on a live broker.

  • Number of reassigning leader partitions on a broker.

  • Number of partitions that have not been fully replicated in the follower replicas.

  • Number of partitions under the minimum In-Sync Replica (ISR) count.

  • Size of Fetch, Produce and Request queues.

  • Status of the broker’s connection to Zookeeper.

  • Number of zookeeper clients that are connected to the ensemble and ready to execute operations.

  • Number of attempts to connect the ensemble but failed due to incorrect credentials.

Figure 1 : The list of tests associated with the Apache Kafka Broker layer