Monitoring AWS Managed Service Kafka

eG Enterprise offers a special-purpose monitoring model for the AWS Managed Service Kafka to monitor the status and overall performance of the target AWS Managed Service Kafka.

Figure 1 depicts the layer model of an AWS Managed Service Kafka.

Figure 1 : Layer model for Apache Impala

Every layer in the Figure 1 is mapped to various tests to determine the critical statistics related to the performance of the target AWS Managed Service Kafka. Using the metrics reported by the tests, administrators can find accurate answers for the following performance queries:

  • Does the target AWS MSK server uses all the physical, buffer and cached memory allocated to it?

  • Are the requests from producers, consumers and broker followers to the brokers are addressed efficiently by the AWS Managed Service Kafka Message System without any communication lapses?

  • Does each partition managed by the controller is active and running?

  • Are there any issues in the broker's connection to the consumer?

  • Does the request from the consumer to the broker and response from the broker to the consumer waits in the queue for a long time?

  • Does the CPU usage of each cluster in the target server is too high leading to sudden spikes in CPU usage?

  • Does the request from the consumer to the follower and response from the follower to the consumer waits in the queue for a long time?

  • Are the network transmitting/receiving errors for the broker is too high?

  • Does the request from the consumer to the producer and response from the producer to the consumer waits in the queue for a long time?

  • Does the average fetch throttle time taken for each request raised by the consumer is too high?

  • Does the message conversion and data transaction between the brokers are stable enough for the target AWS Managed Service Kafka to communicate efficiently?

  • Does the health and availability status of each volume used by the EC2 instances in the monitored region is sufficient?

  • Is the client’s connection to Zookeeper server is stable?

  • Is the network connectivity for the broker is stable enough for executing the Kafka operations?