Kafka Consumer Fetch Manager Test

The consumer periodically sends a request to a Kafka broker in order to fetch data from it. The consumer application then receives a number of records in a response, processes them and asks for more in the form of another request.

This test monitors the number of records consumed during each fetch request raised by the consumer and also identifies the average time taken for each request. Using this test, the administrators can rapidly capture real/probable problem conditions during fetch process and swiftly resolve them, so as to ensure the high availability of the target Apache Kafka Message System.

Target of the test : Apache Kafka Message System

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for the target Apache Kafka Message System.

Configurable parameters for the test

Parameter	Description
Test period	How often should the test be executed.
Host	The IP address of the target server that is being monitored.
Port	The port number through which the Apache Kafka Message System communicates. The default port is 9092.
Consumer Port	The port number through which the consumer communicates with the broker.
JMX User, JMX Password, and Confirm Password	These parameters appear only if the Measurement Mode is set to jmx. If JMX requires authentication only (but no security), then ensure that the JMX User and JMX Password parameters are configured with the credentials of a user with read-write access to JMX.

Measurements made by the test

Measurement	Description	Measurement Unit	Interpretation
Average data consumed rate	Indicates the average consumed data.	MB/Second
Average fetch latency	Indicates the average time taken for a fetch request.	Milliseconds	A very high value is indicative of the inability of the system to fetch requests.
Fetch requests	Indicates the number of fetch requests.	Requests/sec
Average fetch size	Indicates the average number of bytes fetched per request.	MB/request
Average fetch throttle time	Indicates the average throttle time.	Seconds	A very low value is required for this measure.
Records consumed rate	Indicates the average number of consumed records.	Records/sec
Maximum lag in number of records	Indicates the maximum lag in terms of number of records for any partition.	Number
Minimum lead in number of records	Indicates the minimum lead in terms of number of records for any partition.	Number
Average records per request	Indicates the average number of records in each request.	Records/sec