Kafka Consumer Lag Test
Kafka Consumer lag indicates the lag between Kafka producers and consumers. If the rate of production of data from producers far exceeds the rate at which it is getting consumed by the consumers, then the consumer groups will exhibit a lag. This test helps the administrators to monitor how much lag there is between Kafka producers and consumers.
Target of the test : Apache Kafka Message System
Agent deploying the test : An internal/remote agent
Outputs of the test : One set of results for each consumer group in the target Apache Kafka Message System.
Parameter | Description |
---|---|
Test period |
How often should the test be executed. |
Host |
The IP address of the target server that is being monitored. |
Port |
The port number through which the Apache Kafka Message System communicates. The default port is 9092. |
Bootstrap Servers |
Specify the bootstrap server IP details in the BOOTSTRAP SERVERS text box. By default, it is Host Port. |
DD Frequency |
Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 1:1. This indicates that, by default, detailed measures will be generated every time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency. |
Detailed Diagnosis |
To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option. The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:
|
Measurement | Description | Measurement Unit | Interpretation |
---|---|---|---|
Current offset |
Indicates the number of current offsets |
Number |
An offset is a simple integer number that is used by Kafka to maintain the current position of a consumer. If current offset is positive - that means that there is a lag. In most cases, if your Kafka Producer is actively producing messages and the Kafka Consumers are actively consuming, you will have a small lag here. This is expected. The problems start when the lag is significant or is constantly growing. That means that the data is not processed fast enough. |
End offset |
Indicates the number of end offsets. |
Number |
Inside the Brokers, data is stored in one or more Topics, and each Topic consists of one or more Partitions. When writing data a Broker actually writes it into a specific Partition. As it writes data it keeps track of the last “write position” in each Partition. This is called Latest Offset, also known as Log End Offset. Each Partition has its own independent Latest Offset. |
Lag |
Indicates the number of lags between Kafka producers and consumers. |
Number |
A very low value is required for this measure. Use the detailed diagnosis of this measure to know the Client ID and Host details. |
Lag percent |
Indicates the percentage of lags between Kafka producers and consumers. |
|
A very low value is required for this measure. |