Kafka Consumer Lag By Group Test

Kafka Consumer lag indicates the lag between Kafka producers and consumers. If the rate of production of data from producers far exceeds the rate at which it is getting consumed by the consumers, then the consumer groups will exhibit a lag. This test helps the administrators to find out the lags categorized based on the individual group. This test also helps the administrator to monitor the status of the group as well as identify how much percentage of lag there is between Kafka producers and consumers.

Target of the test : Apache Kafka Message System

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for each consumer group in the target Apache Kafka Message System.

Configurable parameters for the test
Parameter Description

Test period

How often should the test be executed.

Host

The IP address of the target server that is being monitored.

Port

The port number through which the Apache Kafka Message System communicates. The default port is 9092.

Bootstrap Servers

Bootstrap Servers is a comma-separated list of host and port pairs that are the addresses of the Kafka brokers in a "bootstrap" Kafka cluster that a Kafka client connects to initially to bootstrap itself. By default, the BOOTSTRAP SERVERS text box will display the value as Host:Port. If a single Kafka Broker is associated with multiple Bootstraps in a Kafka cluster, then specify a comma-separated list of values as Host1:Port1, Host2:Port2,.... in this text box. If Kafka Message system is installed in a container, then the value will be displayed as Container ID:Port in this text box.

Lag Percent Cutoff

If the consumer lag percent is more than a specified cut off value, then the consumer information will be displayed in the UI. By default the Lag Percent cutoff value is 25.

Listener Name

Specify the name of the listener service associated with the managed Apache Kafka Message System. By default, this parameter is set to PLAINTEXT. This means that by default the test will attempt to connect to and check the status of the 'default listener'. However, if the Apache Kafka Message System has been configured using a 'different listener', then you should select the value of the LISTENER NAME parameter.

Truststore

This parameter is applicable only if the target Apache Kafka Message System is SSL-enabled, if not, set this parameter to none. TrustStore is used to store certificates from Certified Authorities (CA) that verify and authenticate the certificate presented by the server in an SSL connection. Therefore, the eG agent should have access to the truststore where the certificates are stored to authenticate and connect with the target database and collect metrics. For this, first import the certificates into the following default location <eG_INSTALL_DIR>/lib/security/mytruststore.jks. As an example, to know how to import the certificate into the truststore, refer toPre-requisites for monitoring Oracle Cluster. Then, provide the truststore file name in this text box. For example: mytruststore.jks. By default, none is specified against this text box.

Truststore Password and Confirm Password

This parameter is applicable only if the target Apache Kafka Message System is SSL-enabled, if not, set this parameter to none. If a Truststore File name is provided, then, in this text box, provide the password that is used to obtain the associated certificate details from the Truststore File. By default, this parameter is set to none. Confirm the password by retyping the password in the Confirm Password text box.

Keystore File

This parameter is applicable only if the target Apache Kafka Message System is SSL-enabled, if not, set this parameter to none.

Keystore contains the private keys for the certificates that the client can provide to the server upon request. eG agent requires access to the keystore where client certificate is stored to send that to the server so that the server validates the certificate against the one contained in its trustore. For this purpose, first create the client certificate in the following default location /opt/egurkha/jre/lib/security/egmqsslstore.jks.

Keystore Password

This parameter is applicable only if the target Apache Kafka Message System is SSL-enabled, if not, set this parameter to none.

If a Keystore File name or file path is provided, then, in this text box, provide the password that is used to obtain the associated certificate details from the Keystore File.

Confirm Password

Confirm the Password for Keystore by retyping it here.

SASL SSL Username

This parameter is applicable only if the target Apache Kafka Message System is SSL-enabled, if not, set this parameter to none.

SASL SSL Username parameter generally refers to the configuration of authentication and encryption protocols in Apache Kafka Message System, where SASL (Simple Authentication and Security Layer) is layered over SSL to provide secure, authenticated connections.

SASL SSL Password and Confirm Password

Specify the password in the SASL SSL Password text box for authenticated connection and confirm the password by retyping the password in the Confirm Password text box.

SASL SSL Mechanism

SASL SSL Mechanism in Apache Kafka Message System refers to the username and credential authentication methods supported through the SASL (Simple Authentication and Security Layer) framework, with communication secured via SSL/TLS encryption The two different types of authentication methods are SCRAM-SHA-256 and SCRAM-SHA-512. Select the required authentication method from the SASL SSL Mechanism list.

DD Frequency

Refers to the frequency with which detailed diagnosis measures are to be generated for this test. The default is 1:1. This indicates that, by default, detailed measures will be generated every time this test runs, and also every time the test detects a problem. You can modify this frequency, if you so desire. Also, if you intend to disable the detailed diagnosis capability for this test, you can do so by specifying none against DD frequency.

Detailed Diagnosis

To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability
  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measurements made by the test
Measurement Description Measurement Unit Interpretation

Consumers

Indicates the number of consumers associated to this group.

 

 

Topics

Indicates the number of topics associated to this group.

 

 

Lag

Indicates the number of lags between Kafka producers and consumers in this group.

Number

A very low value is required for this measure.

Use the detailed diagnosis of this measure to know the Client ID and Host details.

Lag percentage

Indicates the percentage of lags between Kafka producers and consumers in this group.

Percent

A very low value is required for this measure.

Coordinator

Indicates the number of coordinators in this group.

Number

 

State

Indicates the state of the consumer group.

 

The values reported by this measure and its numeric equivalents are mentioned in the table below:

Measure Value

 Numeric Value
DEAD 0
UNKNOWN 5
EMPTY 10
PREPARING_REBALANCE 90
COMPLETING_REBALANCE 95
STABLE 100

Note:

By default, this measure reports the Measure Values listed in the table above to indicate the state of the consumer group.