Flume Channel Test

Apache Flume has three main components - Source, Channel and Sink. A channel is a transient store that receives events from the source and buffers them until they are consumed by sinks. It acts as a bridge between the sources and the sinks. These channels are fully transactional, and they can work with any number of sources and sinks. Some examples of channels are the JDBC channel, the File system channel, the Memory channel, etc.

Given the channel is responsible for storing the data for intermediate duration and acts as a buffer for the critical data, it could lead to significant data loss in case the channel malfunctions or is unavailable. That is the reason it is absolutely important to monitor the channel to fully capture its operations and highlight if there is an issue or error because of which data loss may occur. The metrics and insights from monitoring can help administrators identify and act on potential problems even before they propagate into failure.

This test monitors every Flume channel and collects key metrics like write fails, fill percentage, etc. These metrics help administrators understand the current performance of the system and alerts when it requires intervention to fix the problems.

Target of the test : Apache Flume

Agent deploying the test : An internal agent

Outputs of the test : One set of results for each channel in Apache Flume agent being monitored.

Configurable parameters for the test

Parameter	Description
Test period	How often should the test be executed.
Host	The IP address of the target server that is being monitored.
Port	The port number through which the Apache Flume communicates. The default port is 8080.
FLUME JMX Remote Port	Specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in theflume-env.ps1file, in JVM_OPTS variable.
JMX Username, Password and Confirm Password	These parameters appear only if the Mode is set to JMX. If JMX requires authentication only (but no security), then ensure that the user and password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to Configuring the eG Agent to Support JMX Authentication. Confirm the password by retyping it in the Confirm Password text box.

Measurements made by the test

Measurement	Description	Measurement Unit	Interpretation
Channel fill capacity	Indicates the capacity of this channel as configured in Flume agent configuration file.	Number	Channel capacity refers to the maximum number of events that a channel is configured to hold.
Channel fill percentage	Indicates the percentage of capacityof this channel utilized out of total available capacity.	Percentage	If the channel fill percentage is high, there is high chance of write failures to this channel in next few minutes, administrators should start thinking about reducing the load of the channel.
Number of events currently present in a channel	Indicates the total number of events currently existing in the channel.	Batches/Sec	If the number of events is high, it could be reaching the channel capacity.
Event attempt to write	Indicates the number of event that sources have attempted to write to this channel in a sec.	Events/Sec	A high rate is justified if most of the write attempts are successful.
Event written and committed	Indicates the number of events that sources have successfully written and committed to this channel in a sec.	Events/Sec	A high rate of successful rights to the sink is always preferred but administrators have to keep an eye on the rate over a number of measurements.
Event read attempt from channel	Indicates the number of events that sinks which are connected to this channel, have attempted to read in a sec.	Events/Sec	The number should be high which means most of the events are successfully written to the channel.
Event successfully reads from channel	Indicates the number of events that sinks which are connected to this channel, have successfully read in a sec.	Events/Sec	If there are frequent channel read fails, administrators should investigate the issue.