Flume Source Test
Apache Flume has three main components - Source, Channel and the Sink. A source is the component of an agent which receives data from the data generators and transfers it to one or more channels in the form of flume events. Apache Flume supports several types of sources and each source receives events from a specified data generator, for example an Avro source, a Thrift source,a Twitter 1% source etc. Given that the source is responsible for acquiring data from different sources, any issue or problem with the source infrastructure can lead to data loss which can result in mild to catastrophic failure depending upon the criticality of the business and data lost during the process. That is why it is absolutely important to monitor the source to fully capture the activities of the source and highlight if there is an issue or error because of which the data loss may occur. The metrics and insights from monitoring can help administrators identify and act on potential problems even before they propagate into failure.
This test monitors every Flume source and collects key metrics like events written to the channel, write fails, connection count, etc. These metrics help administrators understand the current performance of the system and alerts when it requires intervention to fix the problems.
Target of the test : Apache Flume
Agent deploying the test : An internal agent
Outputs of the test : One set of results for each source in Apache Flume agent being monitored.
| Parameter | Description |
|---|---|
|
Test period |
How often should the test be executed. |
|
Host |
The IP address of the target server that is being monitored. |
|
Port |
The port number through which the Apache Flume communicates. The default port is 8080. |
|
FLUME JMX Remote Port |
Specify the port at which the JMX listens for requests from remote hosts. Ensure that you specify the same port that you configured in theflume-env.ps1file, in JVM_OPTS variable. |
|
JMX Username, Password and Confirm Password |
These parameters appear only if the Mode is set to JMX. If JMX requires authentication only (but no security), then ensure that the user and password parameters are configured with the credentials of a user with read-write access to JMX. To know how to create this user, refer to Configuring the eG Agent to Support JMX Authentication. Confirm the password by retyping it in the Confirm Password text box. |
| Measurement | Description | Measurement Unit | Interpretation |
|---|---|---|---|
|
Accepted Count |
Indicates the number of events received by this source and successfully committed to the channel. |
Events/Sec |
In this case there is only one event per batch, which is not optimal. The performance of the system can be enhanced with healthy number of events per batch. |
|
Batch accepted count |
Indicates the number of batches of events received by this source and committed to the channel. |
Batches/Sec |
The number should be high which means most of the events are successfully written to the channel. |
|
Batch received count |
Indicates the number of batches of events received by this source from the data generators in an interval. |
Batches/Sec |
High number of events generated by the data sources should be received by source. |
|
Received count |
Indicates the number of events received by this source from the data generators in an interval. |
Events/Sec |
The count of events received with one event per batch, but again the number should be high for good performance. |
|
Channel write fail |
Indicates the number of failed attempts to write to the channel from this source. |
Events/Sec |
If there are frequent channel write fails, administrators should investigate the issue. |
|
Events accepted count |
Indicates the number of events successfully written to the channel from this source in a second. |
Events/Sec |
The number should be high which means most of the events are successfully written to the channel. |
|
Event read fail |
Indicates the number of failed reads of events by this source from the data generators. |
Events/Sec |
If there are frequent channel read fails, administrators should investigate the issue. |
|
Events received count |
Indicates the number of events received by this source from the data generators in a second. |
Events/Sec |
The count of events received with one event per batch, but again the number should be high for good performance. |
|
Generic processing fail |
Indicates the number of events which were failed at processing in this source in a second. |
Events/Sec |
The processing fail should be investigated thoroughly as it could be because of an issue with this source. |
|
Open connection count |
Indicates the number of connections open between Apache flume and data providers for this source. |
Connections/Sec |
If the number of connections is persistently high it could be because the connections are not reused or freed up. |
|
Accepted rate |
Indicates the percentage of events successfully written to the channel out of total number of events received in this source. |
Percentage |
The acceptance rate should be in optimal range. If you notice a downward trend in this rate over a number of measurements, it could be a reason for investigation. |
|
Received rate |
Indicates the percentage of events successfully received by this source out of the total events generated by data generators. |
Percentage |
The received rate should be in optimal range. If you notice a downward trend in this rate over a number of measurements, it could be a reason for investigation. |
|
Batch accepted rate |
Indicates the percentage of batches of events successfully written to the channel out of total number of events received in this source. |
Percentage |
The rate should be in optimal range. If you notice a downward trend in this rate over a number of measurements, it could be a reason for investigation. |
|
Batch received rate |
Indicates the percentage of batches of events successfully received by this source out of the total events generated by data generators. |
Percentage |
The rate should be in optimal range. If you notice a downward trend in this rate over a number of measurements, it could be a reason for investigation. |
|
Events accepted rate |
Indicates the percentage of events successfully written to the channel out of total number of events received in this source. |
Percentage |
|
|
Events received rate |
Indicates the percentage of events successfully received by this source out of the total events generated by data generators. |
Percentage |
|
