AWS Kinesis Firehose Test

Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk.

With Kinesis Data Firehose, you don't need to write applications or manage resources. You configure your data producers to send data to Kinesis data delivery streams using different sources: a Kinesis stream, the Kinesis Agent, or the Kinesis Firehose API using the AWS SDK. After data is sent to a delivery stream, it is automatically delivered to the destination you choose.

For instance, with Amazon Kinesis Data Firehose, you can capture data continuously from connected devices such as consumer appliances, embedded sensors, and TV set-top boxes. Amazon Kinesis Data Firehose loads the data into Amazon Redshift, enabling you to provide your customers near real-time access to metrics, insights, and dashboards.

If data/records sent by the delivery stream do not reach the intended destination, the analytics derived from that data may be incomplete, inaccurate, or unusuable. To avoid this, it is necessary to track the ingestion and transmission of data and records by the data delivery streams to each of the destinations and promptly capture delivery failures. This is where the AWS Kinesis FireHost test helps!

This test auto-discovers the data delivery streams created using Kinesis FireHose. For each delivery stream, the test tracks the delivery attempts made by that stream to each destination and reports the count of successful deliveries. This way, the test sheds light on delivery failures and the destinations to which delivery failed. Additionally, the test also measures the throughput of each delivery stream by reporting the number of records and amount of data received and processed by that stream. In the process, you can accurately identify those data delivery streams that are experiencing processing bottlenecks. To help you isolate what could be causing a stream to process data/records slowly, the test also reports the time taken by the stream to process different API calls. The API call/method that could be contributing to the slowness can thus be identified. The status of data transformation functions is also checked periodically, so that data transformation failures (if any) are brought to light.

Target of the test: Amazon EC2 Cloud

Agent deploying the test : A remote agent

Outputs of the test : One set of results for each data delivery stream

First-level descriptor: AWS Region

Second-level descriptor: Data delivery stream

Configurable parameters for the test
Parameter Description

Test Period

How often should the test be executed.

Host

The host for which the test is to be configured.

AWS Access Key, AWS Secret Key, Confirm AWS Access Key, Confirm AWS Secret Key

To monitor an Amazon EC2 instance, the eG agent has to be configured with the access key and secret key of a user with a valid AWS account. For this purpose, we recommend that you create a special user on the AWS cloud, obtain the access and secret keys of this user, and configure this test with these keys. The procedure for this has been detailed in the Obtaining an Access key and Secret key topic. Make sure you reconfirm the access and secret keys you provide here by retyping it in the corresponding Confirm text boxes.

Proxy Host and Proxy Port

In some environments, all communication with the AWS EC2 cloud and its regions could be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none , indicating that the eG agent is not configured to communicate via a proxy, by default.

Proxy User Name, Proxy Password, and Confirm Password

If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy User Name and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box. By default, these parameters are set to none, indicating that the proxy sever does not require authentication by default.

Proxy Domain and Proxy Workstation

If a Windows NTLM proxy is to be configured for use, then additionally, you will have to configure the Windows domain name and the Windows workstation name required for the same against the Proxy Domain and Proxy Workstation parameters. If the environment does not support a Windows NTLM proxy, set these parameters to none.

Exclude Region

Here, you can provide a comma-separated list of region names or patterns of region names that you do not want to monitor. For instance, to exclude regions with names that contain 'east' and 'west' from monitoring, your specification should be: *east*,*west*

Measurements made by the test
Measurement Description Measurement Unit Interpretation

Data delivered to ES

Indicates the amount of data that this data delivery stream indexed to Amazon ES.

KB

 

Records delivered to ES

Indicates the number of records that this delivery stream indexed to Amazon ES.

Number

 

Total records delivered successfully to ES

Indicates the number of successfully indexed records over the number of records that were attempted.

Number

A value lesser than 1 for this measure is indicative of delivery failures.

Data delivered to RedShift

Indicates the amount of data that this data delivery stream copied to Amazon RedShift.

KB

 

Records delivered to RedShift

Indicates the number of records this data delivery stream copied to Amazon RedShift.

Number

 

Total records delivered successfully to RedShift

Indicates the number of successful Amazon Redshift COPY commands over the number of all Amazon Redshift COPY commands that were issued by this data delivery stream.

Number

A value lesser than 1 for this measure is indicative of delivery failures.

Data delivered to S3

Indicates the amount of data that this data delivery stream delivered to S3.

Number

 

Records delivered to S3

Indicates the number of records that this data delivery stream delivered to S3.

Number

 

Total records delivered successfully to S3

Indicates the number of successful Amazon S3 put commands over the number of all Amazon S3 put commands issued by this data delivery stream.

Number

A value lesser than 1 for this measure is indicative of delivery failures.

Age of the oldest record

Indicates the age (from getting into Kinesis Firehose to now) of the oldest record in this data delivery stream.

Secs

An abnormally high value for this measure could point to a record that is still in the data delivery stream and has not been delivered yet. The reasons for this delivery bottleneck will have to be investigated.

Incoming data

Indicates the amount of data coming into this data delivery stream.

KB

 

Incoming records

Indicates the number of records ingested into this data delivery stream.

Number

 

Describedeliverystream latency

Indicates the time taken by Kinesis Firehose to perform the DescribeDeliveryStream operation for this data delivery stream.

Secs

The DescribeDeliveryStream API command describes the specified delivery stream and gets the status.

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API call is slowing down processing.

Describedeliverystream erequests

Indicates the total number of Describedeliverystream requests for this data delivery stream.

Number

 

Listdeliverystream operation latency

Indicates the time taken by Kinesis Firehose to execute the Listdeliverystreams API call and return output for this data delivery stream.

Secs

The ListDeliveryStreams operation Lists your delivery streams.

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API call is slowing down processing.

 

Listdeliverystream requests

Indicates the total number of Listdeliverystream requests for this data delivery stream.

Number

 

Putrecord latency

Indicates the time taken by Kinesis Firehose to execute the Putrecord API operation to write a record into this data delivery stream.

Secs

PutRecord writes a single data record into an Amazon Kinesis Firehose delivery stream.

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API operation is slowing down processing.

Putrecord requests

Indicates the total number of Listdeliverystream requests for this data delivery stream.

Number

Putrecord data

Indicates the amount of data put into this data delivery stream using the PutRecord API operation.

KB

The PutRecord and PutRecordBatch operations together can put a maximum of 5 MB of data per second into a delivery stream for the US East (N. Virginia), US West (Oregon), and EU (Ireland) regions. The limit is 1 MB/Sec for US East (Ohio), US West (N. California), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), and EU (Frankfurt) regions.

You can submit a limit increase request using the Amazon Kinesis Data Firehose Limits form.

If the increased limit is much higher than the running traffic, it causes very small delivery batches to destinations, which is inefficient and can result in higher costs at the destination services. Be sure to increase the limit only to match current running traffic, and increase the limit further if traffic increases.

Putrecordbatch data

Indicates the amount of data put into this data delivery stream using the PutRecordBatch API operation.

KB

The PutRecord and PutRecordBatch operations together can put a maximum of 5 MB of data per second into a delivery stream for the US East (N. Virginia), US West (Oregon), and EU (Ireland) regions. The limit is 1 MB/Sec for US East (Ohio), US West (N. California), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), and EU (Frankfurt) regions.

You can submit a limit increase request using the Amazon Kinesis Data Firehose Limits form.

If the increased limit is much higher than the running traffic, it causes very small delivery batches to destinations, which is inefficient and can result in higher costs at the destination services. Be sure to increase the limit only to match current running traffic, and increase the limit further if traffic increases.

Putrecordbatch latency

Indicates the time taken to put records into this data delivery stream using the PutRecordBatch API operation.

Secs

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API operation is slowing down processing.

Putrecordbatch records

Indicates the number of records that were added to this data delivery stream using the PutRecordBatch API operation.

Number

The PutRecord and PutRecordBatch operations together can put a maximum of 5000 records per second into a delivery stream for the US East (N. Virginia), US West (Oregon), and EU (Ireland) regions. The limit is 1000 records per second for US East (Ohio), US West (N. California), Asia Pacific (Singapore), Asia Pacific (Sydney), Asia Pacific (Tokyo), and EU (Frankfurt) regions.

You can submit a limit increase request using the Amazon Kinesis Data Firehose Limits form.

If the increased limit is much higher than the running traffic, it causes very small delivery batches to destinations, which is inefficient and can result in higher costs at the destination services. Be sure to increase the limit only to match current running traffic, and increase the limit further if traffic increases.

Putrecordbatch request

Indicates the total number of PutRecordBatch requests for this data delivery stream.

Number

 

Updatedeliverystream latency

Indicates the time taken to update this data delivery stream using the UpdateDeliveryStream API operation.

Secs

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API operation is slowing down processing.

Updatedeliverystream request

Indicates the number of requests for UpdateDeliveryStream API operations for that data delivery stream.

Number

 

Lambda function invocation

Indicates the time taken by Lambda function invocation for this data delivery stream.

Secs

Kinesis Data Firehose can invoke your Lambda function to transform incoming source data and deliver the transformed data to destinations. You can enable Kinesis Data Firehose data transformation when you create your delivery stream.

This measure is reported for only those data delivery streams for which data transformation has been enabled.

If any slowness or low throughput is noticed in a data delivery stream, then you can compare the value of this measure with the other latency measures that this test reports for that data delivery stream to know which API operation is slowing down processing.

Successful Lambda function invocations

Indicates the number of the successful Lambda function invocations over the number of the total Lambda function invocations by this data delivery stream.

Number

This measure is reported for only those data delivery streams for which data transformation has been enabled.

A value less than 1 for this measure is indicative of one/more Lambda function invocation failures.

Common causes for failure of Lambda invocations include a network timeout or reaching the Lambda invocation limit. In the event of such a failure, Kinesis Data Firehose retries the invocation three times by default. If the invocation does not succeed, Kinesis Data Firehose then skips that batch of records. The skipped records are treated as unsuccessfully processed records.

Successfully processed data

Indicates the amount of data that this data delivery stream has successfully processed.

KB

This measure is reported for only those data delivery streams for which data transformation has been enabled.

Successfully processed records

Indicates the number of records successfully processed by this data delivery stream.

Number

This measure is reported for only those data delivery streams for which data transformation has been enabled.

For a data transformation-enabled data delivery stream, the value of this measure should be high ideally. A low value is indicative of data transformation failures. Common causes for failure of Lambda invocations include a network timeout or reaching the Lambda invocation limit. In the event of such a failure, Kinesis Data Firehose retries the invocation three times by default. If the invocation does not succeed, Kinesis Data Firehose then skips that batch of records. The skipped records are treated as unsuccessfully processed records.