AWS DynamoDB Replicating Record Status Test

Amazon Kinesis Data Streams can be used to capture changes to Amazon DynamoDB. Kinesis Data Streams capture item-level modifications in any DynamoDB table and replicate them to a Kinesis data stream. The applications can access this stream and view item-level changes in near-real time. When a Kinesis data stream is enabled for a DynamoDB table, the table sends out a data record that captures any changes to that table’s data. These data records are captured and published in near-real time. After they are written to the Kinesis data stream, they can be read just like any other record. A Kinesis data stream counts its throughput in shards. You will need to provision an appropriate number of shards on your Kinesis data stream to accommodate the change in data capture records from DynamoDB. When the data streams are under-provisioned, that can lead to unsuccessful replications due to excessive throttling. Customer-controlled configurations such as manual updates to your Kinesis data stream’s access policies that prevent DynamoDB from adding data to your data stream can also lead to unsuccessful replication. These unsuccessful replications cause the build-up of unreplicated records, that can eventually lead to critical data loss. However, by continuously monitoring replicating record status, you would be able to determine whether the stream has sufficient capacity or not. The AWS DynamoDB Replicating Record Status Test does exactly the same.

This test monitors the age of the oldest unreplicated records on the target server, which helps administrators to find out the unsuccessful replication attempts due to customer controlled configuration choices. This way, they can proactively increase the provisioning of data streams before it leads to any critical data loss.

Target of the test : An AWS DynamoDB server

Agent deploying the test : A remote agent

Outputs of the test : One set of results for the target AWS DynamoDB server being monitored.

Configurable parameters for the test
Parameter Description

Test Period

How often should the test be executed.

Host

The IP address of the AWS DynamoDB server that is being monitored.

AWS Region

This test uses AWS SDK to interact with AWS DynamoDB and pull relevant metrics. To enable the test to connect to AWS, you need to configure the test with the name of the region to which all requests for metrics should be routed, by default. Specify the name of this AWS Region in this text box.

AWS Access Key ID, AWS Secret Access Key and Confirm Password

To monitor AWS DynamoDB, the eG agent has to be configured with the access key and secret key of a user with a valid AWS account. For this purpose, we recommend that you create a special user on the AWS cloud, obtain the access and secret keys of this user, and configure this test with these keys. The procedure for this has been detailed in the Obtaining an Access key and Secret key topic. Make sure you reconfirm the access and secret keys you provide here by retyping it in the corresponding Confirm Password text box.

Timeout Seconds

Specify the maximum duration (in seconds) for which the test will wait for a response from the server. The default is 120 seconds.

DD Row Count

By default, the detailed diagnosis of this test, if enabled, will report only the top-10 tables. This is why, the DD Row Count parameter is set to 10 by default. If you want to include more or less tables in detailed diagnosis, then change the value of this parameter accordingly.

Detailed Diagnosis

To make diagnosis more efficient and accurate, the eG Enterprise embeds an optional detailed diagnostic capability. With this capability, the eG agents can be configured to run detailed, more elaborate tests as and when specific problems are detected. To enable the detailed diagnosis capability of this test for a particular server, choose the On option. To disable the capability, click on the Off option.

The option to selectively enable/disable the detailed diagnosis capability will be available only if the following conditions are fulfilled:

  • The eG manager license should allow the detailed diagnosis capability
  • Both the normal and abnormal frequencies configured for the detailed diagnosis measures should not be 0.
Measurements made by the test

Measurement

Description

Measurement Unit

Interpretation

Maximum age of old unreplicated records

Indicates the maximum age of oldest unreplicated records on the DynamoDB table in the target server.

Seconds

This measure grows based on unsuccessful replication attempts when these are caused by customer-controlled configuration choices. To keep this metric as low as possible, you might need to ensure the right provisioning of your Kinesis data stream capacity and make sure that DynamoDB’s permissions are unchanged.

Use detailed diagnosis of this measure to find out the Table name, Delegated operation, Age of old unreplicated records.

Minimum age of old unreplicated records

Indicates the minimum age of oldest unreplicated records on the DynamoDB table in the target server.

Seconds