AWS Internet of Things - IoT Test

AWS IoT provides secure, bi-directional communication between Internet-connected devices such as sensors, actuators, embedded micro-controllers, or smart appliances and the AWS Cloud. This enables you to collect telemetry data from multiple devices, and store and analyze the data. You can also create applications that enable your users to control these devices from their phones or tablets.

When using AWS IoT, devices are managed as things. A thing is a representation of a specific device or logical entity. It can be a physical device or sensor (for example, a light bulb or a switch on a wall). It can also be a logical entity like an instance of an application or physical entity that does not connect to AWS IoT but is related to other devices that do (for example, a car that has engine sensors or a control panel). Information about a thing is stored in the registry as JSON data.

These devices/things report their state by publishing messages, in JSON format, on MQTT topics. Each MQTT topic has a hierarchical name that identifies the device whose state is being updated. When a message is published on an MQTT topic, the message is sent to the AWS IoT MQTT message broker, which is responsible for sending all messages published on an MQTT topic to all clients subscribed to that topic. The message broker supports the use of the MQTT protocol to publish and subscribe and the HTTPS protocol to publish.

You can create rules that define one or more actions to perform based on the data in a message. For example, you can insert, update, or query a DynamoDB table or invoke a Lambda function. Rules use expressions to filter messages. When a rule matches a message, the rules engine invokes the action using the selected properties. Rules also contain an IAM role that grants AWS IoT permission to the AWS resources used to perform the action.

Each device has a shadow that stores and retrieves state information. Each item in the state information has two entries: the state last reported by the device and the desired state requested by an application. An application can request the current state information for a device. The shadow responds to the request by providing a JSON document with the state information (both reported and desired), metadata, and a version number. An application can control a device by requesting a change in its state. The shadow accepts the state change request, updates its state information, and sends a message to indicate the state information has been updated. The device receives the message, changes its state, and then reports its new state.

Figure 1 : How AWS IoT works

Typically, AWS IoT policies are used to enable devices to connect to the message broker and get authorized, so they can perform AWS IoT operations, such as subscribing or publishing to MQTT topics via the broker, or get, update, or delete a device's shadow. If any of these policy actions fail or experience errors, then devices may no longer be able to communicate with the AWS cloud. Also, where messages match configured rules, if the defined rule actions fail, the failure may disrupt communication between the devices and critical AWS services such as AWS Lambda, Amazon S3, Amazon DynamoDB, etc. Moreover, AWS also imposes limits on the AWS IOT service; if these limits are violated, it might adversely impact the overall performance of AWS IOT . To ensure the peak performance of AWS IOT therefore, administrators should continuously track the different types of requests to the message broker, capture errors, failures, and violations promptly, and rapidly initiate measures to fix the faults. This is where the AWS Internet of Things - IOT Test helps!

By default, this test tracks the different types of requests (connection, publish, subscribe, unsubscribe, get/delete/update device shadow, etc.) to the message broker, and automatically discovers the protocols in use. For each protocol, the test then reports the count of requests that were successful, that violated the AWS IOT limits, and that which experienced errors. In the process, administrators can determine whether/not specific protocols are failing or are returning error responses frequently; such protocols could be suspect and may have to be investigated. Optionally, you can configure this test to report metrics for each rule or each action type configured for rules. This provides insights into errors (if any) in messages that match each rule and failed invocations of every action type.

Target of the test: Amazon EC2 Cloud

Agent deploying the test : A remote agent

Outputs of the test : One set of results for each protocol / rule / action type

First-level descriptor: AWS Region

Second-level descriptor: Protocol / rule / action type, depending upon the option chosen against the IOT Filter Name parameter.

Configurable parameters for the test
Parameter Description

Test Period

How often should the test be executed.

Host

The host for which the test is to be configured.

AWS Access Key, AWS Secret Key, Confirm AWS Access Key, Confirm AWS Secret Key

To monitor an Amazon EC2 instance, the eG agent has to be configured with the access key and secret key of a user with a valid AWS account. For this purpose, we recommend that you create a special user on the AWS cloud, obtain the access and secret keys of this user, and configure this test with these keys. The procedure for this has been detailed in the Obtaining an Access key and Secret key topic. Make sure you reconfirm the access and secret keys you provide here by retyping it in the corresponding Confirm text boxes.

Proxy Host and Proxy Port

In some environments, all communication with the AWS EC2 cloud and its regions could be routed through a proxy server. In such environments, you should make sure that the eG agent connects to the cloud via the proxy server and collects metrics. To enable metrics collection via a proxy, specify the IP address of the proxy server and the port at which the server listens against the Proxy Host and Proxy Port parameters. By default, these parameters are set to none , indicating that the eG agent is not configured to communicate via a proxy, by default.

Proxy User Name, Proxy Password, and Confirm Password

If the proxy server requires authentication, then, specify a valid proxy user name and password in the Proxy User Name and Proxy Password parameters, respectively. Then, confirm the password by retyping it in the Confirm Password text box. By default, these parameters are set to none, indicating that the proxy sever does not require authentication by default.

Proxy Domain and Proxy Workstation

If a Windows NTLM proxy is to be configured for use, then additionally, you will have to configure the Windows domain name and the Windows workstation name required for the same against the Proxy Domain and Proxy Workstation parameters. If the environment does not support a Windows NTLM proxy, set these parameters to none.

Exclude Region

Here, you can provide a comma-separated list of region names or patterns of region names that you do not want to monitor. For instance, to exclude regions with names that contain 'east' and 'west' from monitoring, your specification should be: *east*,*west*

IOT Filter Name

By default, this parameter is set to Protocol. In this case, this test will report metrics for every protocol. For each protocol, the test reports the count of requests that were successful, that violated the AWS IOT limits, and that which experienced errors.

If required, you can override this default setting by setting the IOT Filter Name to one of the following:

  • RuleName: For metrics on each rule that is configured, select this option. In this case, the test will only report the count of topics that match a rule condition and the count of errors (if any) in matching messages.
  • ActionType: For metrics on every action type that is configured for a rule, select this option. In this case, the test will only report the count of successful and failed invocations of each action type.
Measurements made by the test
Measurement Description Measurement Unit Interpretation

Connection request authorized errors

Indicates the number of connection requests made using that protocol, which could not be authorized by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many connection requests made using a particular protocol were unauthorized.

Connection request client errors

Indicates the number of connection requests made using that protocol, which were rejected because the MQTT message did not meet the requirements defined in AWS IoT limits.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value indicates that a connection request has been rejected.

Typically, AWS requires that a client ID in a connection request should not be more than 128 bytes in size and should consist of only UTF-8 encoded characters. Requests from clients that do not fulfill this requirement will be rejected.

Connection request server errors

Indicates the number of connection requests made using this protocol that failed because an internal error occurred.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value implies that for one/more connection requests made using that protocol, the message broker returned the error code 500.

Connection request success

Indicates the number of connection requests made using this protocol that resulted in successful connections to the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A high value is desired for this measure.

Connection request throttled

Indicates the number of connection requests made using this protocol that were throttled because the client exceeded the allowed connect request rate.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

By default, AWS IoT limits an account to a maximum of 300 MQTT CONNECT requests per second. If this request rate is exceeded, then AWS throttles connection requests.

You may want to observe the variations to the value of this measure over time and figure out if too many connection requests are throttled. If so, you may want to request for changing the connect request rate limit.

Delete thing shadow requests accepted

Indicates the number of DeleteThingShadow requests made using this protocol, which were processed successfully.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A DeleteThingShadow request deletes the shadow of a specified thing.

Get thing shadow requests accepted

Indicates the number of GetThingShadow requests made using this protocol, which were processed successfully.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A GetThingShadow request gets the shadow of a specified thing.

Successful ping messages

Indicates the number of ping messages sent using this protocol, which were successfully received by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Publish in authorized errors

Indicates the number of inbound publish requests made using this protocol, which could not be authorized by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many publish requests made using a particular protocol were unauthorized. Typically, publish requests that return the HTTP status code 401 are classified as unauthorized requests.

Publish in client errors

Indicates the number of inbound publish requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value indicates that a publish request has been rejected.

Publish in server errors

Indicates the number of inbound publish requests made using this protocol, which the message broker failed to process because an internal error occurred.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value implies that for one/more inbound publish requests made using that protocol, the message broker returned the error code 500.

Publish in success

Indicates the number of inbound publish requests made using this protocol that were successfully processed by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A high value is desired for this measure.

Publish in throttled

Indicates the number of inbound publish requests made using this protocol, which were throttled because the client exceeded the allowed inbound request rate.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Typically, AWS IoT limits each client connection to 100 inbound publish requests per second. If this limit is exceeded, then subsequent inbound publish requests are throttled by AWS.

You may want to observe variations to this measure over time and figure out if too many inbound publish requests are throttled. If so, you may want to request for changing the inbound publish request rate limit.

Publish out authorized errors

Indicates the number of outbound publish requests made using this protocol, which could not be authorized by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many outbound publish requests made using a particular protocol were unauthorized. Typically, publish requests that return the HTTP status code 401 are classified as unauthorized requests.

Publish out client errors

Indicates the number of outbound publish requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value indicates that an outbound publish request has been rejected.

Publish out success

Indicates the number of outbound publish requests made using this protocol that were successfully processed by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A high value is desired for this measure.

Rules executed

Indicates the number of rules executed.

Number

 

Subscription request authorized errors

Indicates the number of subscription requests made using this protocol, which could not be authorized by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

The value 0 is desired for this measure. A high value is a cause for concern as it indicates that many subscription requests made using a particular protocol were unauthorized. Typically, subscription requests that return the HTTP status code 401 are classified as unauthorized requests.

Subscription request client errors

Indicates the number of subscription requests made using this protocol, which were rejected because the message did not meet the requirements defined in AWS IoT limits.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value indicates that a subscription request has been rejected.

Subscription request server errors

Indicates the number of subscription requests made using this protocol, which the message broker failed to process because an internal error occurred.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value implies that for one/more inbound subscription requests made using that protocol, the message broker returned the error code 500.

Subscription request success

Indicates the number of subscription requests made using this protocol that were successfully processed by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A high value is desired for this measure.

Subscription request throttled

Indicates the number of subscription requests made using this protocol, which were throttled because the client exceeded the allowed subscription request rate.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Typically, AWS IoT limits each client connection to subscribe to up to 50 subscriptions. A SUBSCRIBE request that pushes the total number of subscriptions past 50 results in the connection being disconnected.

You may want to observe variations to this measure over time and figure out if too many subscription requests are throttled. If so, you may want to request for changing the subscription request rate limit.

Unsubscribe request client errors

Indicates the number of UNSUBSCRIBE requests made using this protocol, which were rejected because the UNSUBSCRIBE message did not meet the requirements defined in AWS IoT limits.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value indicates that an UNSUBSCRIBE request has been rejected.

Unsubscribe request server errors

Indicates the number of UNSUBSCRIBE requests made using this protocol, which the message broker failed to process because an internal error occurred.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

Ideally, the value of this measure should be 0. A non-zero value implies that for one/more UNSUBSCRIBE requests made using that protocol, the message broker returned the error code 500.

Unsubscribe request success

Indicates the number of UNSUBSCRIBE requests made using this protocol that were successfully processed by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

A high value is desired for this measure.

Unsubscribe request throttled

Indicates the number of UNSUBSCRIBE requests made using this protocol, which were throttled because the client exceeded the allowed unsubscribe request rate.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

You may want to observe variations to this measure over time and figure out if too many UNSUBSCRIBE requests are throttled. If so, you may want to request for changing the request rate limit.

Update thing shadow requests accepted

Indicates the number of a UpdateThingShadow requests made using this protocol, which were received by the message broker.

Number

This measure is reported only for each protocol - i.e., only if the IOT Filter Name parameter is set to 'Protocol'.

An UpdateThingShadow request updates the shadow of a specified thing.

Topic match

Indicates the number of incoming messages published on a topic on which this rule is listening.

Number

This measure is reported only for each rule - i.e., only if the IOT Filter Name parameter is set to 'RuleName'.

JSON parse errors

Indicates the number of JSON parse errors that occurred in messages published on a topic on which this rule is listening.

Number

This measure is reported only for each rule - i.e., only if the IOT Filter Name parameter is set to 'RuleName'.

Ideally, the value of this measure should be 0.

Successful rule action invocations

Indicates the number of successful invocations of this rule action type.

Number

This measure is reported only for each rule action type - i.e., only if the IOT Filter Name parameter is set to 'ActionType'.

Ideally, the value of this measure should be high.

Failed rule action invocations

Indicates the number of failed invocations of this rule action type.

Number

This measure is reported only for each rule action type - i.e., only if the IOT Filter Name parameter is set to 'ActionType'.

Ideally, the value of this measure should be 0.