Redis Long Running Client Test

If a client's connection stays alive for too long, it could imply that the connection is idle. It could also mean that the server is taking too long to process the client's queries, forcing the client to wait long for output. By capturing long running clients and by scrutinizing them, administrators can figure out what is causing the client's connection to run as long as it is running - is it because of a bug in the client software? or is it because the server is slow in processing client's commands?

The Redis Long Running Client test monitors the top-20 client connections (by default) in terms of their age - i.e., how long the connection has been alive. For each long running client connection so captured, the test then measures the query and output buffer memory usage, the length of the output list, and the idle time of each connection to help you diagnose the root-cause of the long running time of that connection.

Target of the test : A Redis server

Agent deploying the test : An internal agent (recommended)

Outputs of the test : One set of results for each long running client

Configurable parameters for the test
Parameters Description

Test period

How often should the test be executed

Host.

The host for which the test is to be configured.

Port

The port at which the specified HOST listens.

Redis Password and Confirm Password

In some high security environments, a password may have been set for the Redis server, so as to protect it from unauthorized accesses/abuse. If such a password has been set for the monitored Redis server, then specify that password against REDIS PASSWORD. Then, confirm the password by retyping it against CONFIRM PASSWORD.

If the Redis server is not password protected, then do not disturb the default setting of this parameter.

To determine whether/not the target Redis server is password-protected, do the following:

  • Login to the system hosting the Redis server.

  • Open the redis.conf file in the <REDIS_INSTALL_DIR>.

  • Look for the requirepass parameter in the file.

  • If this parameter exists, and is not preceded by a # (hash) symbol, it means that password protection is enabled for the Redis server. In this case, the string that follows the requirepass parameter is the password of the Redis server. For instance, say that the requirepass specification reads as follows:

    requirepass red1spr0

    According to this specification, the Redis server is protected using the password red1spr0. In this case therefore, you need to specify red1spr0 against REDIS PASSWORD.

  • On the other hand, if the requirepass parameter is prefixed by the # (hash) symbol as shown below, it means password protection is disabled.

    # requirepass red1spr0

    In this case, leave the REDIS PASSWORD parameter with its default setting.

Max Clients

By default, this parameter is set to 20. This means that, by default, this test will monitor the top-20 client connections alone, sorted in the descending ot their age - i.e., connection duration. If required, you can override this default setting by changing the value of this parameter. For instance, to view the top-50 connections in terms of their age, change the value here to 50.

Max Client Age

By default, this test counts client connections that have been running for over 1200 seconds as 'long running' client connections, and monitors them. If you want more or less number of client connections to be included in the scope of this test, you can change the value of this parameter.

Measurements made by the test
Measurement Description Measurement Unit Interpretation

Client ID

Indicates the client ID

 

 

IP address

Reports the IP address of this client.

 

 

Port

Indicates the port at which this client listens.

Number

 

Socket file descriptor entry

Reports the socket file descriptor number of this client.

Number

 

Database ID

Indicates the current database ID of this client.

Number

 

Age

Indicates the total duration of this client's connection.

Seconds

 

Idle time

Indicates the duration for which this client's connection was idle.

Seconds

By default recent versions of Redis don't close the connection with the client if the client is idle for many seconds: the connection will remain open forever.

However, there are two conditions when it makes sense to set a timeout:

  • Mission critical applications where a bug in the client software may saturate the Redis server with idle connections, causing service disruption.

  • As a debugging mechanism in order to be able to connect with the server if a bug in the client software saturates the server with idle connections, making it impossible to interact with the server.

You can configure a timeout value via redis.conf or simply using CONFIG SET timeout <value>.

Channel subscriptions

Indicates the number of channels to which this client subscribes.

Number

Redis Pub/Sub implements the messaging system where the senders (in redis terminology called publishers) sends the messages while the receivers (subscribers) receive them. The link by which the messages are transferred is called channel. In Redis, a client can subscribe any number of channels.

Pattern subscriptions

Indicates the number of glob-style patterns this client subscribes to.

Number

The Redis Pub/Sub implementation supports pattern matching. Clients may subscribe to glob-style patterns in order to receive all the messages sent to channel names matching a given pattern.

Query buffer length

Indicates the length of the query buffer (in MB) of this client.

MB

If this measure reports the value 0, it means no queries are pending.

Ideally, the value of this measure should be low. A high value or a consistent increase in this value could indicate the queries from the client are not being processed fast enough by the server.

If the value of this measure reaches the hard limit of 1 GB, the client connection may close automatically. This is done to avoid a server crash in case of client or server software bugs.

Query buffer free

Indicates the amount of free space in the query buffer of this client.

MB

If this measure reports the value 0, it means that there is no free space in the query buffer. In this case, the client connection may close automatically. This is done to avoid a server crash in case of client or server software bugs.

Ideally, a high value is desired for this measure. A steady drop in the value of this measure could imply that the query buffer is rapidly filling up. This could be because of a probable query processing bottleneck on the server.

Output buffer length

Indicates the length (in MB) of the output buffer of this client.

MB

Redis needs to handle a variable-length output buffer for every client, since a command can produce a big amount of data that needs to be transferred to the client.

Output list length

Indicates the number of responses enqueued in the output list of this client.

Number

If the output buffer is not correctly sized, then it can fill up quickly, causing subsequent responses to clients to be enqueued in an output list.

If the value of this measure is abnormally high for a client, it means that the output list is very long; this in turn is a sign that the output buffer needs resizing.

Output buffer memory

Indicates the amount of output buffer memory used up by the responses to this client.

MB

Sometimes, a client may send more commands producing more output to serve at a faster rate at which Redis can send the existing output to the client. This is especially true with Pub/Sub clients in case a client is not able to process new messages fast enough.

Both the conditions will cause the client output buffer to grow and consume more and more memory. In such a cases, this measure will report an abnormally high value.

To limit the usage of output buffer memory, by default, Redis sets limits to the output buffer size for different kind of clients. When the limit is reached the client connection is closed and the event logged in the Redis log file.

There are two kind of limits Redis uses:

  • The hard limit is a fixed limit that when reached will make Redis closing the client connection as soon as possible.

  • The soft limit instead is a limit that depends on the time, for instance a soft limit of 32 megabytes per 10 seconds means that if the client has an output buffer bigger than 32 megabytes for, continuously, 10 seconds, the connection gets closed.

Different kind of clients have different default limits:

  • Normal clients have a default limit of 0, that means, no limit at all, because most normal clients use blocking implementations sending a single command and waiting for the reply to be completely read before sending the next command, so it is always not desirable to close the connection in case of a normal client.

  • Pub/Sub clients have a default hard limit of 32 megabytes and a soft limit of 8 megabytes per 60 seconds.

  • Slaves have a default hard limit of 256 megabytes and a soft limit of 64 megabyte per 60 second.

It is possible to change the limit at runtime using the CONFIG SET command or in a permanent way using the Redis configuration file redis.conf.