Elasticsearch Index Performance Test

An index is a collection of documents that have somewhat similar characteristics. For example, you can have an index for customer data, another index for a product catalog, and yet another index for order data. An index is identified by a name (that must be all lowercase) and this name is used to refer to the index when performing indexing, search, update, and delete operations against the documents in it. In a single Elasticsearch cluster, administrators can define multiple indexes. A healthy index ensures that search queries are serviced quickly. If the index becomes unhealthy or if refresh/flush operations become infrequent/slow, search efficiency will be adversely impacted. As a result, performance of the Elasticsearch cluster will be degraded and the user experience will be affected. To avoid such anomalies, administrators should track the health of indexes and index-related operations (eg., refresh, flush, merge) at regular intervals. To achieve this, administrators can use the Elasticsearch Index Performance test.

This test auto-discovers the indexes in the Elasticsearch cluster, and reports the current health and change in the size of each index. In addition, this test also reveals the count of refresh/merge/flush operations performed in each index and the time taken for performing these operations. With the help of these metrics, administrators can proactively detect potential slowness in index-related operations, and promptly initiate preventive measures. Besides, this test sheds light on the number of documents added to/deleted from each index, using which the administrators can easily track changes to index size. The count of shards in different states is reported, and in the process, unassigned shards (if any) are also captured.

Target of the test : An Elasticsearch Cluster

Agent deploying the test : An internal/remote agent

Outputs of the test : One set of results for each index in the target Elasticsearch cluster.

This test also reports metrics for the Summary descriptor.

Configurable parameters for the test
Parameter	Description
Test Period	How often should the test be executed.
Host	The IP address of the host for which this test is to be configured.
Port	The port number at which the target host being monitored listens. By default, this is set to 9200.
Cloud Instance URL	By default, the is set to none. If the target Elasticsearch cluster is hosted on the cloud environment, then you need to specify the URL of the cluster on cloud against this parameter. For example: 64bd966328067fd89e0c9b4c3bb8b042.us-east-1.aws.found.io. When the cloud URL is specified, the eG agent will use the cloud URL to monitor the target cluster rather than using the host specified in the Host text box.
Elastic Search User and Elastic Search Password	By default, the Elastic Search User and Elastic Search Password parameters are set to none indicating that the eG agent doesn't require authentication to collect metrics from the Elasticsearch cluster. If authentication is required to access the target Elasticsearch cluster, then specify the valid credentials against these parameters.
Confirm Password	Confirm the Elastic Search Password by retyping it in the Confirm Password text box.
SSL	By default, the SSL flag is set to No. If the Elasticsearch cluster is SSL-enabled by default or hosted on the cluster, then set this flag to Yes. This indicates that the eG agent will communicate with the target cluster via HTTPS by default.

Measurements made by the test

Measurement

Description

Measurement Unit

Interpretation

Index Health

Indicates the current health of this index.

This measure reveals the health of the index at shard level. The numeric values that correspond to the measure values mentioned above are as follows:

Measure Value	Numeric Value	Description
Red	0	Indicates that the specific shard is not allocated to any node in the cluster.
Yellow	1	Indicates that the primary shard is allocated but replicas are not assigned to any node.
Green	2	Indicates that all shards in the cluster are allocated to the nodes.

Note:

This test typically reports the Measure Values listed in the table above to indicate the current health of each index. However, the graph of this measure is represented using the numeric equivalents only.

This measure is not applicable for Summary descriptor.

Indexing Rate

Indicates the rate at which the documents were indexed in this index.

For Summary descriptor, this measure indicates the rate at which the documents were indexed in the indexes of the cluster.

Documents/sec

A gradual/sudden increase in the value of this measure indicates that the index size growing as well. In such cases, administrators may need to allocate more space to the index.

Recently added documents

Indicates the number of documents that were newly added to this index during last measurement period.

For Summary descriptor, this measure indicates the number of documents that were newly added to the indexes of the cluster.

Number

Total indexed documents

Indicates the total number of documents that have been added to this index since this index was created.

For Summary descriptor, this measure indicates the number of documents that are added to the indexes of the cluster.

Number

Recently deleted documents

Indicates the number of documents that were deleted from this index during last measurement period.

For Summary descriptor, this measure indicates the number of documents that were deleted from the indexes.

Total deleted documents

Indicates the total number of documents that are deleted from this index since the index was created.

For Summary descriptor, this measure indicates the total number of documents that are deleted from the indexes.

Number

Index refreshes

Indicates the number of refresh operations performed for this index.

For Summary descriptor, this measure indicates the total number of refresh operations performed for the indexes in the cluster.

Number

An Elasticsearch refresh makes the documents available for search whenever new documents are added to the index. The refresh operation is performed at regular intervals. A very low value of this measure denotes that the newly added documents may not be available for the search.

Time spent on index refreshes

Indicates the time taken for performing refresh operations for this index.

For Summary descriptor, this measure indicates the time taken for performing refresh operations in the cluster.

Seconds

Compare the value of this measure across the indexes to identify the index that took maximum time to refresh.

Average time spent per refresh

Indicates the average time taken for performing a single refresh operation for this index.

For Summary descriptor, this measure indicates the time taken for performing a single refresh operation in the cluster.

Seconds

A low value is desired for this measure.

Index merges

Indicates the number of merge operations performed in this index.

For Summary descriptor, this measure indicates the number of merge operations performed in the cluster.

Number

A shard in the index is broken down into segments. Segments are internal storage elements in the index where the index data is stored, and are immutable. Smaller segments are periodically merged into larger segments to keep the index size at bay and to expunge deletes.

Time spent on index merges

Indicates the average time taken for performing the merge operations in this index.

For Summary descriptor, this measure indicates the average time taken for performing the merge operations in the cluster.

Seconds

Comparing the value of this measure across the indexes will reveal the index that took maximum time to perform merge operations.

Average time spent per merge

Indicates the time taken for performing a single merge operation in this index.

For Summary descriptor, this measure indicates the time taken for performing a single merge operation in the cluster

Seconds

A low value is desired for this measure.

Index flushes

Indicates the number of flush operations performed in this index.

For Summary descriptor, this measure indicates the number of flush operations performed in the cluster.

Number

The flush process is performed to ensure that any data that is currently only persisted in the transaction log is also permanently persisted in Lucene search library. This reduces recovery times as that data does not need to be reindexed from the transaction logs after the Lucene index is opened. By default, Elasticsearch uses heuristics in order to automatically trigger flushes as required. A very low value of this measure may indicate the increase in recovery time of the index.

Time spent on index flushes

Indicates the total time taken to perform flush operations in this index.

For Summary descriptor, this measure indicates the time taken for performing a single flush operation in the cluster.

Seconds

Average time spent per flush

Indicates the time taken to perform a single flush operation in this index.

For Summary descriptor, this measure indicates the time taken for performing a single flush operation in the cluster.

Seconds

Primary shards

Indicates the number of primary shards in this index.

Number