Indexer Test
Indexers manage the content to be included in an index, and propagate that content to query servers where they are stored in index files.
Each item that a connector submits for indexing passes through the indexing pipeline, in which a predefined set of modules manipulates the items one by one. At the end of this, the FIXML needed for indexing is produced and handed over to the indexer by the Indexing Dispatcher. The Indexer receives the item and creates binary indexes in the data folder on disk. These binary indexes are then sent to the query matching component and are used during query evaluation during searching.
If too many feeding operations/documents are sent to an indexer for processing, that indexer may be forced to push many of these operations into the API queue, where they will be held until they are taken up for processing. Administrators need to monitor the length of this queue at frequent intervals to determine the processing power of an indexer. Using the Indexer test, administrators can keep an eye on the number and nature of documents that each indexer processes, track the growth of the API queues, monitor the FiXML fill rate of every indexer, and thus promptly capture potential overloads, processing delays, processing failures, and other inefficiencies in the functioning of every indexer.
Target of the test : A FAST Search Server 2010 for SharePoint
Agent deploying the test : An internal agent
Outputs of the test : One set of results for every indexer configured on the FAST Search Server 2010 for SharePoint farm.
| Parameter | Description |
|---|---|
|
Test period |
How often should the test be executed |
|
Host |
The host for which the test is to be configured. |
|
Port |
Refers to the port used by the specified host. By default, this is 13280. |
| Measurement | Description | Measurement Unit | Interpretation |
|---|---|---|---|
|
Total number of feed operation |
Indicates the total number of feed operations processed by this indexer. |
Number |
The measure indicates the total number of items that are indexed and updated. |
|
API operations processed |
Indicates the rate at which the feed operations are processed by this indexer. |
Operations/Sec |
A consistent decrease in the value of this measure could indicate a processing bottleneck. |
|
API partial update operations |
Indicates the number of feed operations processed by this indexer that are partial update operations. |
Number |
|
|
API queue load |
Indicates the current load on the API queue of this indexer. |
Bytes |
The API queue is used by the indexer to hold pending operations waiting to be persisted. The value of this measure is obtained by the ratio of the current size of the API queue to the maximum size of the API queue. A low value is desired for this measure. |
|
API queue size |
Indicates the number of batches that are currently in the API queue of this indexer. |
Number |
Indexers queue incoming work under high load. This is common, especially for partial updates. If API queues never (intermittently) reach zero, the indexer is the bottleneck. The crawler will pause when the queue reaches 256MB in one of the indexers. This can occur if the storage subsystem is not sufficiently powerful. It will also occur during large redistribution of content between partitions, which temporarily blocks more content from being indexed. |
|
API remove operations |
Indicates the number of feed operations processed by this indexer that are remove operations. |
Number |
|
|
API status update operations |
Indicates the number of feed operations processed by this indexer that are status updates. |
Number |
|
|
API remove collection operations |
Indicates the number of feed operations processed by this indexer that are remove collection operations. |
Number |
|
|
Documents in indexer |
Indicates the total number of valid items that are currently contained in the docindex of this indexer. |
Number |
The docindex is a mapping to the persisted items on disk, so this translates to the number of valid items persisted on disk (FiXML). |
|
API update operations |
Indicates the number of feed operations processed by this indexer that are add/update operations. |
Number |
|
|
API failed operations |
Indicates the number of feed operations that were not successfully processed by this indexer. |
Number |
|
|
Fixml fill rate |
Indicates the ratio of the number of valid items in a persisted file to the total number of persisted items. |
Percent |
FiXML files (internal item storage in the indexers) are compacted regularly, by default between 3am and 5am every night. Low FiXML fill rate (<70%) will lead to inefficient operation. |
|
Last batch size |
Indicates the size of the previous batch that was processed by this indexer. |
Bytes |
|
|
Maximum batch size |
Indicates the size of the largest batch that was processed by this indexer during the last measurement period. |
Bytes |
|
|
Maximum documents per sec |
Indicates the maximum number of items that were indexed by this indexer in an index job per second. |
Indexed/Sec |
By closely observing variations to these measures over time, you can proactively detect potential processing latencies. |
|
Minimum documents per sec |
Indicates the minimum number of items that were indexed in an index job per second. |
Number |
|
|
Number of file receivers currently registered |
Indicates the total number of file receivers that are currently registered on this indexer. |
Number |
The file receivers can either be backup indexers or search controllers. |
|
Number of search controllers currently registered |
Indicates the total number of search controllers that are currently registered on this indexer. |
Number |
The registered search controllers get new indices from over the indexer. |
|
Search controller stubs currently allocated by the indexer |
Indicates the total number of search controller stubs that are currently allocated by this indexer. |
Number |
The search controller stub is used to communicate with a remote search controller. This measure is a good indicator of the number of search controllers that are communicating with the indexer. |
|
Search controller allocated count |
Indicates the number of times a search controller has been allocated on this indexer. |
Number |
|
|
Size of documents |
Indicates the size of all the valid items that are persisted on the disk. |
Bytes |
|
