Regularly optimizing indexes for performance

What is optimization? 

While an index is being used, files are removed from the document stores and new files are added, making it necessary to constantly write to the index files. When files are removed, the existing index entries that refer to those files are marked as unused, and these unused entries can easily add up over time, causing the index to grow.

In a system where files are only added to a large index over time, you will typically see very little fragmentation, whereas, in indexes where files come and go, fragmentation is more frequent.

The index is held in memory and when a user searches, Index Manager searches through the entire index to find matches. With a great number of empty index entries, the server can gradually become slower at processing searches.

To this end, the system administrator can set a global optimization schedule in the Serice options in Index Manager settings at which point the index server will be temporarily shut down, unused entries removed from the index ("defragmentation"), and the service started back up. This can be scheduled to happen at a time when there's minimal activity on the system.

Setting an optimization schedule

1. In Index Manager Settings, open Indexes tab.
2. Select Automatic optimization.

Unused entries threshold: Specify the number of index entries that must be marked as unused before the optimization runs. Keep in mind that an index may well be fragmented even if there are few unused entries in the index. (See point 1 above.) Therefore, if you would like the index to optimize every day, you should set the threshold to zero and set the time as described below.

Time to run optimize: Set the time to run the optimization here. Choose which days you should optimize and the time at which optimization should occur. Because indexes become unavailable while optimization runs, it's generally a good idea to optimize outside office hours.

Rescan before optimize: It's usually always a good idea to enable this option as it forces Index Manager to scan through all the attached document folders and locate any files that may not have been added to the index. This way you can be sure that all files in the document stores are readily available for searches.

Tip: The Index Manager Status screen in the Operations Center shows the number of unused entries in an index: select an index "blob" and read out the detailed information in the column on the right.

Manual optimization

1. You can also manually trigger an index optimization: Go to the Status page and select the index you would like to optimize.
2. Select Optimize.

Tip: Index Manager's cache is also important to maintain a responsive system.  For more information, see Choosing a cache folder for thumbnails and previews.