Vault Index Exceptions Possibly Causing 100% CPU Utilization

[ewrankin]

We started having 100% CPU utilization on our server and the only thing sticking out is the following exception being thrown from the Vault Index Service (pulled from the vaultindexservice.log):

5/7/2013 10:42:10 AM <generic>: During a work request UPDATE, the Repository.Update or Repository.Serialize calls threw an exception. [VaultClientOperationsLib.PluginException_CacheFileList: The file list cache file is is incorrect for the this folder ($/TestPgm). Cache File: C:\Windows\Temp\SGVaultIndex\cache\98E45A17-32A1-46AB-88B4-A6EFD4647238\admin\S\W\SWHEY259.D`9 Cache File Version: 162784, but expected Version: 166397
at VaultClientOperationsLib.PluginFolder.LoadFileListIfOld(String strAbsFileListCacheFile)
at VaultClientOperationsLib.PluginFolder.BeforeUpdateFolderDelta(VaultFolderDelta fd)
at VaultClientOperationsLib.VaultClientFolder.Update(VaultFolderDelta fd, String deltaFolderPath, ArrayList deletedObjects, ArrayList addedObjects, TreeCache treeCache, Object objItemData)
at VaultClientOperationsLib.VaultClientFolder.Update(VaultFolderDelta fd, String deltaFolderPath, ArrayList deletedObjects, ArrayList addedObjects, TreeCache treeCache, Object objItemData)
at VaultClientOperationsLib.PluginRepository.Update(VaultRepositoryDelta delta, Boolean bClearFileListCache)
at VaultIndexService.IndexServiceProcessWorkerThread.HandleRequest_UpdateIndex_Initialize(VaultIndexManager vim, PluginRepositoryConnection repConn, IndexServiceUpdateIndexData updtIndexData)] The file list cache file is is incorrect for the this folder ($/TestPgm). Cache File: C:\Windows\Temp\SGVaultIndex\cache\98E45A17-32A1-46AB-88B4-A6EFD4647238\admin\S\W\SWHEY259.D`9 Cache File Version: 162784, but expected Version: 166397
at VaultClientOperationsLib.PluginFolder.LoadFileListIfOld(String strAbsFileListCacheFile)
at VaultClientOperationsLib.PluginFolder.BeforeUpdateFolderDelta(VaultFolderDelta fd)
at VaultClientOperationsLib.VaultClientFolder.Update(VaultFolderDelta fd, String deltaFolderPath, ArrayList deletedObjects, ArrayList addedObjects, TreeCache treeCache, Object objItemData)
at VaultClientOperationsLib.VaultClientFolder.Update(VaultFolderDelta fd, String deltaFolderPath, ArrayList deletedObjects, ArrayList addedObjects, TreeCache treeCache, Object objItemData)
at VaultClientOperationsLib.PluginRepository.Update(VaultRepositoryDelta delta, Boolean bClearFileListCache)
at VaultIndexService.IndexServiceProcessWorkerThread.HandleRequest_UpdateIndex_Initialize(VaultIndexManager vim, PluginRepositoryConnection repConn, IndexServiceUpdateIndexData updtIndexData)

I presume this could be corrected via clearing the cache but is there a procedure to do that that doesn't mess up the index. I would like not impact our users since a re-index takes a day plus.

[Beth]

We haven't tested what would happen if you cleared the cache for Find in Files.

If you wanted to try clearing the cache first, then to make Vault create a new cache, one of your users needs to check in a file.

[ewrankin]

I went ahead over the weekend and just stopped the indexer, purged the data, and rebuilt it and the error went away. One thing that is showing up in the log is that whenever a user deletes an item we see these messages:

5/13/2013 10:34:17 AM <generic>: Upon processing changes in $/TestPgm/, the index server detected a corruption in the client cache file "". The Find in Files Index server will attempt to refresh index data from the server, and some files may now be out of sync. You may need to delete and purge the index, and then turn Find in Files indexing back on for repository Development to search on that missing data.
5/14/2013 7:46:53 AM <generic>: Upon processing changes in $/TestPgm, the index server detected a corruption in the client cache file "". The Find in Files Index server will attempt to refresh index data from the server, and some files may now be out of sync. You may need to delete and purge the index, and then turn Find in Files indexing back on for repository Development to search on that missing data.
5/14/2013 7:46:58 AM <generic>: PluginFolder::AfterUpdateFolderDelta - The client cache file is missing for $/TestPgm/(C:\Windows\Temp\SGVaultIndex\cache\98E45A17-32A1-46AB-88B4-A6EFD4647238\admin\O\V\OV_IV10N.61A), and the file list could not be refreshed from the server. The Index for any files in this folder will not be found. You may need to delete the index (removing all data), and then turn on Find in Files indexing for repositoryDevelopment.

Is it really necessary to rebuild the index just because they deleted something? That takes 12 hours for our repository!

[ewrankin]

Also now this is popping as well:
5/14/2013 8:57:40 AM <generic>: Failed to delete indexing data for repository 3. [System.Data.SqlClient.SqlException (0x80131904): Violation of PRIMARY KEY constraint 'PK__#t_del_lines______14FDBC8C'. Cannot insert duplicate key in object 'dbo.#t_del_lines'.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultIndexService.VaultIndexServiceSQL.DelIndexDataByObjVerIDs(Object oTxContext, Int32 nRepID, List`1 listObjVerIDs)] Violation of PRIMARY KEY constraint 'PK__#t_del_lines______14FDBC8C'. Cannot insert duplicate key in object 'dbo.#t_del_lines'.
The statement has been terminated.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning()
at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString)
at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async)
at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result)
at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(DbAsyncResult result, String methodName, Boolean sendToPipe)
at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()
at VaultIndexService.VaultIndexServiceSQL.DelIndexDataByObjVerIDs(Object oTxContext, Int32 nRepID, List`1 listObjVerIDs)

[Beth]

At this point I'd suggest turning off the Find in Files and purging the index. Then turn on the Find in Files again and allow the index to be rebuilt.

[ewrankin]

At this point I'd suggest turning off the Find in Files and purging the index. Then turn on the Find in Files again and allow the index to be rebuilt.

Again? My last two posts were logs from right after I had already did that.

[Beth]

Ok. Instead of indexing again, could you send an email to support at sourcegear.com (attn: Beth) with a link to this forum thread?