Vault 10 client hangs indefinitely

[craighann]

Hi,

I've recently upgraded 10 users from Vault 7 to Vault 10. The upgrade appeared to go smoothly, but we're having issues with clients hanging when accessing our repository. The symptoms are the same as the other threads on this topic, but I've tried the remedial and troubleshooting suggestions in the thread viewtopic.php?f=13&t=22876&p=78454&hili ... ver#p78454

I've also followed the processes in this forum thread too - viewtopic.php?f=13&t=22633

The only thing that seems to solve it is deleting the contents of C:\Users\XXXX\AppData\Local\SourceGear\Vault_1\Client. Obviously this then means that our client caches are not in sync with the server. The cache folders aren't very big as we've deleted them to be able to keep working... mine for example currently stands at 2.6Gb.

I'd appreciate some help - Because of this issue it is causing my team to lose productivity. We never had this problem with Vault 7.

We are running on the following platform -

Server : Windows 10 Pro 64 bit, 32GB RAM, i7-4770K, 526GB free space on 4Tb data drive
SQL : SQL Server 12 Standard 64 bit
Clients : Various Windows 10 Pro clients.

Regards,
Craig.

HS: 248900

[craighann]

An additional point - one of my team has discovered that deleting just the CacheMember_Repository file from the client cache allows them to keep working without losing the synchronisation information.

I hope this helps towards identifying a solution and issuing a definitive fix.

Craig

[jclausius]

Craig,

Some other suggestions if you encounter this on other machines - viewtopic.php?f=13&t=22633

[craighann]

Thanks for the reply, Jeff.. to tackle the suggestions from the other thread...

1) Turn off Request Database Delta on Repository Cache Miss. DONE ON ALL CLIENTS

Go to Vault Tools - Options - Network Settings, and uncheck the option Request Database Delta on Repository Cache Miss. Click OK. DONE ON ALL CLIENTS

2) Change the order of the preferred runtime for the Vault client. DOESN'T SEEM TO MAKE ANY DIFFERENCE

3) Check for excessive database log files. By default SQL Server will keep a running transaction log for the database. This log can grow to excessive sizes, thus affecting performance if they are not truncated by running a backup or not using 'Simple' recovery mode. If you need assistance, contact SourceGear Technical Support or search for 'SQL Server Transaction Log' for additional information. LOGS SET TO SIMPLE - DB WAS BACKED UP BEFORE UPGRADE TO V10 AT THE START OF THE MONTH. SG_VAULT LOG IS 60GB HOWEVER

4) Even though Vault doesn't directly touch the network (it goes through IIS and .NET), it has been reported that some network configurations or equipment are not yet ready for IPv6. Disabling IPv6 on the server has shown performance improvements for some users. SERVER ALREADY HAS IPV6 DISABLED

5) Sometimes the notification database can grow excessive if there is an incorrectly configured notification server. Check for excessive rows in sgnotify and remove them. ROW COUNT IS 0

6) Sometimes the issue is overall performance. Performance issues can remedied with the instructions from the following KB articles. The first two listed are critical.

SQL database maintenance: http://support.sourcegear.com/viewtopic.php?t=2924
SQL statistics setting change: http://support.sourcegear.com/viewtopic ... 13&t=22358
Vault Performance: http://support.sourcegear.com/viewtopic.php?t=4206

7) The panel docking information may be causing the tree to not show. Close Vault. Then browse to your Client Side Cache. Delete the file vaultgui_dockpanel.config, and then reopen Vault. CLIENT IS LOCKED UP AND CPU USAGE PEAKS, UI IS UNRESPONSIVE. DELETING FILE MAKES NO DIFFERENCE

8_) Ghosting can cause a performance hit, especially if either a remote location is ghosted or a very large area. The ghosting options are in the Vault Tools - Options - General. Turn off ghosting to see if performance is improved. 70GB GHOST FOLDER APPEARS TO BE OK

9) Folder Security can cause a performance hit if there are many permission settings for various folders. The way to check this is to go to the Vault Admin web page, expand the list of repositories on the left, click on each repository name and uncheck Folder Security. Restart the client before testing. NO FOLDER SECURITY USED

10) CVS mode causes a lot of additional scanning to catch when changes are happening which can slow things down depending on how much is being scanned. Go to Tools - Options - Concurrent Development Style and click the VSS mode button, and don't change any of the settings below the buttons except for Exclusive Locks. You can choose your preference for Exclusive Locks. After this change, perform a Get to change the writable settings on the files, and then check out files you need to edit. CVS NOT USED

[jclausius]

In regards to the database log file, it could be the log file isn't using that much space and it can be reclaimed. If you look at the properties of the Vault related databases in SQL Server Management studio, or check usage:

Ex:
USE sgvault
GO
sp_spaceused
GO

You can see if there's space in the database that the files are not using.

----

In regards to the actual hang, if you checked those things let's see if something shows up in the client side logging. Let's start with a simple one. Start a Vault client, as soon as the UI starts, hit "Control Alt Shift F12" (See viewtopic.php?t=1534) to enable logging. Hopefully this generates some information in a log file. Send that to 'support AT sourcegear DOT com', and hopefully something there can help point to something concrete.

[craighann]

Hi Jeff,

Thanks again for the prompt reply. The results of the SQL query are as follows..



I've asked all my team to enable logging in the VaultGUIClient.exe.config file, so will get back to you next time someone has the issue.

Regards,
Craig

[jclausius]

In regards to SQL Server log file, I'm uncertain the current state of the database. You may want to search for SQL Server log file won't truncate or something like that.

In regards to the other stuff you can come back here or also mail : support AT sourcegear DOT com.

[craighann]

Hi Jeff,

I've just sent you a complete log from one of my colleagues who has had the issue this morning. Below is a truncated (without folder names) section towards the end of the log where it appears to hang the client.

Regards,
Craig.

Code: Select all

27/06/2017 12:02:59 <generic>: [Main:1] Emergency logging enabled in Windows Forms client
27/06/2017 12:03:02 <busy>: [GUIClientWorkerThread:4] Setting GUI to busy from ConnectToServer
27/06/2017 12:03:04 <busy>: [GUIClientWorkerThread:4] Setting GUI to unbusy from ConnectToServer.
27/06/2017 12:03:06 <busy>: [GUIClientWorkerThread:4] Setting GUI to busy from Options
27/06/2017 12:03:11 <busy>: [GUIClientWorkerThread:4] Setting GUI to unbusy from Options.
27/06/2017 12:03:13 <busy>: [GUIClientWorkerThread:4] Setting GUI to busy from ConnectToServer
27/06/2017 12:03:18 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ConnectionStateChangedEvent
27/06/2017 12:03:18 <statussearch>: [Main:1] Connection state changed from Unconnected to Connecting
27/06/2017 12:03:18 <connection>: [GUIClientWorkerThread:4] Login started.
27/06/2017 12:03:18 <pluginnotify>: [Main:1] Connection state changed from Unconnected to Connecting
27/06/2017 12:03:19 <crypto>: [GUIClientWorkerThread:4] Took 1 tries to encrypt the password
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] Login finished.
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.TreeStructureChangedEvent
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ChangeSetItemsChangedEvent
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ConnectionStateChangedEvent
27/06/2017 12:03:19 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:19 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:19 <statussearch>: [Main:1] Connection state changed from Connecting to Connected
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetOptions started.
27/06/2017 12:03:19 <pluginnotify>: [Main:1] Connection state changed from Connecting to Connected
27/06/2017 12:03:19 <connection>: [<No Name>:6] AuthenticateWithNotificationServer started.
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetOptions finished.
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.UserOptionsLoadedEvent
27/06/2017 12:03:19 <connection>: [<No Name>:6] AuthenticateWithNotificationServer finished.
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetHistoryFavorites started.
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetHistoryFavorites finished.
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.MessageEvent
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetServerVersionInfo started.
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetServerVersionInfo finished.
27/06/2017 12:03:19 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.MessageEvent
27/06/2017 12:03:19 <busy>: [GUIClientWorkerThread:4] Setting GUI to unbusy from ConnectToServer.
27/06/2017 12:03:19 <repositoryexplorer>: [Main:1] refresh requested updateKnown is False
27/06/2017 12:03:19 <busy>: [GUIClientWorkerThread:4] Setting GUI to busy from ChooseRepository
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetRepositories started.
27/06/2017 12:03:19 <connection>: [GUIClientWorkerThread:4] GetRepositories finished.
27/06/2017 12:03:19 <connection>: [<No Name>:5] AuthenticateWithNotificationServer started.
27/06/2017 12:03:19 <connection>: [<No Name>:5] AuthenticateWithNotificationServer finished.
27/06/2017 12:03:19 <eventengine>: [<No Name>:6] Event fired: VaultClientOperationsLib.MessageEvent
27/06/2017 12:03:19 <eventengine>: [<No Name>:6] Event fired: VaultClientOperationsLib.MessageEvent
27/06/2017 12:03:20 <generic>: [GUIClientWorkerThread:4] SetActiveRepositoryID(id:3)
27/06/2017 12:03:20 <mutex>: [GUIClientWorkerThread:4] Created mutex 2168 (Vault_CacheMember_Repository_1947225294)
27/06/2017 12:03:20 <mutex>: [GUIClientWorkerThread:4] Took mutex 2168
27/06/2017 12:03:21 <refresh>: [GUIClientWorkerThread:4] Deserialized CacheMember_Repository
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 2168
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 2212 (Vault_CacheMember_WorkingFolderAssignments_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 2212
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 2212
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1832 (Vault_CacheMember_SelectionString_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1832
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1832
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1424 (Vault_CacheMember_CheckOutList_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1424
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1424
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1416 (Vault_CacheMember_ChangeSetItems_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1416
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1416
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 2196 (Vault_CacheMember_PendingServerNamespaceChanges_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 2196
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 2196
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 2196
27/06/2017 12:03:21 <cache>: [GUIClientWorkerThread:4] Did not UpdateObjectFromDisk in VaultClientOperationsLib.CacheMember_PendingServerNamespaceChanges
27/06/2017 12:03:21 <cache>: [GUIClientWorkerThread:4] Cache file LastWriteTime:27/06/2017 11:54:43  _lastSyncTime:27/06/2017 11:54:43
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 2196
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1828 (Vault_CacheMember_ChangeSetComment_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1828
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1828
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1656 (Vault_CacheMember_CommitDialogBugInfo_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1656
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1656
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1688 (Vault_CacheMember_ExpandedNodes_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1688
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1688
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 2184 (Vault_CacheMember_PendingLocalDeletions_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 2184
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 2184
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1660 (Vault_CacheMember_LastStructureGetTime_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1660
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1660
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Created mutex 1692 (Vault_CacheMember_KnownChanges_1947225294)
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Took mutex 1692
27/06/2017 12:03:21 <mutex>: [GUIClientWorkerThread:4] Released mutex 1692

Followed by lots of

Code: Select all

GetRepositoryStructure

and

Code: Select all

GetCheckOutListChanges

calls which reference our users and projects....

Code: Select all

27/06/2017 12:03:31 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ChangeSetItemsChangedEvent
27/06/2017 12:03:31 <refresh>: [GUIClientWorkerThread:4] Calling NotifyTreeStructureChanged
27/06/2017 12:03:31 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.TreeStructureChangedEvent
27/06/2017 12:03:31 <refresh>: [GUIClientWorkerThread:4] Refresh finished
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] GetUserEmailOptions started.
27/06/2017 12:03:31 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:31 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] GetUserEmailOptions finished.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] ListWatchers started.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] ListWatchers finished.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] ListCloakedObjects started.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] ListCloakedObjects finished.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] GetRepositoryOptions started.
27/06/2017 12:03:31 <connection>: [GUIClientWorkerThread:4] GetRepositoryOptions finished.
27/06/2017 12:03:31 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ChangeSetItemsChangedEvent
27/06/2017 12:03:31 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:31 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ActiveRepositoryChangedEvent
27/06/2017 12:03:31 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.TreeStructureChangedEvent
27/06/2017 12:03:31 <busy>: [GUIClientWorkerThread:4] Setting GUI to unbusy from ChooseRepository.
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Took mutex 1416
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Did not UpdateObjectFromDisk in VaultClientOperationsLib.CacheMember_ChangeSetItems
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Cache file LastWriteTime:26/06/2017 16:34:21  _lastSyncTime:26/06/2017 16:34:21
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Released mutex 1416
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Took mutex 1828
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Did not UpdateObjectFromDisk in VaultClientOperationsLib.CacheMember_ChangeSetComment
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Cache file LastWriteTime:26/06/2017 16:34:21  _lastSyncTime:26/06/2017 16:34:21
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Released mutex 1828
27/06/2017 12:03:32 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.LocalRefreshEvent
27/06/2017 12:03:32 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ChangeSetItemsChangedEvent
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Took mutex 1416
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Did not UpdateObjectFromDisk in VaultClientOperationsLib.CacheMember_ChangeSetItems
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Cache file LastWriteTime:26/06/2017 16:34:21  _lastSyncTime:26/06/2017 16:34:21
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Released mutex 1416
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Took mutex 1828
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Did not UpdateObjectFromDisk in VaultClientOperationsLib.CacheMember_ChangeSetComment
27/06/2017 12:03:32 <cache>: [GUIClientWorkerThread:4] Cache file LastWriteTime:26/06/2017 16:34:21  _lastSyncTime:26/06/2017 16:34:21
27/06/2017 12:03:32 <mutex>: [GUIClientWorkerThread:4] Released mutex 1828
27/06/2017 12:03:32 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.LocalRefreshEvent
27/06/2017 12:03:32 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:32 <repositoryexplorer>: [Main:1] refresh requested while we were disabled
27/06/2017 12:03:32 <eventengine>: [GUIClientWorkerThread:4] Event fired: VaultClientOperationsLib.ChangeSetItemsChangedEvent
27/06/2017 12:03:32 <repositoryexplorer>: [Main:1] refresh requested updateKnown is False
27/06/2017 12:03:32 <mutex>: [Main:1] Created mutex 1468 (Vault_WF_1488762371)
27/06/2017 12:03:32 <mutex>: [Main:1] Took mutex 1468
27/06/2017 12:03:32 <wf>: [Main:1] wf C:\KPX mutex locked
27/06/2017 12:03:32 <mutex>: [Main:1] Released mutex 1468
27/06/2017 12:03:32 <wf>: [Main:1] wf C:\KPX mutex released
27/06/2017 12:03:32 <wf>: [Main:1] wf for C:\KPX created
27/06/2017 12:03:32 <mutex>: [Main:1] Created mutex 1728 (Vault_WF_486604357)
27/06/2017 12:03:32 <mutex>: [Main:1] Took mutex 1728

[jclausius]

We've received the log, and are taking a look at the entries.

[Beth]

This issue is continuing discussion offline.

HS: 248900